Sitemaps and Robots around the Web

Robots.txt for websites around the world.

This is to be used for educational purposes and Security Researchers and Bug Bounty Programs.

Robots.txt

US Companies

• Amazon
• Apple
• Facebook
• Github
• Gitlab
• Google
• Target
• Walmart

Sitemaps

Only add sitemaps that are reasonable in size.

• Google

How To

• Robots.txt Basics
• Sitemap Basics

Contributing

Any contributions you make are greatly appreciated.

1. Fork the Project
2. Commit your Changes
3. Push to the Branch
4. Open a Pull Request