Obama's whitehouse.gov nukes 2400-line Bush robots.txt file

Cory Doctorow 3:09 pm Tue Jan 20, 2009

Yesterday, the robots.txt file for whitehouse.gov had ca. 2400 lines worth of files and directories that search engines were not allowed to index. Today, the file is two lines long: "User-agent: *" and "Disallow: /includes/"

User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /omb/search
Disallow: /omb/query.html
Disallow: /expectmore/search
Disallow: /expectmore/query.html
Disallow: /results/search
Disallow: /results/query.html
Disallow: /earmarks/search
Disallow: /earmarks/query.html
Disallow: /help
Disallow: /360pics/text
Disallow: /911/911day/text
Disallow: /911/heroes/text

The country's new robots.txt file

Keep your media fully protected with 10TB of cloud storage from Degoo, now only $99.99

We thank our sponsor for making this content possible; it is not written by the editorial staff nor does it necessarily reflect its views. There are plenty of data storage… READ THE REST
My mom liked this play about Michael Jackson's Alien Glove. Will yours?

If you find yourself in Los Angeles anytime between now and April 8, have a penchant for the crass, and a date to impress or distress, go see the most… READ THE REST
Awe makes for kind and generous kids, study shows

Experiencing art can make kids kinder and more empathetic, according to a study published in Psychological Science. The study shows that experiencing awe, which can be brought on by viewing… READ THE REST
This $145 lifetime subscription to Rosetta Stone can help you learn up to 24 languages

We thank our sponsor for making this content possible; it is not written by the editorial staff nor does it necessarily reflect its views. With a minimum of 350 languages spoken in… READ THE REST
You can snag a refurbished MacBook with a Microsoft Office Home and Business subscription for just $500

We thank our sponsor for making this content possible; it is not written by the editorial staff nor does it necessarily reflect its views. If there's one device almost everyone… READ THE REST
This bundle will make your photos look better than ever for only $79

We thank our sponsor for making this content possible; it is not written by the editorial staff nor does it necessarily reflect its views. If you are into photography, or… READ THE REST