Robots (or spiders, or crawlers) are little computer programs that search engines use to scan and index websites. Robots.txt is a little file placed on webservers to tell search engines what they should and shouldn't index. The Internet Archive isn't a search engine, but has historically obeyed exclusion requests from robots.txt files. But it's changing its mind, because robots.txt is almost always crafted with search engines in mind and rarely reflects the intentions of domain owners when it comes to archiving.
Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes. Internet Archive’s goal is to create complete “snapshots” of web pages, including the duplicate content and the large versions of files. We have also seen an upsurge of the use of robots.txt files to remove entire domains from search engines when they transition from a live web site into a parked domain, which has historically also removed the entire domain from view in the Wayback Machine. In other words, a site goes out of business and then the parked domain is “blocked” from search engines and no one can look at the history of that site in the Wayback Machine anymore. We receive inquiries and complaints on these “disappeared” sites almost daily.
A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to firstname.lastname@example.org). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.
An excellent decision. To be clear, they're ignoring robots.txt even if you explicitly identify and disallow the Internet Archive. It's a splendid remember that nothing published on the web is ever meaningfully private, and will always go on your permanent record.
In 1975, Pol Pot and the Khmer Rouge seized power in Cambodia after expelling a US puppet regime, surviving a brutal US bombing campaign despite the massive asymmetry between the Cambodian forces and the US military. Tian Veasna was born three days after the Khmer Rouge took power, and spent his formative years in forced labor camps as his family were beaten, starved, tortured and murdered. Today, Veasna is a comics creator living in France, and in Year of the Rabbit, Veasna creates a coherent story out of his family's narratives, giving us a ground-level view of the horrors of the Pol Pot regime, whose campaign of genocide led to the deaths of more than a million people.
Sweary historian Eleanor Janega writes on her Going Medieval blog (previously) that there was never a time in medieval Europe when bestiality was socially acceptable, and brings the receipts in the form of eyewatering details on the punishments for having sex with animals.
Peruvian police are deporting five tourists from Chile, Brazil, France and Argentina for allegedly sneaking into the Macchu Picchu citadel, defacating among the Incan ruins and breaking the stonework inside the Temple of the Sun.
The Nintendo Switch is an undeniably awesome gadget, pairing old-school gaming styles with modern-day graphics and functionality for a new generation of gamers. The only complaint people seem to have is that its controllers are somewhat lacking, which is why more and more Switch-enthusiasts are picking up this Gbros. Wireless Adapter that lets you play […]
More and more people are flocking to a wide variety of careers in IT, thanks mostly to the high pay, plentiful advancement opportunities, and an exciting atmosphere that offers new challenges every day. The only problem is that this high demand means competition can be fierce if you’re entering the job market for the first […]
Going to the beach is almost always an enjoyable experience, but trekking back through your house on the way to the shower can leave a trail of sand the quickly saps the day of its sunny fun. Thankfully this BeachBox: Portable Shower & Storage unit has you covered the next time you hit the beach. […]