Internet Archive to ignore robots.txt directives

Robots (or spiders, or crawlers) are little computer programs that search engines use to scan and index websites. Robots.txt is a little file placed on webservers to tell search engines what they should and shouldn't index. The Internet Archive isn't a search engine, but has historically obeyed exclusion requests from robots.txt files. But it's changing its mind, because robots.txt is almost always crafted with search engines in mind and rarely reflects the intentions of domain owners when it comes to archiving.

Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes. Internet Archive’s goal is to create complete “snapshots” of web pages, including the duplicate content and the large versions of files. We have also seen an upsurge of the use of robots.txt files to remove entire domains from search engines when they transition from a live web site into a parked domain, which has historically also removed the entire domain from view in the Wayback Machine. In other words, a site goes out of business and then the parked domain is “blocked” from search engines and no one can look at the history of that site in the Wayback Machine anymore. We receive inquiries and complaints on these “disappeared” sites almost daily.

A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org). As we have moved towards broader access it has not caused problems, which we take as a good sign.

Read the rest

Vintage Chicago street gang business cards

Before the internet, even before desktop publishing, gang members who wanted calling cards headed to a printer with their idea. The results are collected in Brandon Johnson's Thee Almighty & Insane: Chicago Gang Business Cards from the 1970s & 1980s. Read the rest

New documentary on the history of graphic design and technology

The long-awaited documentary Graphic Means just premiered at the ByDesign film festival, describing a half-century of world-changing analog-to-digital shifts in how graphic designers worked. Here's the trailer. Read the rest

The 19th-century teen girl who overpowered men in traveling shows

Bill Kirby shares interesting stories about Augusta, Georgia history. Here, he discusses Lulu Hurst, a local teen girl who wowed audiences by overpowering any man who dared to accept her strength challenge.

Read the rest

A great oral history of Funny Or Die, which just turned ten

Funny Or Die is ten years old this week. The comedy site's launch a decade ago almost didn't happen. Wired has compiled a definitive oral history of the site, right up through its most recent reset as Trump ascended to the Presidency.

Read the rest

Neo-fascist presidential candidate Marine Le Pen says France was not complicit in rounding up Jews

Marine Le Pen says that she is not like her father, the notorious fascist political leader Jean-Marie Le Pen, founder of the far-right National Front party (she excommunicated him from the party, but remained chummy enough to borrow millions from him for her presidential bid). Read the rest

How America's obsession with hula girls almost wrecked Hawai'i

Over at Collectors Weekly, Lisa Hix has just written an incredibly in-depth history of the hula, from its roots as a sacred dance to its kitschy personification as a dashboard doll. For her piece, Hix spoke with Constance Hale, a hula dancer herself, whose new book, The Natives Are Restless, focuses on authentic, 21st-century expressions of the hula.

Snip:

In his journal, Captain Cook described the Hawaiians’ hula: “Their dances are prefaced with a slow, solemn song, in which all the party join, moving their legs, and gently striking their breasts in a manner and with attitudes that are perfectly easy and graceful.”

In The Natives Are Restless, Hale explains, “To be sexually adept and sensually alive—and to have the ability to experience unrestrained desire—was as important to ancient Hawaiians as having sex to produce offspring. The vital energy caused by desire and passion was itself worshiped and idolized.”

Cook and his men—and the merchants, whalers, artists, and writers who followed—mistook the hula’s sexually charged fertility rituals as a signal the Hawaiians’ youngest and loveliest women were both promiscuous and sexually available to anyone who set foot on their beaches. In her 2012 book Aloha America: Hula Circuits Through the U.S. Empire, historian Adria L. Imada explains how natural hospitality of “aloha” culture—the word used as a greeting that also means “love”—made Hawaiians vulnerable to outside exploitation. To Westerners, the fantasy of a hula girl willingly submitting to the sexual desires of a white man represented the convenient narrative of a people so generous they’d willing give up their land without a fight.

Read the rest

Historically accurate leggings based on medieval armor

Lorica is a fully-funded Kickstarter campaign that started with the promise of making leggings inspired by medieval armor, with rigorous and accurate depictions -- now that the campaign has reached its stretch goals, it has grown to incorporate a range of armor-inspired clothes. Read the rest

America has spent more rebuilding Afghanistan than it spent rebuilding Europe under the Marshall Plan

After WWII, the US launched the Marshall Plan to help Europe rebuild, spending about $120B in inflation-adjusted dollars on the project, which lifted the war-stricken European nations out of disaster and launched them into post-war prosperity; the US has spent even more than that on rebuilding projects in Afghanistan since the official cessation of hostilities there, but Afghanistan remains a crumbling, corrupt, failed state where violence is rampant, opium exports are soaring, and soldiers and civilians alike are still dying. Read the rest

Charming animated short on The Power of Privacy

The Power of Privacy is a brisk animated jaunt through the legal development of privacy, starting with the fireplace chimney. Read the rest

Stunning images of church organ pipes

German photographer Robert Götzfried has published his latest series on the original high-tech wall of sound: church organ pipes. His beautiful symmetric photos show the remarkable variations possible. Read the rest

What will the 25th century call the 21st century?

Polymath historian-novelist Ada Palmer has just published Seven Surrenders, the long-awaited sequel to her astounding debut novel Too Like the Lightning, in which she continues to spin tales in an intricately devised, wonderfully original 25th century. Read the rest

Supply of old-fashioned CRT arcade monitors dries up

The last manufacturer of arcade-sized cathode ray tubes is out of the business, with one supplier having only 30 or so in stock and no chance of ordering more. The manufacturing process is difficult enough that it's unlikely anyone will step into the breach; Venturebeat's Jeff Grubb reports that times will be good for skilled repairers.

“I have a feeling that — y’know how there are those guys doing pinball repair on the side — there will probably be some guy you can send your monitor to and have him rewind the bulb,” says Ware. “I think it’s going to be really expensive.” A CRT tube is very heavy, so shipping costs alone would be costly. “Right now, I don’t know of anyone who does [the winding].”

To fill the void, Day suggests that new companies will emerge to reproduce those old machines using only modern-day technology. An LCD screen connected to a PC running a piece of software that approximates the original experience will be adequate for most people.

CRT emulation is amazing, but still obviously such to me. But I bet using curved OLED panels embedded in thick CRT-style glass would fool my eye in darkness. There's yer Kickstarter. Read the rest

For sale: the toy blocks Einstein played with as a kid

Richard Davies writes, "Einstein's childhood building blocks have been listed for sale on the AbeBooks.com marketplace for books and collectibles." Read the rest

In 1937, a judge quietly asked Meyer Lansky to form a squad of Nazi-punching gangsters to raid Bund meetings

Meyer Lansky was an infamous and ruthless gangster -- albeit one so personally charming that his life is chronicled in a book called But He Was Good to His Mother -- and no friend of New York State Judge Nathan Perlman; nevertheless, as the Nazi-supporting German-American Bund staged more and more toxic rallies in New York City, Perlman quietly asked Meyer to form a squad of Jewish gangsters to disrupt their meetings. Read the rest

Milosevic, Berlusconi, Trump

I saw this coming, for the past ten years or more. I saw small Trumps, rising and tramping around, first timidly, then bravely, and finally boldly.

Read the rest

Femme Magnifique! A Kickstarter for an anthology about amazing women!

The fabulous Shelly Bond, former DC Vertigo editor and head honcho, just launched a kickstarter for an anthology called Femme Magnifique that she’s doing in conjunction with Kristy and Brian Miller at HiFi Color. Read the rest

More posts