I used to develop in Java and found it almost impossible to get support from the global community (assuming I could even find a well populated support forum in the first place). Then I started developing in ASP.NET and it was like a breath of fresh air. Well featured, with a fantastic IDE, a friendly helpful support community, plenty of tutorials and demos on how to do the most basic (and advanced) things, etc… Quite simply, I can do more with ASP.NET in a fraction of the time it takes to do in Java. I have never looked back since.

On the few occasions I am asked to do Java development, I fill with dread. It seems these days that Java is an elitist, overblown, oversized, sprawling inconsistent mess with no standard framework or development environment. And that makes it difficult for programmers to actually develop in it if they wish to have a life outside of programming.

User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /help
Disallow: /slideshow/gallery/iraq

User-agent: whsearch
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /help
Disallow: /sitemap.html
Disallow: /privacy.html
Disallow: /accessibility.html

(From http://web.archive.org/web/20080325202806/www.whitehouse.gov/robots.txt)

So really (because the rest only applies to their own crawler) all they excluded is:

/cgi-bin, /search, /query.html, /help and /slideshow/gallery/iraq

Seems reasonable to me, and I think holding this up as an example of “Change” doesn’t really hold water.

The file would be a hell of a lot shorter if the robots.txt syntax allowed for /*/text and /*/print, but it doesn’t so the file ends up verbose.

In future, would BoingBoing kindly include some more background on what they’re linking? Especially technical articles.

ASP.NET (and the corresponding servers) is a lot like the M16…except easier to shoot yourself in the foot with.

(And frankly, even if they DID do it that way, it would have been Cheney et al hitting the delete button ;-))

They probably just did a coordinated server swap of some kind. It’s probably all load-balanced, anyway, so it might have just been as simple as flicking a switch.

There is this concept of a “Steve Jobs Reality Distortion field” and another called “Bush Derangement Syndrome”, I think this is the start of a new phenomenia, where in every step Gov’t takes is a direct result of a positive action by our new president, and we need to compare it to the previous administration to underscore the change.

I fall in with the “good website management” group – Bush, Cheney, Rove, Rice, and Rumsfeld all were smart enough to know you don’t post secrets on a public gov’t website, robots.txt file or not…

http://news.cnet.com/8301-13578_3-10146802-38.html?part=rss&subj=news&tag=2547-1_3-0-5

It’s ASP.NET because the contractor refuses to acknowledge any other language/technology. I used to work for them, and this was DEFINITELY a “lowest bidder” job. This was an utter clusterf— of a project and, remembering the work environment, I’m utterly amazed it turned out as well as it did.

(/includes is just where we kept css and javascript and disallowed in the default builds we had.)

If anything, Obama’s robots.txt file is too short. It doesn’t currently block search pages, meaning they’ll show up on search engines–something that most site operators don’t want and which runs afoul of Google’s Webmaster guidelines. Those guidelines say: “Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.”(Emphasis Added)

I’ve actually worked closely with the web staff of several federal agencies, building the software infrastructure, so I can answer that, to a certain extent.

ASP and ASP.net developers are inexpensive, plentiful and easily trained. Of the sites that I worked on, most of the work was done by contractors and subs (and even sub- subs). In a typical project, I’d talk with a GS a few times over the life of the project, but almost all technical details went to/ through the contractors. Typically the GSs were in managerial roles and more concerned with what the final website looked like, rather than what the back end technologies are. With the mantra of “noone ever got fired for buying Microsoft” alive and well in the US government, there’s no incentive to choose anything else. Plus, ASP and ASP.net are (relatively) easy technologies to learn. If you’re planning, long term, to have a site that has to be maintained by unknown people (who knows who the contractor dejure will be next year), there is a certain logic to going with ASP. A lot of the folks I worked with were not the types of people I would expect to learn anything new if they considered it ‘hard’. I’m not advocating ASP over any other technology (I’m a apache/Java type person myself), but it works well for them.
As someone else said, a lot of these sites have a legacy of being originally written in front page, and having migrated over time to ASP and then ASP.net as it’s sometimes easier to slowly upgrade the system than it is it re-architect on a new platform (and to teach your lifers a totally new set of skills).

Obama is in the pocket of Big Spider.

I would suggest the implementation of X-Robots in the WhiteHouse.gov site .htaccess file like this:

Header set X-Robots-Tag “noindex”

User-agent: *
Disallow: /includes/
Disallow: /search/
Disallow: /omb/search/

Robots
User-agent: *
Disallow: /includes/

Humans
User-agent: *
Disallow: /encrypted/

While part of me feels that it’s probably because they want a guaranteed level of support on products like that, I realize that they could get the same level of support while paying for more in people time as opposed to license fees. If you hire a bunch of people to work on open source solutions, you have hired people who have a connection to your project; rather than being forced to deal with some vendor’s busy and distracted support staff.

It’s just an interesting choice for the ‘Change’ we’re supposed to be getting.

Think of it like a no-entry sign stuck on a door. If you really don’t want strangers going in, you’d lock the door. It’s even less than that, because robots.txt only applies to robots, humans *are* supposed to ignore it.

The bush version is actually the preferred method… see googles webmaster guidelines.

The post shows the partisan and technical ignorance of the person posting it.