Friendster spider

Ben Discoe has created a system to "spider" Friendster.com and build diagrams of the results. His website points to source code and binaries, so you can dink around with it yourself:


I wrote a spider (a.k.a. "crawler", "scraper", "robot") to browse the Friendster site, recording the following information for each person: id (e.g. 867412), nickname (e.g. "Ben"), address (they call it "Location"), type (valid or suspended), friends (array of friend IDs). My spider program starts with a given node, and walks outward through the network (breadth-first) for as long as you tell it. It writes its output to an ad-hoc text file. On the day that i first run it, i have around 14 friends and 220,000 people in my "Personal Network". I run the spider until i have around 1000 of the people (friendsters) closest to me. The spider is then tweaked to write output that can be used for graphing the friend network….

Link, Discuss , (Thanks, Rob!)