MediaDefender attacks and cripples Revision3 for locking out its spy-bots

Discuss

33 Responses to “MediaDefender attacks and cripples Revision3 for locking out its spy-bots”

  1. takeshi says:

    “Because in this country, as far as I know, we’re still innocent until proven guilty – not drawn, quartered and executed simply because someone thinks you’re an outlaw.”

    I assume they’re talking about America? In the United States, accusing law-abiding citizens of nefarious and illegal activities has become the norm. Where the idea of justice is founded on clemency, and not retaliation, we just don’t get it here, and we probably never will.

    Turn on the television, and you’re guaranteed to see some blithering moron like Nancy Grace talk about how everyone who’s on trial deserves the death penalty. Look around… no one believes O.J. Simpson is innocent. Perhaps there is good reason for that, but he was found not guilty in the criminal proceedings surrounding his wife’s murder. Still, he’s out of a job. Not that I’m complaining about that, but it does perfectly illustrate how a person might be considered guilty, even after their innocence has been established.

    Anyone remember Kobe Bryant? Robert Blake? Phil Spector? All guilty, according to former prosecutor and current eyesore Nancy Grace. What about all the “terrorists” locked up in Gitmo? Throw away the key, says Fox News. Even underage victims like Elizabeth Smart and Shawn Hornbeck get blamed for their “complicity” by blowhards like Bill O’Reilly. One needs no evidence of wrongdoing to perpetuate this culture of blame and misguided vengeance. You only need a water cooler and an “opinion” about something you know next to nothing about.

  2. NickD says:

    I am imagining a bunch of Digg kids panning their retaliation right now. Go get ‘em!!!

  3. Jake0748 says:

    [Villagers begin to assemble, grumbling ominously]

  4. zedza says:

    @Enochrewt – Maybe they’ll use that excuse, too. That or the Chewbacca Defense. :)

    http://en.wikipedia.org/wiki/Chewbacca_defense

  5. Fnarf says:

    No brackets here. Death penalty.

    Shut down the business, jail the principals, subpoena every bit and scrap of paper in the place, find out who paid them, and fine them all $100 million apiece. This is outrageous. If Teen Hacker Jr. did this he’d be in prison for ten years.

  6. klobouk says:

    Hopefully, this sort of proof of corporate wrong-doing may actually force some reform in their tactics, at least.

  7. hyperkine says:

    If it were anyone else besides MediaDefender I’d say it was a mistake.

    But this organization has a track record of outrageous behavior. They should be punished, severely.

  8. WeightedCompanionCube says:

    MD flooding revision3 when they closed their tracker – negligent programming, but not malicious.

    revision3 allowing ANYTHING but their own torrents on that tracker – just as negligent. Basic security, folks… please!

    so far we’re even, however:

    MD injecting fake torrents into every tracker it can? That’s as low as warez kiddies taking over unsecured FTP sites. It’s illegal, and now that they messed with a tracker that was being used for expressly legal purposes, they can be taken to court. End of story.

    Speaking of being taken to court, I would like to see BB be more careful about making outright accusations in headlines/summaries. Revision3 was careful not to accuse MD of intent (other than the obvious tracker exploitation) or retaliation in regards to the effective DoS.

  9. Pablo Defendini says:

    I applaud Jim Louderback for ostensibly giving MediaDefender the benefit of the doubt (at least in public), but I highly doubt this was a mistake: MD’s behavior here tracks closely with their usual mafia-like M.O. (as has been chronicled by Ars Technica, among others, for quite a while now).
    I’m glad to see that the Revision3 crew are getting the FBI involved, and agree with some of the commenters on the original post: (I can’t believe I’m saying this, but) this calls for some aggressive and intentionally destructive litigation.

  10. Todd Knarr says:

    @WCC #20: Rev3 did block outside torrents. Or rather, they blocked all torrents that didn’t have the same hashcodes as the official torrents they were supposed to host. That’s a lot faster and avoids having the tracker have to retrieve the entire torrent it’s being asked to track before deciding whether to accept it. But there is a loophole: if the uploader manipulates their torrent carefully they can generate a hash collision, a torrent with the same hashcode as another one. Doing this is highly non-trivial and the odds on it happening by accident make the odds of 4 poker players all being dealt royal flushes in the same hand look like a downright certainty by comparison, so MD pretty much had to deliberately break the tracker’s security to get their torrents hosted.

    As far as negligent programming on MD’s part, it goes beyond that. I program TCP networking for a living. Standard TCP SYN retry timing is *very* non-aggressive. To get the observed 8K SYNs/second, MD’s servers would have to be trying some 120-240K simultaneous connections to the tracker. That volume’s high enough that even a marginally competent programmer can tell it’s going to DoS the target server even if everything goes right. And doing it with a lower volume requires bypassing the normal network stack and generating your own SYN packets ignoring the normal TCP retry timings specified in the RFCs. This isn’t going to happen by accident, and any programmer good enough to work down at the raw IP packet level knows the consequences of being too aggressive. The field’s full of stories about stupid network programmers who tried exactly that for “TCP network accelerators” and completely destroyed their customer’s networks as a result. There’s no way MD did this without knowing (from their techies telling them) *exactly* what was going to happen. They may have decided not to listen, but that doesn’t get them off the hook.

  11. WeightedCompanionCube says:

    Todd – are you assuming one server was SYN flooding? Isn’t it more likely MD has a cluster behind a NAT? The blog hints that the fake torrents had been deauthorized. That leads to one of two possibilies:

    – The fake torrents were removed. Attempts to access them would have led to more than a SYN flood, because you have to go thru the 3-way handshake before you can send the request for a specific torrent. So, someone is possibly not telling the whole truth, and a good part of that 8000 pkts/sec was other packets as well. “observed” is taking Rev3′s word as fact… and, well, it might not be. That would be up to the authorities who (hopefully) investigate this to determine.

    – the MD IP ranges were banned, most likely with a TCP RST reject. (Using drop would (or at least should) tarpit the source.)

    …and who said anything about marginally competent programmers? They put all their effort into matching hashes with Rev3′s torrents :) Say you’ve got a big cluster of BT poisoners, normally cycling thru lots of trackers, but so poorly designed that if something unexpected happens, they stop going thru the queue and instead try to connect over and over. Now imagine a whole cluster eventually getting to the Rev3 tracker and getting stuck. On a 9Gbps pipe.

    I’ve have personally seen a clusterfsck of clients behind a NAT blow away a server, and it looks just like an intentional DoS from one, or a few, IPs.

    But wow, debating technical points on BB is.. well.. pretty much digg/dot. Which this what this is turning into.

  12. KeithIrwin says:

    Rev3 didn’t mention anything about blocking IP addresses. From what they’ve said (and their posts to their user forum are consistent with this as is this guy’s blog post: http://www.lazycanadian.ca/2008/05/29/it-was-an-accident-i-swear/ ), all they did was disable the torrents which the RIAA had added. Once they did this they shortly started getting a flood of incoming SYNs.

    Someone suggested that this could be a natural result of dropping the torrents. It could not. BitTorrent trackers use the HTTP protocol to communicate. So first a TCP connection is established. This requires that both sides talk to each other. Then once the connection is established, a request would go over that connection for information about a particular torrent. Then the server would respond that the torrent in question is not being served. All of this would take somewhere between about a tenth of a second and two seconds. It would start with a SYN packet and then result in an ACK and then several more packets of different types. But Rev3 wasn’t seeing packets of different types, they were just seeing lots and lots of SYN packets. If, for some reason, the initial SYN was not responded to, eventually, another connection would be attempted, but not until the first connection had timed out, which would take several seconds at minimum. It appears that in practice Rev3′s servers were responding to the SYNs and establishing connections, but MD’s clients weren’t then using the connections.

    WCC #26 suggested that Rev3 had blocked MD’s IP addresses by simply dropping all incoming SYN packets at the firewall stage. Aside from not being consistent with Rev3′s accounts, this isn’t consistent with what we know happened. If this were the case, the SYN flood attack would not have worked. SYN flood attack work by tying up the system resources required by connections. If the system is simply dropping incoming SYN packets, then those don’t tie up resources, and hence it would’ve been an unsuccessful DoS attack, perhaps causing some slowing, but not shutting down the server. Any average firewall connected to a 100-base T line could easily drop 8000 packets per second without getting overwhelmed by it.

    So we know that it isn’t a natural consequence of anything which Rev3 did. Could it have been an accident on MD’s programmer’s part? No, it couldn’t. There’s no way to accidentally generate a SYN flood. In order to generate a flood of SYN packets out of a machine, the first thing you have to do is either a) to bypass the TCP/IP stack and write your own networking code completely from scratch or b) find some SYN flood code which someone else has written. Neither of these are the sort of thing which you would do in the course of normal network application writing, even if you were writing some special P2P fake-traffic-injection application. Even a P2P application whose job it is to inject fake traffic into BitTorrent would still use a standard TCP/IP stack. Writing your own TCP/IP stack would be a huge amount of effort (hundreds, if not thousands of man-hours) for zero benefit (every modern OS already has a good quality TCP/IP stack written for it) unless you were doing something like writing a program to do SYN floods.

    So, unless Rev3 is lying or someone spoofed MD’s IP addresses, we know that MD built a SYN-flood program and then launched it against Rev3. Now, they may have launched it by accident due to poor programming, but that’s the extent to which this could conceivably be an accident. They have also previously been suspected of using DoS attacks against peer-to-peer sites and sites which shared the big archive of their internal emails which got leaked.

    My best analogy is layman’s terms is presented as a play in one act.

    Danger on the Air

    The Characters:

    The Reverend Robert Vision, the Third (Rev3 for short), a humble community television personality who hosts a show called “Ask the Reverend”.

    Mediya D. Fender, an unlicensed private investigator. Dangerous and unscrupulous, but men in business suits find her very sexy.

    No Blearc, friendly bystander.

    Scene One: Community Television Studio

    Rev3: Hello, and welcome to Ask the Reverend. Today our topic is going to be Tarzan: man of nature’s god or unrepentant jungle savage. If anyone would like to discuss this topic, please call in. The number is JKL-WAKR, that’s 555-9257. Oh, I see we have a caller. Hello, caller, you’re on the air with Ask the Reverend, what are your thoughts on today’s question?

    No Blearc: Hello, Reverend Vision. Long time listener, first time calling in live. I’m actually not calling to talk about the topic. I’m calling because I wanted to let you know that I called the show’s answering machine yesterday to leave a question for you and someone has tampered with your message.

    Rev3: Really? Let me see.
    [Pulls out an answering machine from behind the couch, and presses play.]

    Message: [rev's voice] Hello, you’ve reached Ask the Reverend. We’re not on the air right now, but if you leave a message we’ll address you question in the next show. If you want a transcript of the program please send a self-addressed stamped envelope to the Community TV Studio, Box 34, Paris, New Jersey.
    [other voice] And if you want some bootlegged DVDs, come to booth 623 at the flea market.

    Rev3: Oh my. Well, thank you for letting us know caller. I’ll have to fix that.

    [Pulls out an answering machine from behind the desk, and presses play.]

    Answering Machine: Incoming. Call. Message. Erased.

    [Presses another button]

    Answering Machine: Record. New. Message. Now.

    Rev3: Hello, you’ve reached Ask the Reverend. We’re not on the air right now, well, I mean, we are on the air -right now-, but well. Maybe I should try this-

    [BOOOOOOM. Studio suddenly explodes and is covered in smoke.]

    Rev3: What in god’s name was that?

    [Smoke clears, and Rev3's desk has been destroyed by a giant cannonball.]

    Rev3: Someone appears to have shot our show. And with a cannon, no less. That’s really mean. I wonder who would do that. Wait a second, I think that there’s a name on it. It says “Property of Mediya D. Fender, PI”. Huh. I wonder if they’re listed in the telephone book.

    [Salvages a telephone book from the rubble]

    Rev3: Oh, here she is. She’s even got a full page add in the yellow pages: “Mediya D. Fender, unlicensed Private Investigator and Bootlegger Exterminator. We investigate and disrupt bootleggers by all means available, from spying on them to hacking their answering machines to shooting them with cannons. For prices, call us!”

    [Picks up the telephone from the ruins of his desk and dials a number]

    MD: Hello, Mediya D. Fender, unlicensed P.I. What can I do ya for?

    Rev3: Hi, this is Reverend Robert Vision the Third, of the Ask the Reverend show, did you just shoot my show with a cannon?

    MD: No, don’t be silly. We’d never do that. We don’t shoot people with cannons.

    Rev3: But it says in your ads that you do.

    MD: Only bootleggers, they don’t count.

    Rev3: It also says that you hack into answering machines. Did you change the message on my answering machine?

    MD: Oh, sure, yeah, we did that. I’ve been surreptitiously changing your answering machine messages for months now. Don’t worry about it. It’s not real. There’s no such booth at the flea market as number 623. The numbers only go up to 100. We’re just faking out the people trying to buy bootlegs.

    Rev3: But why on my answering machine?

    MD: Well, you were distributing transcripts of a copyrighted show, so your answering machine is the sort of place bootleggers might hang out.

    Rev3: What are you talking about? All I’m doing is offering transcripts of my own show. There’s no bootlegging going on.

    MD: Well, now, how were we to know that? We don’t have time to investigate things like that. We have to fight bootleggers.

    Rev3: And how come as soon as I changed the message you shot me with a cannon?

    MD: Look, I already explained to you, we didn’t shoot you with a cannon. Now, when you erased our message, it would’ve triggered some alarms from our side. So, we might’ve checked on things some. We maybe shot some BBs over towards your studio, you know, just to make sure it was still there and see if we could find where the message went, but we definitely didn’t shoot you with a cannon.

    Rev3: But there’s a cannonball here with your name on it.

    MD: Look, we changed your answering machine message, yeah. That was us. Sorry. And we might’ve shot a few BBs towards you when you changed it back, but we don’t know anything about any cannon ball.

    Rev3: I’m calling the cops.

    Fin.

  13. jowlsey says:

    [rabble rabble rabble]
    [waves pitchfork menacingly]

  14. oldnumberseven says:

    I would like to see Seth Bullock drag media defender across the camp by the ear and toss them into lock-up. It is all Amalgamation and Capital.

  15. prom77 says:

    If we want to make the general public aware, and care about, this kind of activity, perhaps we should frame it in the context of terrorism. Because as far as I’m concerned, this attack is internet terrorism.

    Sorry if that sounds hyperbolic, but I’ve been steaming about this attack for two days, and that’s the calmest statement I’ve made so far.

    I’m just afraid that the only way to get MediaDefender and its ilk to stop this kind of activity isn’t a lawsuit (companies who are bold enough to engage in this sort of shady activity are sure to have the liability insurance to absorb a lawsuit) but public exposure. That means not just an outcry among blog readers and media nerds like us, but among the general public. The people for whom computers are still the scary mystery boxes that live in their basement and get turned on a couple of times a month.

    So I’m just wondering… If the debate weren’t about a “DoS attack against the servers of a video podcaster” but rather a “terrorist-like attack against an American company… and the consequences it could have for YOUR FAMILY” would that get people’s attention?

  16. Agent 86 says:

    These are the people that are still illegally operating without a PI license, no?

  17. proto says:

    After reading what I can find on this situation, it’s pretty obvious that MediaDefender acted with intention and malice-aforethought.

    I’m tired of these corporogreedsters acting like whiney, selfish 5-year-olds, stomping around the internet as if it’s their private sandbox.

    MediaDefender needs the legal equivalent of being grasped securely around it’s scrawny neck and shaken until it’s little pea-brain rattles around in its otherwise empty skull-pan.

    Do it Revision3. Do it for us all!

  18. Ugly Canuck says:

    Sue ‘em for interfering with your contractual relations with you customers….

  19. Michael Brutsch says:

    “Because in this country, as far as I know, we’re still innocent until proven guilty – not drawn, quartered and executed simply because someone thinks you’re an outlaw.”

    That’s rich, “as far as I know”. Perhaps he should, you know, turn on a television, or read a paper.

  20. zedza says:

    If you Google their tracker address, you can see that they were tracking all kinds of pirated material.

    http://tinyurl.com/424ab5

    If they’d kept on top of what their servers were doing (or if they didn’t turn a blind eye to it) then this probably wouldn’t have happened.

  21. dculberson says:

    Zedza, what do you mean? The.Bucket.List[2007]DvDrip-aXXo sounds like a perfectly legit torrent to me!!

  22. ncl says:

    So, Zedza, the right way for MediaDefender to respond to the possibility that Revision3 was turning a blind eye to piracy was to destroy their site and not give them a call?

    When you notice that someone has parked illegally on your street, do you complain to them, or do you just set their car on fire to teach them a lesson?

  23. Pablo Defendini says:

    Zedza, regardless of what Rev3′s severs were tracking, DoS attacks are very illegal, as Louderback points out in his post:
    “Denial of service attacks are illegal in the US under 12 different statutes, including the Economic Espionage Act and the Computer Fraud and Abuse Act.”
    Even assuming that Revison3 was doing anything wrong (which I’m not implying), two wrongs don’t make a right.

  24. sum.zero says:

    zedza – if you read the blog post, you’ll see that rev3 clearly stated that md placed a bunch of fake torrents on their tracker.

  25. mzinformed says:

    [watches while sharpening quillotine blade]

  26. Todd Knarr says:

    @WCC #26: Rev3 probably put a drop rule on their firewall for MD’s IPs. Nobody’s going to set it up to go through the whole TCP handshake and then reset the connection. Too costly in terms of resources. Standard method is a drop rule (all incoming packets from that range get dropped at the router). Correct method technically is to send an ICMP “administratively prohibited” error in response to the initial SYN, but most admins prefer the tarpit effect of a drop rule.

    And it may well be a cluster, but it still requires (assuming standard TCP retry timings) a quarter-million simultaneous connect attempts to generate 8K SYN packets a second. Rev3 may be misstating things, but all their statements do seem to be consistent with a network that added a drop rule. And MD does advertise the disruption of P2P networks by exactly things like a SYN flood (which is what Rev3 describes) against the trackers to take them off-line.

    I won’t go into the technical aspects, but with the finite number of TCP ports available and the fact that a fair number of those aren’t available for outgoing client connections, if their programmers were using a standard TCP stack then they could handle a maximum of about 63K attempts every 15 minutes on a single machine. Beyond that rate, their client would be getting continuous system errors trying to allocate a new socket and failing, and would probably be blowing up because of that.

  27. SeppTB says:

    Zedza – along with what Sum.Zero said, they found lots of fake torrents up on their tracker from Media Defender. But also the entire thing started when they did notice that these torrents were there (a user on the forums pointed them out), and began closing them down. Then it seems MediaDefender’s servers couldn’t hit their fake torrents for tracking, freaked and launched into DoS mode.

  28. zedza says:

    @NCL – It seems more likely that when Revision3 blocked MediaDefender that their (MD’s) servers just kept trying. Sounds more like bad programming than a malicious attack since it was over the weekend, but who knows.

    @Pablo – I agree, but I guess it depends if MD’s servers/software were being malicious or just coded badly. If a SYN isn’t replied to you don’t know if you’ve been blocked or if the remote host has gone offline. It just seems more likely to be bad code.

    @sum – Then they must have been doing it for quite a few years, if you look at some of the Google cache links. Again, from the Google search results, it seems more likely that it was also being used to track pirated torrents.

  29. Enochrewt says:

    [Enochrewt begins to hand out pitchforks and torches]

  30. Enochrewt says:

    #16 ZEDZA: Bad programming? So I can go hit people in the streets with my car and call it bad driving? Maybe using the excuse “I was drunk” would help.

  31. Anonymous says:

    You have a legal claim against them for their “hijacking” of your business tools. Sue their pants off.

  32. Anonymous says:

    Correct me if I’m wrong but isn’t this type of activity ILLEGAL under current legislation?

    Revision3 should haul their sorry asses into court like Media Defenders clients do.

    Perhaps a few $100 million judgements against this ILLEGAL behaviour would loosen the minds of these stunningly stupid “defenders”!

Leave a Reply