Can you audit the software that goes in your body?

The Software Freedom Law Center's latest white-paper, "Killed by Code: Software Transparency in Implantable Medical Devices," examines the strange circumstances around pacemakers and other implanted medical devices. Regulators like the FDA inspect the hardware designs for these devices in great detail, but the crucial software that runs the devices is a closed book -- a proprietary secret that's only ever called in for examination when the devices start to crash, with disastrous circumstances.
In 2008, the Supreme Court of the United States' ruling in Riegel v. Medtronic, Inc. made people with IMDs even more vulnerable to negligence on the part of device manufacturers.4 Following a wave of high-profile recalls of defective IMDs in 2005, the Court's decision prohibited patients harmed by defects in FDA-approved devices from seeking damages against manufacturers in state court and eliminated the only consumer safeguard protecting patients from potentially fatal IMD malfunctions: product liability lawsuits. Prevented from recovering compensation from IMD-manufacturers for injuries, lost wages, or health expenses in the wake of device failures, people with chronic medical conditions are now faced with a stark choice: trust manufacturers entirely or risk their lives by opting against life-saving treatment.

We at the Software Freedom Law Center (SFLC) propose an unexplored solution to the software liability issues that are increasingly pressing as the population of IMD-users grows--requiring medical device manufacturers to make IMD source-code publicly auditable. As a non-profit legal services organization for Free and Open Source (FOSS) software developers, part of the SFLC's mission is to promote the use of open, auditable source code5 in all computerized technology. This paper demonstrates why increased transparency in the field of medical device software is in the public's interest. It unifies various research into the privacy and security risks of medical device software and the benefits of published systems over closed, proprietary alternatives. Our intention is to demonstrate that auditable medical device software would mitigate the privacy and security risks in IMDs by reducing the occurrence of source code bugs and the potential for malicious device hacking in the long-term. Although there is no way to eliminate software vulnerabilities entirely, this paper demonstrates that free and open source medical device software would improve the safety of patients with IMDs, increase the accountability of device manufacturers, and address some of the legal and regulatory constraints of the current regime.

Killed by Code: Software Transparency in Implantable Medical Devices (via /.)

(Image: Medtronic EnRhythm Pacing System, a Creative Commons Attribution (2.0) image from winton's photostream)


    1. Utopian or Dystopian? I guess neither is an option, but it makes things so much more difficult.

  1. I’m 22 and have had a pacemaker for over 10 years and I had no idea this was the case. I can’t say I would feel any better having the code open source though.

    I do want them to add a flash drive in them so I can live out my Johnny Mnemonic fantasy.

  2. But all implanted medical devices come with a lifetime guarantee! What more do you want?
    The legal status of software is pretty much undefined: even though crappy software has demonstrably killed people. Is software a product? If it is then the producer is screwed if anything goes wrong. Is it a service? Then the producer must show that he undertook “due diligence” (ie. industry standard procedures were followed). More realistic but then there’s the problem that there are no industry standards of any kind.
    It would be great if someone got an enormous settlement in a lawsuit based on the theory that software writing is a service. Then people writing software that people’s lives depend on (something I’ve done a lot of) would have standards forced on them by their insurance companies.

    Think how great it would be to have a Lloyds of London guy saying “Sorry, that program will have to be redone if you want us to cover it: it has spaghetti logic/GOTOs/polka-dot objects/multiple inheritance/56 levels of inheritance/no regression test suite/…

    1. I think between this topic and the issue of Toyota cars/trucks having snafu’ed firmware, there needs to be legislation on embedded software that can affect lives. Open source all the way.

  3. lets not forget what pacemakers and internal defibrillators are made for – to interact with pathological stuff in your heart. now, that stuff is not 100% understood, is it?
    i would find it hard to prove a malfunciton in many cases, apart from failure of the device to call for a recharge or obvious false action when nothing is wrong with heart frequency at one specific point in time. but it is a very interesting issue, thanks for bringing it up!

    to maybe spice things up a bit, some info from (german) forensics –
    i remember one guy killed by malicious code a few years back in a forensic facility i worked as a student. the man worked in a factory where a robot was to store huge rolls of cable by putting them on big bolts in a wall. because of an error in the code, the machine tried to put the cable roll where there was simply no bolt in the wall – cable roll dropped and crushed the guy :(

    when dead bodies are cremated, internal defibrillators should be removed (at least over here), in some cremation facilities pacemakers must be removed as well (older ovens cannot handle the blow up of those things during cremation, while most modern ones can).
    now when the colleague usually doing the “crema tour” is on holiday, i have to do it.
    the little paper explaining how to remove an internal defibrillator reads like some bomb defusion manual – “first remove cable X, do not (!!) never ever, remove cable Z first! after X, then remove cable Y, it should then be safe to remove Z” :)
    XYZ means colors, but i forgot. so do not try this things at home ;)

  4. Commerical confidence may well come into play (they’ll certainly claim that it does).

    A non-profit industry auditing body to look at software code might be a middle-ground that gets “many eyeballs” without revealing either the Secret Algorithm or the Hideous Tarball.

    Aircraft are life-support devices with lots of software, and many more times the daily usage… how are they audited?

    1. Some independent auditing body would be necessary anyway, even if the code went open source.

      Open source only gives you many potential eyeballs. In practice virtually no-one is reviewing the vast majority of code that’s out there because there’s usually no incentive to do so unless you’re looking to change it or fix something that’s evidently not working.

  5. Open Source (FOSS) software developers, part of the SFLC’s mission is to promote the use of open, auditable source code5 in all computerized technology


  6. I know the point of this post was the open source software, but the specifics of medical devices raise another interesting question: If a medical device extends your life for 5 years longer than you’d otherwise have survived, is it right to sue because it didn’t extend your life 6 years?

  7. It’s less of a problem now, but in the 1980’s, a lot of the software they installed would also install a free copy of AOL as well.

  8. how does this compare to the situation for plain old medication? can i audit the drugs that go in my body?

  9. I work in the medical device industry, and spent many years doing Quality Assurance work on implantable medical devices and their software. It is not at all true that the software in such devices is not available for inspection to the FDA. ANY component of a medical device (implanted or otherwise, hardware OR software) is under their jurisdiction. The problem oftentimes is that the FDA inspectors are hardly educated enough about what software is, or how to evaluate software, that they are unable to make any judgments at all about whether a given piece of code is safe and/or effective. For example, only a few years ago, I had to explain the extreme basics of software to the MOST qualified FDA inspectors, answering questions like “What is a compiler?” and so forth.

    If Boing Boing readers want to change that situation, tell them to petition lawmakers for more FDA funding and ALSO, at the same time, higher standards for FDA field inspector training.

  10. A little quibble about the Medtronic decision. The Court did in fact say that you can’t sue medical device makers in state courts. What you fail to mention is that you can still sue in federal court.

    I don’t agree with the decision on a few wonky grounds.

    However, I think that in practice it wouldn’t make a big difference for public interest suits about disclosure of code but will be bad for any party (plaintiff or defendant) seeking advantage by contributing heavily to elected state court judges.

    Also, the decision doesn’t change the FDA’s ability to change its policies or for a suit in federal court against the agency itself.

    Medtronic is indeed a big case for people who care about these things, but it isn’t the triumph of big business the author makes it out to be.

  11. Just a few things to clarify that medical device software isn’t like most software you may be used to:

    1: “Software Validation” means testing every possible software state. This process is so expensive and such a pain, medical device makers shun using any software unless they absolutely positively have to and then use the simplest software on the simplest circuitry with the simplest interface that gets the job done. I’m aware of a project that may have spent several hundred million dollars TESTING software that may have been written for half a million. That 100+:1 ratio is common.

    2: Peer review and third party code audits. Before you start spending enormous cash on validation, there are multiple reviews, peer reviews, 3rd party review. You want your software rock solid. Because once you start software validation, any changes mean you have to start all over again with peer review.

    3: Jail time. Dozen’s of people have to sign hundreds of times. Signatures for test completion, results, traceability, biocompatibility, on and on and on. Signatures on specifications, test protocols, test reports, test summaries, etc etc etc. Each signature is effectively an affidavit to a federal agency (FDA). And what happens if you knowingly sign off on anything not true? You go to jail! What if you knowing skip a step (to avoid signing)? You go to jail! And they go after the CEO, Quality, and Regulatory people first.

    If you ever wonder why medical devices are so expensive, this is part of the reason: Development costs are 10 to 100 times more expensive than a similarly complex consumer widget.

  12. This is a really important issue. Its important for the QA/QC reasons but probably more so for the security of implanted health software.

    If, for example, Ray Kurzweil’s predictions of wide use of implantable information technology are a guide, we can expect pervasive and deep use of biomedical nanotech by 2034.

    There is good possibility there will be various scenarios where we have nanotech roaming our bodies, for example artificial respirocytes to bolster red blood cell function, insulin pumps, or neural nanotech connecting to our brains and nerves.

    All plans for these include making them software programmable, and communicating external to body, very likely to a wider network, than just our personal computers, for example to the patient’s doctor, device manufacturer, regulatory agencies, and private third parties, ostensibly over the internet.

    It is one thing to have defective devices or software and entirely another to have malicious code affecting the function of technical devices intimately interacting with our bodies and brains.

    Anti-virus functionality is going to have to be a huge part of this use of IT as biomedical prosthesis.

    It is also going to have to have an equally massive regulatory oversight given that FDA role in medical matters. But if regulatory agencies are struggling with single, macro-scale, fixed location implanted devices, how in the world will they deal with 1000’s (or millions) of a dynamic eco-system of nano-sized devices roaming about the body, each programmable, communicating with (effectively) the internet, in millions of people?

    The promise of nanotech is great but the accompanying threats are almost greater. Imagine Google or Facebook dragging all of your biomedical information into their matrix of data?

Comments are closed.