Saving Lives With Analytics

Fortune has a brief article on aneurysm-spotting analytic software developed by IBM in collaboration with Mayo Clinic (HT to Satish Bhat for bringing this article to my attention).

To help in their aneurysm hunt, radiologists at Mayo Clinic use special software developed with IBM that analyzes a three-dimensional brain scan. Computer algorithms process information in the images, pick out abnormal areas where fragile blood vessels might be hiding, and flag the potential trouble spots for Mayo doctors. So far the results are promising. In trials the software found 95% of aneurysms; a typical radiologist would have found 70%.

95% vs 70%. How many lives saved as a result? I couldn’t find anything in the article on this question so I did some Googling.

Here’s what I found:

perhaps 25,000 to 50,000 people a year in the U.S. have a brain hemorrhage caused by a ruptured aneurysm.

Of these 25,000-50,000 people,

One-third to nearly half of patients have minor hemorrhages or “warning leaks” that later lead to a severe devastating brain hemorrhage days later.

So 8,000-25,000 people come in with a “warning leak”. Every one of their brain scans is presumably looked at by a radiologist. According to the Fortune article, radiologists have only a 70% success rate so let’s assume that 30% of the scans (i.e. 2,500 to 7,500 people) are mistakenly thought to be normal and, therefore, left untreated. They return days later with a burst aneurysm. What happens next?

The overall death rate once the aneurysm ruptures is about 40%

So, between 1000-3000 patients will die because the aneurysm wasn’t caught during the first visit.

Now, let’s look at how the analytic software will perform. According to Fortune, the software yields a 95% success rate so 5% of the scans (i.e. 400 to 1200 people) will be mistakenly thought to be normal and left untreated. Of these patients, between 160-480 patients will die (using the same 40% death rate as before).

Incremental lives saved? Between 800-2500 patients annually. Wonderful! Kudos to IBM and Mayo.

Here’s a little (hopefully) self-explanatory graphic. The blue box represents the incremental lives saved by the software; the red represents the lives that could be saved if the software’s accuracy goes to 100%.

p.s. I realize that numerous assumptions have been made in this back-of-the-envelope assessment. Feel free to criticize and improve. I just wanted to get a quick sense for how many lives would be impacted.

Share/Bookmark

3 thoughts on “Saving Lives With Analytics”

  1. Hello this is kinda of off topic but I was wondering if blogs use WYSIWYG editors
    or if you have to manually code with HTML. I’m starting a blog soon but have no coding expertise so I wanted to get guidance
    from someone with experience. Any help would be enormously appreciated!

  2. Great point, Badri. The article left out the important sensitivity vs specificity question.

    I am still hopeful that the algorithm adds value. Here’s my convoluted 🙂 logic: the IBM software is likely used as a preprocessor with the cases that it flags getting sent to a radiologist for a deeper look. If the algorithm is achieving a high true-positive rate by flagging just about every case, then, from the radiologists’ point-of-view, things have gotten worse. Apart from having to look at every case like they did before, they have to work harder to show that the algorithm was wrong in the numerous false-positive cases. And that wouldn’t have been newsworthy enough to catch Fortune’s interest (or, at the very least, not in IBM’s interest to be written up as an article).

    What do you think?

  3. Rama,

    I think the key measure left out by Fortune is not necessarily the accuracy (= True Positive, or Detection Rate).

    One can trivially extend the True Positive to 100% by simply flagging every lesion as an aneurism. What the article fails to mention are other key measures, most importantly what is the False Positive rate, the rate at which the software raises an alarm when in reality it is not an aneurism. If at 95% True positibe the False Positive was around 25% you have a terrific statistical pattern recognition product at hand. If, on the other hand the false positive is around the 75% level at 95% detection, you are not that much better than random guessing (which of course the wise readers of this blog would know would lead to 95% false positive rate).

    In medical applications, the cost of a False Positive versus a False Negative (i.e., missing an aneurism when really there was one) are very highly skewed – obviously, the cost of a false negative is extremely high.

    My only point here is that Fortune should have spelled out some other measures before suggesting that IBM has a true life-saving Analytics based tool. But then, that’s our role as geeks to educate them 🙂

    Badri.
    http://www.predictionmachines.net

Leave a Reply

Your email address will not be published. Required fields are marked *