Automatic Diagnosis Machine -- FDA debate

Today’s post is about an article (http://www.nytimes.com/2013/07/21/business/dissent-over-a-device-to-help-find-melanoma.html?hpw) describing a new medical device to detect melanomas, and the factors that affect the FDA’s decision whether to approve it, and individual doctors’ decisions whether to adopt it.

One thing that struck me about the quotations from various FDA officials and doctors, is that academics – especially in fields such as information systems, and industrial engineering -- may have a lot to offer, and that as a community, we may want to think about how to make our knowledge more visible and available to policy makers.

One example that struck me, was a thread of quotes about the machine’s rate of false positives. A member of the FDA panel expressed concern that the false-positive rate was too high. But anyone with an understanding of the technology will realize that this rate is trivial to alter, and that the key metric is not either false-positive or false-negatives in isolation – since either of these can be trivially set to zero – but some combined measure of them both (e.g. ROC, average-precision, etc.). It is hard to imagine – but seems to be the case -- that the FDA panel did not know this. It also appears that the FDA was not provided with information that properly compares the machine against a human on such a combined measure. This is scary to me.

A slightly subtler thread that runs through the article, is about how the device is likely to be used. The argument is raised that doctors may get lazy and rely on the machine, in which case one has merely replaced the person with a machine. But let’s consider this argument in more detail. First, even if this is true, the machine may be better than the human, though I am still surprised that the FDA is asked to approve or reject a device without a clear answer to which (person or machine) works better when acting alone. But there appears to be lurking a second, stronger version of this argument, according to which the result of human+machine is worse than human alone (or machine alone). That is to say, the increased laziness of the human more than offsets the benefit of his/her having a better predictor. Can this be? Is there research that shows a phenomenon such as this?

I have posed this question to a friend who is more expert in this area, and I will post a follow-up....stay tuned