Welcome to the Pondera FraudCast, a weekly blog where we post information on fraud trends, lessons learned from client engagements, and observations from our investigators in the field. We hope you’ll check back often to stay current with our efforts to combat fraud, waste, and abuse in large government programs.
At Pondera, we are often asked whether fraud detection algorithms will ever completely replace human investigators. And while I can’t address the “ever” part of the question, I can confidently state that it will not happen in the foreseeable future. One of the major reasons for this? Prediction models, like many people, struggle to distinguish between cause and effect.
A Stanford University professor recently shared her studies on this topic which support many of our own findings. She noted that while prediction algorithms are excellent at finding patterns in large data sets, their effectiveness is limited because they struggle with determining causation. An example she used is that algorithms have been shown to help identify patients who should not receive hip surgery because they would likely die of other causes. However, the algorithms are unable to prioritize those patients who should receive the surgery.
In several cases, the professor notes that correlation can be as low as 50%. And she properly notes that while this may be fine in certain situations, governments simply cannot conduct such high-risk experiments with social welfare, economic policies, and other important matters. And unlike controlled environments, such as those that use placebos to test medications, the real world is simply too messy and unpredictable to control all factors.
This problem of causation identifies an important intersection between human reasoning and prediction algorithms. We believe that in complex, rapidly changing environments like fraud detection, effective detection systems combine the power of modern detection algorithms with experienced human reasoning.
By leveraging the individual strengths of both machine and human learning, we can analyze massive data sets and make sense of the findings. We regularly use the system to find the problem and ask the human experts to help explain the problem. This makes the results actionable, which ultimately is what our government partners require.
One of my colleagues recently returned from a conference on government program integrity with an interesting anecdote. He recounted a vendor presentation where the speaker was touting a 52% accuracy rate in their fraud lead generation system. So… nearly half of the system’s leads generated false positives. Not so sure I’d brag about that.
High false positive rates lead to wasted investigative time and money and unwarranted intrusions into the lives of legitimate program beneficiaries and service providers. Ultimately, they lead to a lack of confidence in the system itself and investigators revert back to more manual detection methods. When one considers all the important services governments deliver and the immense political pressure they endure, this is obviously not acceptable.
Shortly after hearing this story, we were asked to respond to a question about false positive rates and any existing industry standards or even benchmarks. While every vendor, including Pondera, makes claims about our system efficacy, very few standards actually exist. Conversely, our clients (the government program administrators) generally are subject to improper payment standards placed on them by the federal government.
I think there is a great opportunity, even responsibility, for governments to create these standards. Fraud detection standards would challenge the vendor community to “put up or shut up”, leading to more innovation. They could also be adjusted as the standards are met and surpassed leading to constant improvement. And they would provide governments with a uniform method for measuring vendor performance.
It is true that fraud detection systems still rely on quality program data and can suffer from the old adage "garbage in, garbage out”. So government would still share in the responsibility of meeting any new standards. But clearly, there is more we can do. And this would benefit all parties involved… except, of course, the fraudsters.