Or “How to Lie With Statistics”. As I read James Taranto’s column today on the early reports of successful enrollments at Healthcare.gov, I realized that many people might not understand how sensitive error reporting can be to just how you discern and report errors. Take optical character recognition (OCR) software, for example. When reading the advertising blurbs for it you frequently encounter extremely high recognition rates, 99% or higher. 99% of what?
What they don’t generally tell you is that the recognition rate is characters. Not pages or, worse yet, documents. When you’re talking about a recognition rate of 99% of characters that means that on average the program will correctly recognize 99 out of 100 characters. That in turn means that on average for every page with more than a tweet-full of characters the page recognition rate is zero. The document recognition rate is zero. That can make the difference between a particular project making financial sense or not.
In the real world as opposed to in a lab that’s an important distinction. In dealing with a large project of thousands or even millions of documents it determines how many proofreaders you’ll need. It can determine how long the project will take and how much it will cost.
Twenty years ago the company of which I was a principle at the time had a project for the Federal Reserve that required us to scan, store, process, recognize, and index about a million pages worth of documents in a very tight timeframe. It was a monumental task, as you might imagine. The niggling little details matter.
That’s what I think of when I read reports of 50,000 or 500,000 enrollments. What’s the enrollment rate and how is it measured?
Let me also remind you that the open enrollment period for the healthcare exchanges ends on March 31, 2014 and that time is passing. What was 200 days is now more like 150. That affects the peak load requirements of the system. If anybody tells you that a system that can accept and process 100 applications a day can definitely accept and process 100,000 applications per day, fire him.
Can Healthcare.gov meet the challenge? Sure. Will it? Who knows? Pointing to the tremendously different processing requirements of the Massachusetts system as a model is fatuous. They are not comparable. We’re in unknown territory.
I was trying to make the point yesterday at OTB that the Massachusetts and federal programs are not comparables.
In theory, the federal program allows enrollment on the first day and the last day.
It was impossible to enroll in the Massachusetts program on the first day and one couldn’t wait to start the enrollment process on the last day.
The federal program is a technological improvement, and it could encourage reluctant enrollees to wait to the last minute, and will that be technically possible? As I understand the problems, its not simply a matter of having enough servers, but that the program can only be as good as the weakest system it integrates. I wouldn’t be surprised if in Massachusetts, the application reviewer would physically go to different computer terminals to verify different parts of the application.
Interestingly, Massachusetts kept with its antiquated enrollment system until October 1, and now its having similar technical problems, particularly when the state system fails to connect to necessary federal databases. Massachusetts hired the same consulting group as the feds.
So we really are in unfamiliar ground here. Massachusetts didn’t try to do it this way, and frankly couldn’t use the federal tax code if it wanted, as a means of subsidizing insurance purchases.
Sadly, OTB is becoming increasingly a site whose commentariat is very intolerant of analysis as opposed to ideology.
Implementing a system without a purely manual fallback is a tyro’s error. Purely manual administration is something that’s been done successfully for thousands of years. When you have a volume problem, you throw more people at it.
Another issue: I don’t think there’s a state in the Union in which people can be deemed insured until they’ve made their first premium payment.
Sadly, OTB is becoming increasingly a site whose commentariat is very intolerant of analysis as opposed to ideology.
OTB’s ideological intolerance mirrors that of some of the most intolerant right-wing sites. They basically have become the Limbaughs of the left, and their shrinking number of opposing opinions are easy targets for red-meat type taunting by those joy-riding on the high-horse of unabridged ideology.
@Dave Schuler
Years ago, I was also involved in an OCR project. We finally gave up and switched to manual entry. It was technical data, and we were spending a lot of time finding errors. Two data entry clerks were able to do in weeks what was taking months.
Instead of an automated website, the government could hire a lot of people to do it manually. This would employ some of the unemployed. This could be the WPA of this depression. Security could be maintained by compartmentalizing the workflow.
Sadly, that would pose insurmountable political problems for the Obama Administration. They’ve been selling sizzle for four years. They can’t stop now and tell people that not only are they not getting sizzle, they’re getting hamburger instead of steak. Even if it’s true.
The Strata-Sphere is a blog run by a guy whose day job is working with government IT problems. Usually his focus is on global warming, presenting scientific evidence flying in the face of de facto warmists. However, since the Obamacare rollout he’s shifted over to examining the flow of IT problems damning the consumer experience.
His latest commentary is appropriately entitled: When Arrogance Meets Ignorance You Have Obamacare.
Actually, his running analysis of the continuing crop of ‘glitches’ makes for good reading, especially those with tech-type minds. I’m not one of them. However, even inexperienced people, like me, can pick up on and understand the expansiveness of the problems inherent in the creation, architecture, and testing of the government’s healthcare web site, ultimately leading to it’s consummate failure.
According to Strata the WH excised out many laborious details, called “tailoring,” which may have provided a higher level of web site understanding and functioning. Why? Seemingly, WH paranoia was the prevailing reason, as there were thoughts that the R’s might exhume too much defamatory data from too much diagramming of the program, before the ’12 election, to use against the Obama Administration. Consequently, it appears IT implementation, associated with a greater operational capacity, yielded to what kind of political implications it would manifest in the process. If it explicitly produced clarity negative to Obama’s reelection chances, it was summarily nixed
The conclusions of Strata, along with a growing number of IT people, in the know, is the following:
Strata-Sphere is a former Watcher’s Council member.
There are competing temptations in looking at major software development projects. Developers tend to undervalue the stuff that’s already been developed while the folks who pay the bills tend to overvalue it. Starting over might be the right decision but it’s politically impossible.
The only way the Administration has a chance of saving face is if they keep pressing the “Fix it” button as long as they possible can. So that’s what they’ll do.
re payment; I don’t know how insurance companies deal w/ payment , but I would assume they accept non-cash payment conditionally, i.e. a check delivered (perhaps even simply put in the mail) is sufficient to initiate the policy, but until such time as the check clears, the policy might be vulnerable to rescission.
And a related problem is that state laws might allow insurers to rescind insurance policies when information in the application is untrue, even if its due to an innocent mistake. If the federal exchanges are miscommunicating information to the insurance companies, the policies might be vulnerable to being set aside, unless this was addressed specifically in the rules setting up the exchanges.
Yeah, that’s the normal policy. I’ve never heard of any other way of doing things. Further, my understanding is that in Illinois, Michigan, and most other states a check and complete, accurate paper work submitted binds the insurance company. It’s a contract.
Several years ago, the state was trying to set up a system that would integrate its law enforcement / security data with the Dept. Homeland Security. This was either a federal requirement, emerging out of 9/11, or the ability to network with the fed databases was its own encouragement.
So, my friend in the responsible agency (with a business background) said that we knew at the time that California had established such a system and it had passed FBI clearance. (FBI approval being the main criteria here) The logical thing to do would be to find the firm that did the work and try to get them to replicate it at a lower price, while finding other similarly situated vendors to compete. What he was suggesting was fixed bids at best, criminal corruption at worse. The state went another way and sent out an open bid, which was won by the lowest bidder, who hadn’t set up such a system for any other state. The low bid proceeded not to ever finish the job, and started charging for each “house call” to fix enumerable glitches, and then slowly stopped returning the calls altogether.
Why don’t you sue them? He said he thought management was too embarrassed about the whole thing, and were just going to let it drop and send out new bids. Start again from scratch.