Stealth on Health: A Surreptitious Study

 Media Blog Post 2

We have discussed many ways in class on how a study or experiment can take a turn for the worst. A telephone survey completed by the National Federation of Independent Business completed in July 2011 illustrates almost every point on how not to conduct a study. In Mismeasures of Man, we have read about faulty calculations. We have read and learned about the danger of generalizations (either in the statistics or the meaning of the results). The study which I will now discuss could be known as an emblem of such a problem of cultural context and observer background which affects the results.

To begin, the National Federation of Independent Business (NFIB) is a national, advocacy organization dedicated to the concerns of small-businesses and the "related-policy problems" that employers might encounter. The study is titled "Small Business and Health Insurance: One Year After Enactment of PPACA" (also known as the Patient Protection and Affordable Care Act. It is also popularly known as "Obamacare," but that phrase is made to sound insulting and minimize the issue at hand. Let's refrain from using it). This study was discussed in Fox News. The study conducted a questionnaire with 26 questions (some questions with multiple parts as well) and 13 demographic questions. The original hypothesis of the study was that PPACA would result in "bad business" mainly for small businesses, and would hurt and not help employers and employees with healthcare insurance.

I have several problems with how the survey was conducted as well as how the calculations were completed. First of all, their "representative sample" was chosen for "simplicity." There was confusion on how the PPACA had two different definitions on what a small business could be defined as. It could either be a firm with 50 or fewer employees or a firm with fewer than 50 employees. For "simplicity" sake, they decided to conduct using the former operational definition. Both definitions should have been represented if there was confusion. Also, the survey was not a random sample of the representative population. The sample was "drawn from" the files of a national database, Dun & Bradstreet. The survey was taken by 750 owners or managers of small businesses. According to the survey, this number is supposedly representing the millions of small businesses in the U.S. The healthcare debate is a very controversial issue around the country, and this was also not taken into account. I have read many analytical papers, and the terminology and phrases used in this study were very bias. For instance, the study concluded, "It is possible, some might argue likely, that PPACA will simply collapse, politically, legally, economically, and/or
administratively." That was the study's overall conclusion about the survey. At other times, the paper admits to places of error such as, "The percentage of those with 100 percent participation [in business health insurance] intuitively seems high. Since the survey asked about participation and did not mention eligibility, there is a possibility that the two concepts were confused. The potential effect of any confusion
is likely to increase the percent of participation...the data from firms to employees suggests a modest amount of confusion occurred and the survey estimates are somewhat high," but then tries to state that there is no problem: "Regardless, when small employers offer, their employees typically participate" (3). Another bias is shown when the word "promises" showed up in quotes when discussing the guidelines of PPACA.

Popular words and phrases such as, "virtually," "It is likely," "may lead," "almost assuredly" are in every paragraph. No definite conclusions were reached with the survey, especially with the drastic differences in N answering the questions. For example, one question (asking employers how PPACA will affect employees) had only N of 62 people answer.  Most of the other answers were answered by either 317 or 617 people out of the 750 sample size. The paper only draws conclusions off the reported knowledge that the small business owners they contacted answered in the survey, and uses it as a definitive on what should be expected in the future from PPACA. There was no relationship discussed between the demographics of the people who took the survey, and how that might have affected the results. Most of the survey was answered by suburban and rural/small town areas (64.6%). This could have impacted the results. Also, the study suggests that the statistical observation of employer contribution shares are produced in a "bell-shaped" distribution curve which would represent a Gaussian distribution (3). In class, we discussed how this distribution would almost never be the correct distribution to use.

How the questions were worded could have also affected the results as well. For Q 10, there were 10 parts that surveyors had to answer. Questions were answered with a strongly agree, somewhat agree, strongly disagree, and somewhat disagree. Examples include: "It will infringe on the rights of Americans," "It will lead to a government take-over of health care," "It will add to federal budget deficits" and  "It will separate doctors and patients." These questions are read as negatives and are not explained. No one likes infringements, take-overs and separations. Especially when it is not further described. All we ever hear on the news is the federal deficit as well. These questions were obviously asked in a very aggressive-sounding tone, and resulted in most of the sample size stating Strongly or Somewhat Agree. Confusion from the surveyors is seen throughout the survey by also most of the sample size agreeing to "positive"questions such as "It will expand health insurance coverage to more people" and "The new option would make employees happier."

There was one other issue that I would like to bring up in relation to some of the statistics presented. "Since enactment, one in eight (12%) small employers have either had their health insurance plans terminated
or been told that their plan would not be available in the future" (Q.5). Curious, I looked at the actual percentage of people that agreed and it was 11.7 %, not 12%. One in eight employers also is defined as 12.5%, not 12%. So I am wondering why they would state "one in eight" when that produces .8% error in total. I also noticed that "one in eight" sounds like a larger population rather than 90 employers out of 317.

There were also other problems that arose within the report and how the survey was conducted, but the general concept of the issues at hand are now realized. The resulting statistics were already skewed from the beginning, and other variables that might have negatively impacted the hypothesis were not taken into account.

 Report: http://www.nfib.com/Portals/0/PDF/AllUsers/research/studies/ppaca/NFIB-healthcare-study-201107.pdf