In my Research 2 class (in graduate school for social work) we have reached the dreaded world of statistics. There’s a lot of math involved in this process, and even more incomprehensible data-to-Math-to-Greek-to-Computational-Tables-to-English translations. And what I’m realizing is that a lot gets lost in the translation from reality to statistics.
It’s not that I think research is a waste of time. It matters. But not enough time is spent on elucidating the data, and remembering the anecdotal evidence that makes up the data. Anecdotal evidence (or individual stories told to the researcher) is often considered unreliable, but masses of data, detached from its origins as anecdotal evidence, is considered reliable. We end up taking a lot of valuable information, and turning it into numbers and graphs, and forgetting where the data came from in the first place. People.
As we have discovered over the past year in the United States, polling is only as valuable and legitimate as the questions asked and the answers recorded. If people are asked the wrong questions, or distrust the person asking them, then the data that results will be incomplete, if not completely wrong.
If we looked at certain data about Butterfly, like: heart disease, diabetes, aged twelve out of a 13-14 year expected lifespan, few teeth and those that are left are not good, persistent cough – you’d think she was at death’s door, and miserable. But she has the biggest smile in the world, runs like the wind, comforts her sister, loves to be petted, loves food, licks me to death, and I could go on and on. You wouldn’t know any of that if all you asked was “What’s wrong with Butterfly?” or “Describe Butterfly’s health.”
“What’s wrong?”
“Absolutely nothing!”
The reliance on big data, and mass polling, has developed (as far as I can see) as a good faith effort to get a sense of what’s going on with everyone, instead of just with the easiest people to access. A doctor who sees a hundred patients on a regular basis may have a very good sense of the health issues of those hundred patients, and no clue whatsoever about how her patients fit into the patterns of the population at large. They may be anomalies – because they can afford her fees, live in a certain geographical area, and have certain specific symptoms – or they may be average, she can’t know. That doctor needs access to a wider swath of the population, in order to put her patients into better perspective. But what is the quality of that data? Who chose the questions to ask? What biases were at work? Which questions, that she would have known to ask based on her experience, were left out of the questionnaires filled out by all of those anonymous people that she cannot call and follow up with?
Recently, I heard about research done on the question of abortion. It’s a thorny area to begin with, but the way the polling is done can make it even more confusing. If the question is, do you support abortion? Or, would you have an abortion yourself? A lot of people will easily, and quickly, say no. But if the question asked is, do you think abortion should be legal? Many of those same people will say yes. It turns out that, on this specific question, people have different opinions about what they themselves would do, than on what they think others should be able to do in their own lives. The people setting up the poll would need to understand that gap in order to ask the right questions and really understand the data they are receiving.
This kind of gap can exist on any subject, and it requires open-minded researchers with a willingness to question the data and look deeply at their questioning process. Without those extra steps, the data can profess things that are not actually true, or that are, at best, incomplete.
If I asked Cricket if she prefers peanut butter or chicken, chicken would win every time. And if that were the only question asked, you might come to the conclusion that she doesn’t like peanut butter at all – especially if you could see the way she sneered at the peanut butter on her way to ripping the chicken from my hand. But the fact is, she loves peanut butter. She will take any medication offered, as long as it is covered in peanut butter. But we didn’t ask her the right questions, so we never found that out.
When we hear about study results in the news, especially on TV or from the mouths of politicians, we rarely hear about the context of the study, or the methods used. We are given simple numbers, or better yet, bar graphs and pie charts, to make the point very clear. But once a study’s results have been translated into numbers and graphs, our ability to determine for ourselves the validity of the study’s methods, questions, and analysis, disappears. In fact, people rarely take the time – or even get the chance – to read through a full study report, even though researchers put a lot of effort into examining and going into detail about the choices they made, why they made them, and where they may have gone wrong.
What if, after hearing the results of all of these polls and studies, and staring at bar graphs and pie charts and news anchors for hours and hours, we come away believing that we know each other perfectly, and can therefore dismiss each other? And what if we’re wrong?
“We’re never wrong. Right?”
“What a relief.”
rachelmankowitz 