First things first:
>Identify the research question<
Its amazing how many articles you find online that do not clearly do this. They might mislead you in exactly what they're inferring especially if the article is biased. They might say this and this about this, without even telling you what it is they're looking for or why it is a significant question to ask.
>Identify the type of study<
Quantitative or Qualitative? Or perhaps a little bit of both? I once encountered a study that was mostly qualitative (details, more then numbers or yes/no questions) but they converted these details into numbers that didn't quite represent their full research. I think this goes along with the whole 'its math - its arithmetic - its simple - its believable' persuasion tactic that you see in many politics today.
>How was the sample size chosen?<
Does it even state where they got this sample size? Here are some questions to ask. For example:
"We interviewed 500 students from all across America."
What kind of students? What does 'all across america' mean? How old were these students?
"We interviewed 500 students at UNM"
Is that an accurate demographic for the question you're asking?
"We interviewed 500 students and faculty members at UNM"
Are students and faculty on the same ground for the demographic the question is after?
>Have the authors taken both practical significance as well as statistical significance into account?<
This one can be a bit tricky! While there may be a numerical statistical concern with a certain problem, does it actually make practical sense to implement a change or make a choice? For example: If you are comparing two ways to do something, there is statistical significance in the amount of time spent doing laundry one way then they other, but is it a practical way. In other words: If carrying laundry on my head saves me time, then is it always the better option?
These are just a few. If you want more you can visit here: http://www.ma.utexas.edu/users/mks/statmistakes/readers.html
Thanks so much for this Rebecca, it's helpful to have these as reminders as we approach the readings or our own blog posts. I think your first point of "identify the research question" is a really crucial idea to keep in mind. It's very easy to get lost in the embellishing details thrown into articles or studies that consequently distract from the main question at hand. In such instances, as you point out, it's important to remember to ask these sorts of "so what?" questions to pinpoint what the data is supposed to represent, and how it's being used, either successfully or unsuccessfully.
ReplyDeleteThanks, these are great tips! Especially when the media are reporting statistics, it can be difficult to find the what question was asked in the study. The media like to publish findings and will usually lead an article with a general statement of the practical conclusions because this is the most interesting part. They may later on give details about a particular study or several studies. This leaves it up to the reader to look for the original research questions, methods of data collection and methods of analysis. Sometimes these details are not given in the news story, but usually there is a reference to the original study.
ReplyDelete