All Models are Wrong and Only Some are Useful
It is now #week3 of my doctoral adventure and I have read no less than 17 academic articles. In that time, I have scribbled in the margins of at least 4 of those articles the words "ALL MODELS ARE WRONG AND ONLY SOME ARE USEFUL!!!"
I will not forget the first time I heard those words in my graduate statistics course circa 2014. I chuckled to myself and thought "what a funny thing to say about a little curve that took us hundreds of observations and some intense calculating power to create. If it isn't even right, why bother?"
Little did I know how much this phrase (and even a statistical viewpoint in general) would have on how I think critically as an adult. This phrase means what it says and demonstrates one of the fundamental issues with research in both educational technology and the social sciences generally. When you study humans, each has their own diverse viewpoint on any given subject. As helpful as it is to bring all of those viewpoints together, it will never be able to really capture the data it is trying to represent. For example, say you want to know if people like your new t-shirt. You set up a Likert scale and ask your friends to rate it between a one and a ten. At the end of the day, the mean score is 7.5 and you determine that your friends did indeed like this shirt. Next time you wear it, one of your friends remarks "eww, why are you wearing that again? I hated that shirt." You are confused... the data told you they liked it! Surely the data was wrong then!
This idea comes in many forms; most recently I've been hearing about it as "generalizability." When you zoom out and assemble the micro data into macro data,the conclusions drawn go toward the generalizability of whatever is being studied, whether it be the effectiveness of a drug or if your friends liked your shirt. This manifests in many ways in research articles:
"The general rule of teaching is that general rules don't help very much." - Labaree, D., 2003
"Those same critics argued that to individual, theories of motivation cannot realistically
apply to each single employee; however; they are useful for identifying the main overall ways in
which people are motivated." - Loiseau, J., 2011.
The frequent appearance of sentences like these make me think that we need a better understanding of data and data generalization. If readers of articles looked at statistics the way they were meant to be seen (as a macro image of intensely varying micro data) it wouldn't be necessary to restate this in so many papers. I understand the importance of bringing this to the attention of the reader, but at a certain level it is disappointing that it must be brought up so often in order to properly evaluate a study.