But the Research Says....
How often do we as coaches revert to “this is what the research states” when defending a point? I know, I know, we all do. Me included. If you’re like me you never stop asking why, until you seek an “answer.” Yes, I understand the answer is temporary as superior explanation is an iterative process, but the point is passionate exploration. I listen, and read well outside the scope of physical performance, but as I’ve aged, the more and more I believe it directly relates. One such podcast I listen too frequently is EconTalk. A fantastic episode in 2018 was that of Dr. John Ioannidis titled Statistical Significance, Economics and Replication. A masterful conversation ensued which led me to one of John’s papers titled: Why Most Published Research Findings are False. Important to note, Dr. Ioannidis was referencing epidemiology and medicine NOT sport science, but what better measuring tool to use as a proxy for gold standard.
Here are a few highlights:
“Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.”
Ioannidis outlies four corollaries pertaining to the probability that a research finding is true (or false):
Corollary 1: The smaller the study conducted, the less likely it is to be true. This has to do with the power of the study (1- Type II error rate). Moreover, power can be thought of as the probability of capturing an effect when in fact it does exist.
Corollary 2: The smaller the effect, the less likely it is to be true. Power is related to effect size. Effect size is the magnitude of change, or how big the effect is.
Corollary 3: The greater the number of tested relationships, the less likely the research findings are true.
Corollary 4: The greater the flexibly in study design, definitions, and outcomes, the less likely it is to be true.
“The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field.”
So, how does this relate to sports performance? Many of the studies in the medical literature are randomized controlled, and blinded studies, this typically does not happen in our field. Certainly, there are other challenges facing research in sports performance such as sample size (a sample of 25 professional hockey players is quite large as not many people make a living playing the game), bias design, confounding, and limitations of observational findings.
Perhaps the answer to these problems is pre-registration? This has been something Franco Impellizzeri has spoken about for some time. When you preregister research, researchers specify their plan in advance of the study and submit it to a registry. Preregistration separates hypothesis-generating (exploratory) from hypothesis-testing (confirmatory) research. It also prevents P-Hacking, HARKing, flexibility in study design, and reduces researchers’ degrees of freedom after the fact. In addition, replication papers attempting to answer the same question using the same methods are also extremely important. The problem is, most of these papers will not be published because journals aren’t overly interested.
Our field is extremely noisy as we study complex beings. The cumulative approximation of explanatory knowledge will continue to move the needle forward. What the research says is important...we just need to do our best to find the right kind.