A Fishing Expedition: Measuring Noise
For the past year and a half, I have volunteered as an Assistant Coach for my nephew’s hockey team. He recently completed his 8U season, and his now preparing for his 10U select tryouts. At a recent coaches meeting, a board director was eager to mention that the organization recently partnered with a tech company. The goal was to “objectify” performance development by attaining “hard numbers.” Each player would wear an RFID bracelet, “beams” would be used for “timing”, with the goal of collecting information and sharing it with other organizations creating an age-appropriate longitudinal data stream. “Data” would be collected twice/year. “Wow,” one would think. “This is impressive.” When one critically thinks about this in more detail. It’s simply a fishing expedition hooking noise as the catch. We do this many times in sport science. Tech companies love it as it improves their bottom line. Here are a few questions this individual should think about prior to this “investment:”
What is the organizations definition of development? Is it skating speed? Hockey sense? Compete level? A positive hockey experience? Certain things cannot be measured.
What are the tests? Are the tests reliable? Have they been tested-retested with the same population? If so, what’s the variability? If so, what population?
Re: population: what’s the learning effect? We’ve all seen this in action as coaches. Young Tommy “miraculously” adding 3” on is vertical jump in 2 weeks. Tommy didn’t do anything but learned the skill of jumping more effectively.
Frequency of “testing.” Is speed improved, or did Tommy gain 15lbs and 2” over the course of the hockey season?
Who’s administering the “test?” Same coaches, tech companies? This may lead to drastically different results.
What does “success” look like? How big? Expectations?
Who communicates results that aren’t favorable?
Why are the results not favorable? Who’s fault?
This seems to be happening more and more in the circles I frequent. If you’re a “numbers guy,” that’s fine, but understand the limitations of measurement and think critically before tech procurement. “But something is better than nothing” some may say. My response. No, it’s not. Nothing is better than something. Otherwise, you’re on a fishing expedition, and more than likely, that “something” you’re reeling in is noise.
