Generalizability of Coaching Methods

Research-based and evidence-based are mostly just buzzwords.

Two opposing viewpoints dominate basketball discussions, and specifically skill development. Theorists argue their methods are evidence-based or research-based and disagree with any methods that do not conform to their theories. Empiricists rely solely on their own experiences and perpetuate the drills and teaching from their playing and early coaching days. The biggest problem is not deciding which side is right or determining which side to join, but the inflexible thinking. Neither side considers generalizability.

Generalizability is important in research, and is essentially the degree to which a study is applicable in different environments. For instance, I used junior-college women’s basketball players when I studied the hip turn and drop step. I wrote in the conclusion, “The purpose of this study was to determine whether female college basketball players changed directions more quickly using a DS or a HT in a task similar to playing defense in basketball” and “The findings from this study suggest the HT may be a quicker method of changing direction than the DS for female basketball players.”

The conclusions did not suggest anything about men’s players or younger players or athletes in other sports. We can suggest nothing should prevent the findings from transferring to men or all basketball players, but we need to replicate the study with different populations to state this scientifically. From a non-scientific standpoint, one can argue the study is generalizable.   

The same non-scientific generalizability is used when coaches provide content, whether videos, articles, books, or coaching clinics. We assume the information generalizes to different populations. When an NBA coach demonstrates a drill with NBA players, we assume the drill generalizes to WNBA players, college players, youth players, and more. We believe in a certain universality; as they say, basketball is basketball. Basketball may not change, but the specific environments change: Players’ skill levels, backgrounds, experiences, previous coaching, and more; the coaches’ playing styles, systems, philosophies, and more; the competitive levels, competition schedules, practice durations, and more. Do the disparate environments affect the generalizability?

There are very few concrete answers in sports learning and performance because so many factors affect each process. Nobody plays basketball perfectly; there is always room for improvement, which makes many statements unfalsifiable, even when they are unlikely to be absolutely correct. It depends is a more accurate answer, but we dislike ambiguity.

I engaged in a discussion about direct instruction with a Theorist. I lean toward some beliefs of  theorists. I often advise, “Say as little as possible, but instruct as much as necessary.” I posed the question about an endgame situation. When I trail by two with five seconds left, I draw a play to generate a specific shot for a specific player. I tell players where to go and what to do. I do not encourage them to perceive the affordances in their environment. He argued they would perform better if I did.

Here’s the thing; it is entirely possible he is right, although to my knowledge he has never been in such a situation personally and therefore speaks purely theoretically, but it is impossible to know. We cannot exist in two timelines. We cannot A/B test because the first experience always affects the second. A different team, different players, different environment and more would confound the comparison or results.

Unfalsifiable claims are a characteristic of pseudoscience. Unfortunately, Empiricists engage in similar arguments. I once wrote that my team finished top 10 nationally in three-point shooting percentage, three-point field goals, and free-throw shooting percentage without engaging in traditional form shooting. An Empiricist told me we would have shot even better had we used form shooting.

Again, the claim is unfalsifiable. Unless one shoots 100%, improvement is always possible. We cannot replay the season with more form shooting. How much? Who knows? Of course, what about all the programs invested in form shooting who were not top 10 in multiple shooting and offensive-related categories? Does their poor shooting invalidate form shooting?

Sports are complex. Players are complex. Skills within sports are complex. Attributing success to a single factor or even several factors is impossible. Instead, as I learned in my research methods courses, research can prove what something is not, not what it is. My study did not prove the hip turn was the best movement technique, only that the drop step was not. These are related, but not the same. As Nassim Nicholas Taleb wrote in Antifragile: Things That Gain from Disorder, “Knowledge grows by elimination more than by addition. What is wrong cannot turn out to be right as easily as what is right can turn out to be wrong.” The drop step is not faster than a hip turn, but we may discover the hip turn is not faster than another method.

One key from Evolution of 180 Shooter: A 21st Century Guide was the multifactorial nature of shooting development, which is only one element of overall basketball performance. Trainers, content creators, and others focus on the physical, the things we see easily in sound bites and video clips, which generally are the drills, possibly coupled with some instruction and/or feedback. The trainer shows a clip of a practice session and highlights an uptick in shooting percentage or scoring average, and we are supposed to conclude the trainer or the drill or the instructions caused the improvement.

I quit private training because, in part, I watched one player excel and become an NCAA Division 1 conference record holder, while a second player who worked out alongside him did not have any offers beyond the local (non-scholarship) junior college. The other parents blamed everything but my training; they blamed his athleticism, work ethic, desire, confidence, and more. His parents blamed his coach, the system, and his teammates not getting him the ball. Nobody questioned me or my training. Except me.

If two players at the same position from the same school with the same coaches and same height had such disparate outcomes, how was I responsible for one’s success, but not the failure of the other? Why did my training help only one? If the results were due to individual differences, and one was destined for success due to genetics or individual qualities, and the other never stood a chance for the same reasons, what was the point? Success as a trainer would be the same as coaching: Recruit the talented players with the right mix of genetics (height, speed) and individual qualities (determination, focus, grit) and take credit when they achieve success.

I wrote about several players who improved during their junior-college careers in Evolution. I cannot credibly identify the specific causes of their improvements. Did one improve her shooting or simply strengthen a nagging ankle injury enabling better balance? Did one improve her shooting or derive confidence from her coach finally believing in her and encouraging her to shoot as much as possible? Did one improve her shooting or simply thrive when moving from shooting guard to point guard? Did percentages improve because of shot selection? Ball movement? Better practices? Confidence? Team camaraderie? How does one separate the various factors? There is no way to isolate a single variable and test; they are enmeshed.

They improved. Their percentages before and after demonstrate performance improvements.  However, the specific path to the improvements is uncertain. I was involved in most of the basketball-related activities as their head coach, skill development coach, and strength & conditioning coach, but even those do not capture all of the factors related to their improvement and performance. Possibly, their off-court friendships enhanced their development and performance; maybe it was the post-practice shooting games. Possibly, they ate better or slept more or stressed less.

Performance is impossible to simplify to one or two factors and attribute improvement to a single drill or coaching behavior or playing style. Macnamara and colleagues (2016) found “deliberate practice accounted for 18% of the variance in sports performance” and “only 1% of the variance in performance among elite-level performers.”

I can transport many of the same ideas, concepts, drills, strategies, philosophies, behaviors, and more when moving to a new environment. I can replicate my behaviors and activities with a new group because I ran the practices, skill sessions, and weight room. I can tolerate mistakes similarly and attempt to build confidence in similar ways, but ultimately every environment is different: Personalities, backgrounds, habits, previous experiences, competitive successes, expectations, and more affect improvement and performance.

I have reasonable confidence I can duplicate the improvements with new players and teams when I control so much of their basketball experience. I am less confident in situations with less control, as with the players I trained individually.

Overall, my teams generally improve in shooting throughout a season and from one season to the next. However, I cannot distinguish a single factor causing the improvements or most of the improvement. There is no magic drill, instruction, or shooting style. It is the cumulative environment and philosophy.

I am not an Empiricist or a Theorist. I do not trust only my experience as a player and assistant coach, nor do I fully trust research. I am confident in my thoughts, ideas, and philosophies because I have coached in many different environments. In a sense, they have been shown through demonstrated success to have generalizability.

I write about my experiences and beliefs; I do not write to convince anyone else of anything. I am unconcerned with other teams and coaches. If my writing helps a coach or player, great. If you disagree, I do not care, although disagreements and criticisms often cause reflection and help to refine my thinking and ideas, which is the reason most ideas from my books appear in my newsletters first. But, I am not invested in trying to convince anyone of anything.

I have coached beginners and professionals. I have worked as a private trainer, a head coach, and a strength & conditioning coach. I have coached boys and girls and men and women. I have coached non-English speakers. I have coached eight-year-olds and players who were older than me. I have coached players for a matter of weeks and over the course of nearly ten years. I have a breadth of experiences across the lifespan of players. I tend to have similar successes and strengths with each player and team, and I write based on these experiences.

Generally, I trust concepts that work in multiple environments, whether different ages, genders, sports, and more. People send the tweets of the Timberwolves playing tag in training camp. I suggested roughly a decade ago that one reason NBA players did not improve their dribbling skills like children (an NBA coach’s opinion, not mine) is because of their risk-adverse attitude toward practice, and they would improve more if they played tag. Numerous coaches scoffed at the idea and said I was out of my depths because I did not train NBA players. Now, here we are.

I believed tag would help because I have seen tag work with various ages, genders, and skill levels, and the basic underlying concepts are logical. I do not need to wait for an Empiricist with more credibility (NBA coach) or a Theorist to fit tag into a new theory to understand how and why tag helps players. I do not believe in tag because it is research-based or evidence-based. To me it is self-evident, and I have generalizable experiential knowledge to support the common sense.

There is no single best way or one size fits all approach. Experience and research offer some ideas and guidance, but also are not flawless. Often, we misattribute our or anther’s success, simplifying success or skill to a single factor. Research tends to simplify because it is impossible to study every factor related to success at once, so we compare two things to find which is better, but even the one that is better is only better on average, not unequivocally, and is not the only related factor. The hip turn may be faster than the drop step, on average, but there are many other factors related to defensive/change of direction speed, defensive success, and more.

I trust my experiences because of their generalizability. My experiences are my own, and in one sense are also a subject of one and biased. However, my beliefs, instructions, and philosophies have generalized to many different age groups, genders, skill levels, experience levels, and more. My authority is not because I work with professional players, and therefore have credibility, although I have not tested my ideas and instructions with other groups. Instead, my (modest) authority is derived from the my broad experiences. I have had teams near the top of the league in shooting in youth, junior college, and professional leagues, and I had little control over the roster construction in all but two seasons. I inherit teams described as poor shooting or lacking shooters by the club’s management, and by the end of the season, we are among the best, time after time. This is generalizability.

I have not won an NCAA or NBA championship as a coach or player. I am not the coach or writer to read if one bases credibility by stature or championships. I am not interested in specific theories or dogmatic approaches. I am not the coach or writer to read for science lessons. I am neither a Theorist, nor an Empiricist. However, I possess a varied background and experiences different than most, and these experiences provide a different form of credibility, especially when speaking about generalizing lessons across genders, ages, levels, and skill groups.

I am a flexible thinker who has evolved over time, questioned orthodoxy to find better ways, grown knowledge by elimination, and solved individual and group problems in a variety of environments. I share my work to inspire discussion to further refine my thoughts. I am not beholden to a certain theory, methodology, or mentor. I search only for better ways to help players.

Macnamara, B.N., Moreau, D., & Hambrick, D.Z. (2016). The relationship between deliberate practice and performance in sports: A meta-analysis. Perspectives on Psychological Science, 11(3), 333-50.