Data, analytics, and the human condition: life lessons from baseball’s data-driven revolution

The history of professional baseball is, I think, the story of talented, skilled and experienced individuals relinquishing some of their decision-making autonomy to better coordinate their actions with others for the overall benefit of the group. In recent years, data, analytics and the people who effectively evaluate them have played a key role in this coordination effort. As a result, baseball’s history is, I think, a useful case study with which to better understand the value of the broader data-driven revolution that is well underway in many parts of our lives.

In the early days of professional baseball, individual players played as they pleased within the rules and conventions of the game. The manager was able to exercise some control over some on-field decisions because he decided who played each day. He used that authority to bend players to his will, whether or not his will led to success. In some remarkable instances, players were “too good not to play,” and they continued to play as they pleased, succeeding and failing according to their own set of rules. Their natural god-given talent was taken as proof that they could play by a different set of rules or none at all.

Today, because of data and modern analytics, managers and players are now relying on the advice and decisions of people who have often never played the game and who rarely step on the field. At first, these data-driven and analytical outsiders had to persuade the insiders to act on their insights and recommendations. Eventually, the people who control the pursestrings recognized the value of data-driven analysis and decision-making. As a result, the data nerds are now themselves insiders and enjoin rather than entreat. It also seems likely that their influence on the game will continue to grow. For example, data-driven analysis is now influencing player development, which historically, as far as I can tell, has been an unfathomable mix of toxic masculinity, superstition, blind luck, and occasional strokes of genuine and well-intentioned pedagogy.

This turn towards player development is happening in large part because most teams for the most part have now embraced data, analytics, and the people who effectively evaluate them. As a result, the competitive edge associated with the analytics  revolution has been blunted somewhat. For example, even if a clever analyst is able to identify an innovative way to evaluate players, whatever advantage that is gained will be short-lived because player acquisition is a very public activity. Eventually, some other team’s analyst will crack the code underpinning the innovative and unexpected acquisition. In contrast, if a team can use data and analytics to improve their player development, which happens behind the mostly closed doors of their training facilities, to turn average players into good players and good players into stars, there is a huge opportunity for teams to win more games at a much lower cost. They can sign players based on their current value and develop them into higher value players while under the original contract. Crucially, because teaching and development must always be tailored to the student, even if we imagine that an ideal system for player development can be broadly identified and it becomes widely known and understood, there will be plenty of room, I think, for innovation and competitive specialization. Although a handful of very successful teams already have a history of identifying and nurturing talent in-house, the future of player development will probably look a lot like the recent history of data’s influence on player evaluation, tactics, and strategy. Data, analytics and the people who effectively evaluate them can be expected to identify more effective approaches for player development, discredit others, and more accurately explain why some traditional approaches have worked.

I suspect that the analytics revolution has had such a profound impact in baseball only because baseball’s culture was ruled for so long by superstition, custom, and arbitrary acts of authority. This culture likely emerged, I am prepared to speculate, because there were so many exceptionally talented players competing for so few spots. Because all of these players were willing to accept pretty much whatever signing bonus or salary they were offered, if these exceptionally talents guys failed for whatever reason, from a team’s perspective, it didn’t much matter because there were plenty of hungry, talented and cheap guys waiting to take their place. Some guys made it through and some guys didn’t; as far the teams were concerned, it didn’t much matter who made it through or why they made it through — so long as those that did could help to win games. Of course, this model only works when players are cheap. It should come as no surprise that teams have become much more interested in accurately evaluating their players and investing in their development now that signing bonuses and player salaries are substantial and much more reflective of a player’s true contribution to the team’s bottom line. Thanks to collective bargaining and free agency, an economic motive was created that forced teams to treat players as valuable assets rather disposable widgets.

For a fan of baseball — or a fan like me, anyway — one of the unexpected outcomes of a deep dive into baseball’s analytics revolution* is the realization that the action on the field is very much an effect rather than a cause of a team’s success. Evaluating and acquiring players, developing them, motivating them, and keeping them healthy is the key to winning pennants. Yes, there will always be room for individual on-field heroics that help turn the tide of a game or series, but a player is on the field and physically and mentally prepared to make the big play only if a tremendous amount of work has already been done to put him there. And while I will resist the temptation to make the intentionally provocative claim that the analytics revolution in baseball highlights that on-field play is the least important aspect of a team’s success in baseball, it is nevertheless clear that that the data-driven evaluation of all aspects of the game highlights that the managers and players are only one piece of a very large system that makes on-field success possible. At this calibre of competition, with so many talented players on the field, an individual game is probably  best understood as a competition between two teams of finely-tune probabilities working through the contingencies of chance brought about the interactions of those probabilities. This, I think, not only explains the strange cruelties of the game for both players and fans, but it is also a pretty plausible description of the human condition. Once again, even from the cold dispassionate perspective of data, baseball looks like a pretty useful metaphor for life.

If my version of the history of professional baseball is (within the ballpark of being) correct, data, analytics and the people who effectively evaluate them have played a revolutionary role in baseball not because they revealed and reveal previously unseen truths. Instead, they are revolutionary because they broadened the scope of the kinds of people involved in baseball’s decision-making processes and, in doing so, changed how those decisions are made. By creating a more sophisticated, systematic and coherent methodology to measure and evaluate the contributions of players, the data nerds created a tool with which to challenge the tactical and strategic decisions of the old guard, which too often relied on appeals to custom, individual intuition, and authority. In this way, the data nerds earned themselves a place at the decision-making table. Crucially, baseball’s analytics revolution reminds us that people are the true catalyst and vehicle for change and innovation. It doesn’t matter if some new tool unearths previously unseen truths. If the people in charge aren’t willing to act on them, for all intents and purposes, the earth will remain at the centre of the universe.

The history of baseball also reminds us that a group of individuals working together to achieve some shared goal is much more likely to achieve their goal if they relinquish some of their decision-making autonomy in order to effectively coordinate their efforts. This is as true for hunters and gatherers working together to collect life-sustaining berries as it is for disciplined armies fighting undisciplined hordes. Communities, armies and sports teams that rely on an “invisible hand” to coordinate the actions of their individual members simply aren’t as effective as those that consciously and effectively coordinate their actions. We shouldn’t have to look to baseball’s history to be reminded of this simple truth. Unfortunately, western culture’s misplaced faith in the hope that individuals doing pretty much as they please will accidentally lead to the best outcome has created a culture in which we too too often organize ourselves along feudal lines, ceding absolute authority to individuals over some piece of work or part of a larger project, creating silos of silos within more silos. Yes, some leaders have made terrible decisions on behalf of the group, but that is an indication that we need better approaches to leadership not less coordination.

Baseball’s analytics revolution also reminds us that the coordination of individuals will be most effective when it takes into consideration the actual contributions made by each individual and that this assessment requires a systematic and coherent methodology to be effective. Quick judgements about a person’s contribution based on a small or irrelevant dataset is not an effective way to manage a team for success. An individual’s contribution to their team needs to be assessed based on a significant amount of meaningful, relevant and consistent data, which often needs to be collected over a significant period of time. Additionally, the tactical and strategic decisions based on those evaluations must also be subject to regular assessment and that assessment must be made in terms of the ultimate aim of the shared endeavour. Effective team management requires time, a historical frame of reference, and a long-term vision of success. In other words, there is much more to the successful coordination of a team than a rousing locker room speech or a clever presentation at an annual off-site meeting.

Baseball’s increased interest in data-driven player development also reminds us that the bedrock of long-term success for any team is an ability to recruit and nurture talent, where talent is largely understood to be a willingness to learn and evolve and a willingness to mentor and train. On the one hand, people who are set in their ways are unlikely to adapt to the culture of their new team; additionally, as the team and the work evolves, they won’t evolve with it. On the other hand, if they aren’t willing to mentor and train others, whatever knowledge and skills they have and develop won’t be shared. Yes, data and analytics, like any new tool, can create a competitive advantage in the short-term, but the bedrock of enduring success is people who are committed to learning and developing, and a culture and leadership team that supports and rewards their development.

The final insight from baseball’s analytics revolution might be more difficult to tease out because it challenges a habit that is so perennial that it is probably difficult to see it as anything but natural and given. I said earlier that a data-driven evaluation of all aspects of baseball’s operations is bringing into focus the idea that the action on the field is an outcome of a very complex process and that the success of that process is the fundamental cause of success on the field. If every aspect of a baseball team’s operations is designed and coordinated to ensure that the best players can play as effectively as possible during a game, that team is much more likely to succeed against the competition. An essential feature of this model is the important distinction between the activities undertaken to prepare and train for execution and the execution itself. Crucially, there is substantially more preparation than execution, and it is the quality and effectiveness of the preparation that determines the effectiveness of the execution. With that observation in mind, I’m willing to bet that in work, life and play, you and your team (however you conceive it) spend most of your time executing, very little time preparing, and a whole lot of time not living up to your full potential either as an individual or as a team. In theory, it is possible to approach execution in such a way that it becomes a kind of preparation and training opportunity, but, in practice, it will never be as effective as regularly setting aside time for dedicated and focused periods of training, planning and preparation. Essentially, whatever it is you do and whomever you do it with, if you aren’t taking time to train, practice, and prepare, you aren’t going to be as effective as you otherwise might be.

Ultimately, professional baseball is, I think, a useful case study with which to better understand the potential of the broader data-driven revolution taking place today  because of its unique gameplay, specific history, and the financial incentives which rule its day-to-day operations. Because of these factors, the ecosystem of baseball has embraced data, analytics and the people who effectively evaluate them in a way that lets us more easily see the big picture. Because of baseball, it is easy to see that the data-driven revolution is very real but that its potential can only be fully realized if it is the catalyst for welcoming new people and new forms of decision-making into the fold. There are no silver bullets. There are, however, when we are lucky, new people testing new ideas that sometimes work out and insiders who recognize — by choice or by necessity — the value of the new people and their ideas. Unfortunately, this also means that the very people, communities, and organizations who are most likely to benefit from the data-driven revolution and other forms of innovation — those that are ruled by superstition, custom, and arbitrary acts of authority — are the least likely to embrace the people and ideas most likely to make the most of it. And that, I think, is one more important insight into the the human condition brought neatly into focus thanks to baseball.

* If enough people express interest, I can put together a bibliography/reading list. However, any good local library should get you headed in the right direction.

My answer to the ultimate question of life, the universe, and everything: four four through it all

If the mystery of the human condition can be characterized as a kind of puzzle or riddle, the answer and/or punchline can be aphorized, I think, through four banal facts and four mollifying delusions.

I can’t say that anyone will necessarily gain anything by knowing and understanding these facts; nor can I say that they will gain anything by ridding themselves of the delusions.

If anything, I am pretty sure the delusions persist precisely because they are useful to most people most of the time. Whether or not they become more or less useful will be settled by evolution eventually — and not by you or I.

Four banal facts:

  1. Almost all human behaviour is perfectly predictable. Some human behaviour may be random or the result of chance.
  2. Human behaviour and all the products of human behaviour are expressions of the human disposition to allocate resources according to status.
  3. Human society and its organization requires the exercise of power. The risk of abuse is omnipresent. Some will guard against it; some won’t.
  4. We die and will be forgotten.

Four mollifying delusions:

  1. Humans have free will and are masters of their own destiny.
  2. The truth will set you free.
  3. Democracy is the worst form of government, except for all the others.
  4. Immortality is possible.

That’s it; that’s all. If you like my solution or enjoy talking about the puzzle, let’s start a club. You bring the (alcoholic) punch. I will bring the (vegan) pie.