It turns out that even very intelligent human beings are very bad at making optimal strategic decisions in a world of dynamic complexity.
For well over twenty years, the management gurus at MIT’s Sloan School of Management have been showing just how badly sharp undergraduates, brilliant graduate students, and experienced executives can be at making decisions with even a simplified model of the real world. When asked to participate in the “Beer Distribution Game,” even the brightest among us find themselves frustrated, confused, and most importantly, wildly wrong.
The game is set up as follows. Participants are asked to divide themselves into four groups: retailer, wholesaler, distributor, and the factory, and told to minimize costs. The retailer is presented with customer demand for beer. Each team is eager to sell off its inventory because buildup costs money. After four weeks of depleting inventory and making orders to replace that inventory, the consumer demand spikes upward. At that point, chaos reigns.
In most cases, the participants are not allowed to see the basic “board” state, but here’s a good picture of what’s going on.
The retailer has a two-week shipping delay in receiving his orders from the wholesaler who has the same delay in receiving from the distributor and so on. There is also a two-week delay from orders place to orders made. Most importantly, there is an even longer production delay as raw materials are shipped into the factory and processed. Although demand is held constant at 4 cases of beer in the first four weeks and then 8 cases for the remainder of the game, the teams of wholesalers, distributors, and factories sketched a pattern of perceived consumer demand with huge amplitude fluctuations. The end result was that average costs were more than ten times greater than optimal.
When it was revealed that customer demand was in fact constant, many voice disbelief. According to Professor Sterman, “many participants are quite shocked when the actual pattern of customer orders is revealed.”
These sorts of studies are well-known and easily replicated in business schools.
In another study, subjects responsible for investing money into a large economy experienced a boom-bust cycle with massive losses following heady profits, even though consumer demand was constant. Average costs in that study were 30 times greater than optimal.
These experiments go beyond business school pedagogy. Participants in simulations ranging from putting out fires to treating medical patients show that participants often let their headquarters burn in spite of best efforts to put out a fire, or let patients die while ordering more tests (Sterman 304), all for the same reasons.
Whether it is managing an ecosystem or controlling a factory, humans are bad at managing complexity. In fact, it’s why Magic is so compelling.
Magic models a system with a very high level of dynamic complexity. From the metagame to the operation of a deck, dynamic complexity abounds. It’s not just that metagames are constantly evolving dynamic systems that often resemble the complexity of a small economy; decks are complex systems as well. Each of the various parts has internal interactions that produce particular outcomes. For example, Fetchlands and Sensei’s Divining Top interact in a particular way to help you see more cards of your library than you would without. To take another example, the consideration of whether to include a card necessitates the removal of another, with a multiple feedback loops. Adding more mana to increase mana consistency might ultimately hurt a particular matchup because you removed a silver bullet to make room.
Here is an image modeling just a fraction of the complexity found in Legacy Flash decks from Grand Prix: Columbus, and explanation can be found here.
Even a simplified iteration of Magic, such as the Elves versus Goblins holiday gift pack, poses complex systems questions with so many possible interactions and lines of play, including mulligan decisions (by the way, I urge you to pick one up if you haven’t — it’s loads of fun).
This is also why Magic is such a skilled game, and yet so many Magic players, including the best, are so “bad.” Magic is a game that is won by the player who makes the fewest mistakes (in the broadest sense – not just within games, but also in deck selection, deck tuning, sideboarding, mulliganing, etc).
Worse for human capacity to make sound decisions, other experiments have shown that the greater the complexity, the worse people do relative to potential. (Sterman 305). In an experiment monitoring the management of a realistic product market, including variables of product life cycle, marketing, capacity acquisition delays, and original and replacement demand, after five trials the subjects demonstrated attempts to manage the complexity met with utter failure. A simple naïve strategy that did not engage in any strategic or game theoretic reasoning outperformed the subjects in 90 percent of cases (and that’s not a good thing). Worse still, after 50 years of simulated experience, the naïve strategy still outperformed the managers 83 percent of the time. So, it’s not just that we make bad decisions, it turns out that we have trouble learning why we make bad decisions.
There are several key reasons. First and foremost is misperception of feedback. Think about the beer distribution game. By the time spike in customer demand reached the factory, how was the factory manager supposed to interpret that data? It had already been filtered through three other teams. There was no obvious way that the factory manager could attribute this spike in orders to consumers any more than it could have been a management error created mid-way through the supply chain. The original cause of this event could be hidden behind many other intervening events. Perhaps the wholesaler simply failed to keep up inventory and now is forced to restock.
In Magic, there are many variables that interact to produce outcomes. How are we to properly disaggregate all of the variables that come into play to isolate the cause of a particular outcome? In Chess, there is only one variable: your moves. You can replay your fully annotated game later on, examining other, superior, lines of play. After no more than a crumb of analysis, your mistakes become bloody obvious. And while you can do that in Magic as well, there are other, innumerable factors at play: your metagame predictions, your deck choice, your deck tweaks, your sideboard, your sideboarding plans, your mulligan decisions, and so on. How do you isolate the cause of a game or match win to any one of these factors over another?
In Vintage, it is very easy to say that “Yawgmoth’s Will won me that game,” as after playing Yawgmoth’s Will, you replayed the entire contents of your graveyard, including Black Lotus, Dark Ritual, and Ancestral Recall, and then tutored for Tendrils of Agony and killed your opponent.
Yet, that is merely the most proximate “cause” of victory (aside from the Tendrils itself, of course). In complex systems, however, cause and effect are often distant in time and space and actions have multiple effects with delayed consequences (Sterman 308). In a provocative essay, Patrick Chapin argued that Ancestral Recall is the best card in Vintage because it “wins” you the greater number of games. While his argument is slightly different, one way of interpreting such an argument would be that Ancestral is the “root” cause of a series of events that follow and lead to Tendrils. However, this fact is obscured by the intervening causes that seem to “supercede” the playing of Ancestral Recall. The problem is that interpreting feedback is hard. We have no easy way to separate causes. So we tend to select the “cause” that appears to be the most obvious to us. And studies show that once a sufficient cause has been found, people stop looking for other reasons. (Sterman 308). What appears to Beer Game Distributors as actually a huge fluctuation in consumer demand may actually be something else entirely.
It could be that your opponent mulliganed, sideboarded poorly, make tactical errors, lost the coin flip. It could be any one of these or all of these. We are inundated with information in Magic, and the studies and experiments of those involved in dynamic systems theory suggest that, if anything, we are probably wrong in our search for causes.
This leads to a second major factor that explains why we make so many bad decisions. We tend to attribute unfavorable outcomes as a consequence of “exogenous factors” — or factors outside of ourselves. In the Beer Distribution Game, subjects overwhelmingly blamed their own poor performance on what they saw as “perverse” patterns of customer demand: “the customers increased their demand, encouraging them to order additional beer, then pulled the rug out just when the tap began to flow.” (Sterman 22). Few of these students or managers ever suggest that their own decisions were the cause of the behavior they experienced, yet we know this is the case. Consumer demand was constant. It was the behavior of each team on each other than produced what seemed to be wild fluctuations in consumer demand.
The same is true in Magic. How many times have you heard someone say that their opponent just “got lucky?” Or that they opened a “bomb rare” and that’s why they won?
A third reason for faulty decision making is that we are bad at recognizing time delays and accurately incorporating these delays into our decision-making. This is similar to our understanding of Yawgmoth’s Will versus Ancestral Recall as a cause of victory. Human beings seem wired to encode and remember things that fit a narrative or a story. We tend to focus on the interesting, memorable, noteworthy ones, rather than on the most common ones. This means both that we tend to overemphasize dramatic wins and losses that have happened instead of remembering the cards we play every game (hence Cunning Wish sideboards that have twice as many instants as they really should), and that anything we don’t make an effort to fit into the narrative of the event gets conveniently forgotten about or misinterpreted.
Time delays in management modeling present most of the problems that caused the Beer Distribution Game to fall into anarchy. Teams at various points in the exercise did not properly account for how those delays would affect them, especially in the midst of other constantly fluctuating variables and decision-making. Because of the way that we tend to think and remember information, we place undue emphasis on events that seem closely connected in time rather than in fact. Hence, the emphasis people place on Yawgmoth’s Will as a trump rather than on the cards that led up to Yawgmoth’s Wills resolution, such as Ancestral Recall.
Time delays also present problems in Magic analysis in other ways. Metagames are constantly evolving. A month after winning the Vintage Championship, I was confronted with the question of what to play at SCG Indianpolois. Without reflecting on possible metagame changes or necessary tweaks, I decided to play, card-for-card, the same list with which I won the Vintage Champs. One of my reasons was that I won the GenCon Prelim tournament with the same decklist. If I could win two tournaments in a row, surely I shouldn’t change my decklist.
In an experiment with MIT and Harvard economics graduate students, students were asked to manage a firm in an experimental economy under various conditions (Sterman 306). The “high complexity” condition included production delays (as in the Beer Distribution Game) as well as information on production to total demand. Three different price options were presented. Although these students were to be paid in proportion to their profits, they performed far from optimal. It turns out that rather than respond to all of the feedback and try to figure out what was really going on, the students tended to simply forecast using past values and extrapolating past trends. Ironically, subjects spent less time making their decisions in complex markets than a simpler experiment with no dynamic complexity.
My approach to deck selection and deck tuning at SCG Indy was the same. I had won two tournaments with the same list, why change? Well, it’s not that I performed poorly per se. And I definitely felt that I had a bad matchup in Feinstein and unluck in running into the Stifle in round 2. But what I hadn’t realized was that I made my biggest mistake in not properly adjusting to the dynamic complexity of the metagame. What I failed to appreciate was the time delay.
The Vintage Prelim tournament I won wrapped up at 4 in the morning the night before the Vintage Championship. No one, except my opponent and whoever was silly enough in his crew to stay up that late would know that GroAtog won the 50+ man prelim tournament. There could be no metagame change between Friday and Saturday night, aside from their own tweaks based upon what they saw in the swiss, which isn’t really enough information to change the metagame itself. But a full month after the Vintage Championship, surely people will have made major adjustments. And indeed they did. Just like the economics students, in the face of difficult and sometimes inscrutable complexity, I opted for the simple decision rule: just extrapolate past performance in the future and play the same deck, to my detriment.
To acknowledge all of these obstacles to sound decision-making isn’t to say that there is an easy solution to it. We are wired in certain ways for the survival of our species. We are wired to categorize (create schemas) and react faster than the rational-conscious mind can operate. If we weren’t, we wouldn’t have been able to escape predators that might devour us, or even type efficiently on keyboard.
We are also not computers. We are poor at modeling complexity. Our leaders often make bad decisions and our top minds do the same. Whether it is a military escapades or the setting of interest rates, human beings are far from perfect. Education, IQ, and experience don’t seem to help nearly as much as we’d like. However flawed we are and will likely always be, the first step in even beginning to overcome our measured limitations is to acknowledge them and better appreciate them. The very best Magic players are far from perfect; they’re only a little bit less likely to make the same mistakes as everyone else. And in Magic, that’s good enough.
Have a happy new year and wonderful holiday. I’ll see you in two weeks,