Where Good Ideas Come From
Steven Johnson, 2010
Steven Johnson reflects on the last 400 years of world-changing ideas in an attempt to identify key environmental catalysts human innovation. While we often attribute breakthroughs to the lone genius — Darwin, Galileo, Newton — a core argument in the book is that solo epiphanies are exceptions, not the rule, in the history of human innovation.
Instead, good ideas that end up changing the world tend to be…
found within the realm of the adjacent possible. The right idea without the right social, intellectual, and technological prerequisites struggle to find success in its own time. It is shelved only to be rediscovered decades later.
a cumulative product of “slow hunches” that occur over long timeframes (sometimes decades)
the result of existing technologies from an unrelated domain repurposed — or exapted — for use in another
formed collectively as ideas propagate through liquid networks
riddled with errors that may lead to serendipity
built on agnostic and permissionless platforms — aka “on the shoulders of giants”
Johnson dedicates each chapter of the book to each of these key ingredients (bolded above) and their role in facilitating human innovation, drawing on parallels on how nature too, leverages these catalysts to innovate in the process of evolution.
Overall an enjoyable and insightful book driven by stories of how various individuals came to make groundbreaking technological or scientific breakthroughs. What I found most interesting is the underlying theme of collective intelligence, and how the majority of world-changing ideas weren’t actually standalone strokes of genius, but was a result of the right individual being plugged into a network of ideas. This points to the potential of what we can do to facilitate these connections better with modern technologies like the internet, online knowledge management tools, and AR/VR. The book also paints an optimistic picture for those with infinite curiosity with a propensity to go down many different rabbit holes. “Chance favours the connected mind”.
The rest of this will be less a review and more a summary of insights…
The Adjacent Possible
The “adjacent possible”, coined by biologist Stuart Kauffman, is the idea that at any given moment, progress can only make progress in certain ways. In the context of technological innovation, the adjacent possible is limited by the tools and knowledge are available to us.
Think of it as a house that magically expands with each door you open. You begin in a room with four doors, each leading to a new room that you haven’t visited yet. Those four rooms are the adjacent possible. But once you open one of those doors and stroll into that room, three new doors appear, each leading to **a brand-new room that you couldn’t have reached from your original starting point**. Keep opening new doors and eventually you’ll have built a palace.
Given N objects, there are 2^N ways to pick a subset of them. In other words, 2^N “innovations” lie in the realm of the adjacent possible. Over time this can ultimately result in a combinatorial explosion if, with every step, we can build upon the realised potential that previously was in the adjacent possible.
The adjacent possible is a kind of shadow future, hovering on the edges of the present state of things, a map of all the ways in which the present can reinvent itself.
The history of life and human culture is a story of a gradual, relentless probing of the adjacent possible. In gradually turning the adjacent possible into reality, we increase the diversity of what can happen next. Innovative environments are good at helping their inhabitants explore the adjacent possible by exposing them to a diverse sample of parts—mechanical or intellectual—and encouraging novel ways of recombining existing parts.
Biochemistry - “the original innovation engine on earth” - has two essential properties:
the capacity to make new connections with as many other elements as possible
a “randomising” environment that encourages collisions between all the elements in the system
Carbon is crucial in nature because of its role as a connector. Water is the randomising environment that facilitates its flow. Together, they unlock the possibilities of life.
Johnson posits that ideas or innovations that are deemed to be “ahead of their time” fail to gain traction in their days precisely because they skip the adjacent possible. There are environmental prerequisites for the successful proliferation of the right idea. These prerequisites can be societal norms, political environment, other technologies.
Charles Babbage’s Analytical Engine is an example of this an innovation that was “ahead of its time”. First conceived in the 1800s, the contraption was mainly shelved and rediscovered by the visionaries of WWII-era computer science. Babbage effectively designed a machine for the electronic age during the middle of the steam-powered mechanical revolution. Programmable computers needed vacuum tubes or integrated circuits, where information flows as tiny pulses of electrical activity instead of rusting steam-powered metal gears.
Liquid networks
Liquid networks create a fertile environment for systems to explore its adjacent possible.
A liquid network is one in which molecules are free to move around and from new configurations through random connections. Consider the states of human networks using phases of matter (gas, liquid, solid). In a “gaseous” network, collision and new configurations frequently occur, but immediately break apart due to the volatile environment. Gaseous networks are akin to hunter-gatherer tribes pre-agricultural revolution. In a “solid” network we get stability, but the network is incapable of change. Solid state societies are akin to pre-Renaissance medieval cultures, where information was largely passed down by monastic scribes.
Innovation thrived as societies became less hierarchical and more networked. No matter how smart "authorities" may be, if they are outnumbered a 1000-1 by the marketplace, there will be more good ideas lurking in the marketplace than a feudal castle.
Johnson is careful to note that this is different from a "hive mind"…
It’s not that the network itself is smart, it's that individuals get smarter because they are connected to the network.
Organisational workplaces can be designed to facilitate liquid networks by treating information spillover as a feature, not a flaw. MIT's Building 20 has movable walls that allowed the physical space to quickly be repurposed. The walls of Microsoft's Building 99 can be written on with erasable markers, designed to capture as many ideas and increase the surface area for idea collision as much as possible. These examples strike a good balance between order and chaos. A lot of open-office plans fail because they replaced excessive order with excessive chaos.
Slow hunches
Epiphanies are the exception, not the rule, in the history of world-changing ideas. Most start off as hunches that find another hunch, the main idea unfolding through accretion of hunches over much longer time frames.
Most hunches don't last long enough to turn into something useful because they are murky. Lots of innovators kept slow hunches alive by writing everything down. For Enlightenment-era innovators, reading and writing were inseparable.
Commonplace books were a popular kind of personal knowledge management system. Not only were they transcriptions of the authors' ideas, but they facilitated new insight as they are revisited. They broke texts into fragments and reassembling them into new patterns, annotated with their own notes in different sections of their notebooks. In looking at Darwin's old notebooks, we can somewhat trace his series of slow hunches over time. John Locke detailed his indexing method in An Essay Concerning Human Understanding, citing it as serving the higher purpose of "facilitating reflexive thought".
It's important that ideas not be categorised prematurely or too rigidly. Imposing too much order runs the risk of orphaning a promising hunch in a larger project that has died, and it makes it difficult for those ideas to mingle and breed when you revisit them. We understand something better by studying its behavior in different contexts. So you need a system for capturing hunches, but not necessarily categorizing them, because categories build barriers between disparate ideas, restrict them to their own conceptual islands. In using alternative systems such as associative indexing, you are more likely to stumble across notes later in a different context might spark new ideas and move your "slow hunch" along.
Flexibility and connection gives slow hunches to combine across minds and complete each other. The Web came into being as Tim Berners-Lee built on a hunch over decades. The seed was planted when he stumbled across Enquire Within Upon Everything as a child, grew through his freelancing projects organising a database of his colleagues, then given an environment to really foster at CERN, where it could be worked on separately from the immediate, day-to-day work. Information networks let ideas he had over the years to be augmented and polished.
Serendipity & ERROR
Serendipity needs noise, which give ideas a possibility of combining in novel ways. This is why it has become conventional wisdom to "sleep on a problem", and why we have heard multiple accounts of ideas coming to someone while they are driving, are on a walk or in the shower. The brain is away from regular task-based focus and enters a more associative state to experiment with new connections.
The brain has periods of chaos in which neurons fire out of sync with each other, neuroscientist Robert Thatcher believes these controlled bursts of chaos allows the brain to experiment with new links between neurons that would otherwise fail to connect in more orderly settings. During REM sleep, memories and associations are triggered in a chaotic, semi-random fashion, creating the hallucinatory quality of dreams. Most of those new neuronal connections are meaningless, but every now and then the dreaming brain stumbles across a valuable link that has escaped waking consciousness.
When nature finds itself in need of new ideas, it strives to connect, not protect. Sexual reproduction was nature's innovation to increase organisms' chances of survival in challenging conditions. Although asexual reproduction is more energy efficient than sexual reproduction, evolution is not just about quantity. When conditions are challenging, nature needs to innovate by making novel connections swapping gene with another organism increases introduces noise into the mix, introducing "errors" that just might turn out to increase the chances of survival.
Good ideas are more likely to emerge in environments that contain a certain amount of noise and error. Noise-free environments end up being too sterile and predictable in their output. The best innovation labs are always a little contaminated.
Thomas Kuhn argues that paradigm shifts begin with anomalies in the data, when scientists find that their predictions keep turning out to be wrong. Being wrong challenges our assumptions and forces us to adopt new strategies.
Evolution is possible thanks to errors. While most of the machinery in our cells preserves and reproduces useful genetic code, nature has not eliminated error completely. Bacteria increase their mutation rates dramatically when confronted with the “stress” of low energy supplies. When the living conditions are good, bacteria have less of a need for high mutation rates, because their current strategies are well adapted to their environment. But when the environment grows more hostile, the pressure to innovate shifts the balance of risk vs. reward involved in mutation.
Exaptation
The history of human activity is full of exaptations: ideas and technologies from unrelated domains being put to work on a new problem. Gutenberg printing press was exapted from screw-press used in wine making. Charles Babbage's Analytical Engine took inspiration from punch cards used in mechanical looms.
Most problems cannot be solved with a single discipline. Entrepreneurs who expand their curiosities outside their intellectual “islands,” were able to co-opt new ideas from external domains and put them to use in a new context. A common trait with early innovators is that they have many hobbies. Having many hobbies increase the "spare parts" of ideas that might find a random connection. Novel connections brew in what Johnson calls “slow multitasking”
Hobbies themselves might linger on for days or weeks before giving way to the next project. But there is steady variation nonetheless, not just in the subject matter but in the kind of work performed in each task. In slow multitasking mode, one project takes center stage for a series of hours or days while other projects linger in the margins of consciousness throughout.
Cities are great places for innovation because they nourish unconventionality, enabling subcultures to thrive. The inevitable spillover that happens between these subcultures create fertile ground for idea exaptation. A world where a diverse mix of distinct professions and passions overlap is a world where exaptations thrive. Cities create a liquid network where information can leak out of subcultures and influence their neighbors in surprising ways. This is one explanation for superlinear scaling in urban creativity.
Shared environments often take the form of a real-world public space, what the sociologist Ray Oldenburg famously called the “third place,” a connective environment distinct from the more insular world of home or office. The eighteenth-century English coffeehouse fertilized countless Enlightenment-era innovations; everything from the science of electricity, to the insurance industry, to democracy itself. Psychoanalysis and Freud's Wednesday night gatherings in Vienna, Parisian cafes and modernism, Homebrew Computer Club.
Platforms
Emergent platforms derive much of their creativity from the inventive and economical reuse of existing resources. Nature’s innovations rely on spare parts. Evolution is good at taking cobbling available resources together for new uses. The infinite variety of life in the coral reef exists because it is really good at recycling and reinventing the spare parts of its ecosystem, one species waste becomes another's nutrient.
Ecosystem engineers are a biological term used to describe species that not only has a disproportionate effect on its ecosystem (keystone species), but also play a key role in shaping the habitat itself. Beavers are a classic example. Beavers build dams to protect itself against predators. But this has an emergent effect of creating a space where kingfishers, dragonflies, and beetles can make a life for themselves. In creating forest wetlands, beavers are constantly toppling trees. Pileated woodpeckers, who make their homes by drilling large holes in dead trees, don't have to do the tree toppling themselves and thrive from the available habitat that beavers made available.
Platforms also have a natural appetite for trash, waste, and abandoned goods.
Modern paradigms are rarely overthrown. Instead, they are built upon. They create a platform that supports new paradigms above them. The most generative platforms come in stacks.
Tim Berners-Lee built the web on the open internet stack, YouTube was built on the web and Adobe Flash. Existing platforms explain why three guys could build it in six months, while expert committees and electronics companies took twenty years to make HDTV. Permissionless platforms allowed people to build more complex applications without reinventing the wheel.
The real benefit of stacked platforms lie in the knowledge that you no longer need to have. Platform builders can thought of as ecosystem engineers in this sense. Platform builders may have created infrastructure for their specific, niche purpose. But that infrastructure can be built on top of in ways they couldn't have imagined.
Zooming out…
If we take the 200 most influential innovations of the past 600 years and categorised them by where they were conceived (market vs. non-market environments) and by whom (individual vs. collective), we can see that as information networks become more fluid, most innovations cluster at the networked quadrant, specifically ones that not commercialised.
Renaissance-era (1400-1600): innovations can mainly be found in the non-market, individual quadrant. Information networks were slow (the printing press and postal service were still novelties), and entrepreneurial conventions poorly develop. There are no market infrastructures upon which to commercialise ideas. And so this era marks the notion of rogue visionaries and lone geniuses like da Vinci, Copernicus, and Galileo.
Enlightenment (1600-1800):. The majority of breakthroughs occur in collaborative environments thanks to the commercial adoption of the printing press, maturity of the postal system, and people congregating in cities. The latter gave rise to coffeehouses (birthplace of modern insurance) and intellectual hubs like the Royal Society. Most of these innovations still existed outside of the marketplace. The first English patent laws were not codified until the early 1700s. Nevertheless, this period marked the beginning of a rise in industrial capitalism in England.
Present (1800-): If capitalism so rewarded competition, one might expect that most of the innovations over the last 2 centuries were to be found in the market, individual quadrant. However, that quadrant is actually the least populated. While conventional wisdom states that competition between rival firms leads to innovation in their products and services. When you look at innovation from the long-zoom perspective, competition turns out to be less central to the history of good ideas than we generally think.
The long-zoom approach lets us see that openness and connectivity may, in the end, be more valuable to innovation than purely competitive mechanisms. Though commercialisation comes with greater financial rewards, market forces push toward secrecy, making it harder for open patterns of innovation to work their magic. We are often better served by connecting ideas than we are by protecting them.