Distributed Systems — Destruction of the Status Quo

One thing we can say for sure is that there will be a lot more start-ups. The monolithic, hierarchical companies of the mid-twentieth century are being replaced by networks of smaller companies. This process is not just something happening now in Silicon Valley. It started decades ago, and it’s happening as far afield as the car industry. It has a long way to run. —Paul Graham, Startup Investing Trends

This is one of my core beliefs, that the world is in the process of becoming much more distributed, that the traditional hierarchies (as articulated by Fred Wilson in his Le Web talk here) we’ve organized around are changing. It’s not just social or organizational hierarchies, though; enabled by technology, our infrastructure systems (energy, manufacturing, farming) are also becoming much more distributed. These newly formed distributed systems are having  profound consequences for large legacy institutions and creating opportunities for smaller start-up companies. What follows is a discussion of old versus new in myriad industries to illustrate this shift.

Mainframes vs. Data Centers

We’ve seen computation go from large mainframes in the 1960s (think of this as Big Computation) to personal computing in the 80s, with a desktop in every home and office, and now to data centers. We seem to have settled in on the right distributed size — distributed data centers with mostly laptops and mobile phones connected to them. And this pattern can be seen now in more traditional industries as well.

Power Plants vs. Distributed Generation

I think what happened to computers is a good analogy to understand what is happening in energy. In energy, we have Big Energy — giant nuclear, coal, and natural-gas plants that deliver our power over large transmission lines. These are the equivalent of the mainframe computers from the 1960s. Now we are seeing a large increase in demand for distributed generation, one form of which is rooftop solar — this is the personal computer of the 1980s. But it’s incredibly inefficient for each of us to build our own power plant. A similar thing happened to computers — rather than build your own data center, we are now moving to “the cloud.” The cloud is just a form of distributed computation — regional distributed data centers. My guess is that we’re going to end up with an integrated, multi-layered system of power generation that consists of regional distributed generation and small modules at the household level. This might mean that today’s rooftop solar companies will become the personal computer companies of tomorrow (whatever happened to Gateway?).

Factories vs. 3D Printing

Manufacturing is about to be disrupted by 3D printing and automation driven by increases in artificial intelligence (AI). To date, labor gains have offset transportation costs in manufacturing. This is why the world’s manufacturing routinely moves to the lowest-cost labor markets — to China and now to Southeast Asia — despite the fact that these locations are thousands of miles from where the product will ultimately be distributed. With 3D printing, the costs of both manufacturing and the necessary investments will collapse. In addition, much of the assembly is already being automated by robotics and AI. So Big Manufacturing (factories and assembly lines) is already being disrupted and is about to be displaced by distributed manufacturing (3D printers and robotics). We talk of local food, but we’re on the verge of local everything — why produce our goods halfway around the world? Local will be cheap. If your economy is driven by cheap labor, you’re in trouble. (This phenomenon could also collapse trade, leading to more insular economies, but this is a discussion for another day.)

Big Farming vs. Farmers’ Markets

Food is slowly following energy into a more distributed system as well. We’ve seen it with the abundance of farmers’ markets, the return of gardens, and the movement to grow food in cities. To this point, it’s been more small-scale hobby-sized projects. But like manufacturing and energy, there will come a time where the cost to produce is less than the labor and transportation costs. You’ll see distributed farming systems, not unlike data centers, that deliver local, fresh produce year round. This will become even more pronounced as protein-based alternatives chip away at traditional meat demand.

Universities vs. Online Education

Big Education is on the verge of a massive disruption. The value of a degree seems to be collapsing as all the information anyone might need is available free and online provided by sites like Kahn Academy, among others. Universities are struggling with new models — put classes online for free, extend online degree programs, or ignore it all and hope it goes away. If we follow the other trends and apply them to education, it probably means fewer big universities and more small schools. Or perhaps we move even more closely to virtual education: one can easily imagine students in groups of 25-50 in towns around the world gathering together to take online classes from the very best professors. In fact, this is exactly what the Acumen, a patient-capital fund, is doing. My prediction is that the best universities, by changing their business models, have a chance to take market share from the middle tier — the best and the boutiques will be okay — but the middle is in trouble. We’ll also see a surge in start-up colleges and universities as the barriers to entry are gone.

The Paper vs. Twitter

I grew up in a house where my parents got the paper in the morning. They physically walked to the end of the driveway, which seems so odd, so archaic, now. My paper today is Twitter and RSS feeds. The decline of the newspaper industry has been well documented — print journalism is being replaced with a global distributed network of millions of bloggers and tweeters around the world. If I’m interested in a news event, I simply find and “turn on” the hashtag on Twitter.

Federal- and State-Level Governments vs. Cities

There has been a lot written about cities and their rise on the global stage. In a sense, government has become more distributed. Here in the US, it seems that cities are more important than ever and are increasingly the only governable unit — state and federal government seems to matter less and to be less effective. As Richard Florida has observed, we will have a “spikey” world with concentrations of people, wealth, and ideas in the best cities. The effective unit size to govern — just like a data center — has become a city.

We’re moving from a large, hierarchical, centralized world to one that is small, networked, and distributed. It’s as if each industry is homing in on their own Dunbar number — the perfectly sized data center, solar installation, or global classroom. The amazing thing to me is the breadth of this phenomenon, from education to utilities to manufacturing. Over the next 10-20 years, many of the institutions that have been in place for the last 100 years will be fundamentally altered. These systems will be smarter, stronger, and faster than the previous ones.

Increasing Density — Corn, Cities, Fuels and Circuits

Whether it’s the number of transistors on a microchip or the number of bushels of corn per acre, there is an undeniable trend toward increasing density. This creates efficiency and thus leads to an increase in productivity. In fact, one of the key components of successful technology is its ability to be miniaturized. The rate of change, governed by different parameters, is different for each industry, but the trend is clearly up and to the right everywhere you look.

Agriculture: Bushels of Corn per Acre (USDA)


Farming productivity has steadily increased. For example, from 1950 to 2000, the average yields of America’s three most important crops (corn, soybeans, and wheat) rose 3.5x, 1.7x, and 2.5x, respectively. (SMIL/USDA 2000) It was the continuous introduction of new technologies that enabled these gains, allowing us to meet the caloric needs of a rising population. The technology came first (before 1950, of course) in the form of draft animals and the use of manure for fertilization; then came synthetic fertilizer, pesticides, and combustion engines to drive harvesters and planters. From here, the transition to automated labor (think the Google car plus a combine) and more controlled environments like greenhouses and eventually vertical farms will inevitably lead to further gains.

Urbanization: From the Fields to the Cities (the Economist)


The number of Americans working on farms has steadily decreased, and by 2000, less than 5% of the US population were farmers. Gains in agricultural efficiency led to a mass migration of people from rural areas to cities, resulting in a large increase in the density of people per acre. This transition further led to gains in productivity as people lived closer, shared resources, and collaborated more. These advances in productivity mean that places like New York City can have the lowest energy emissions per capita. Cities are now cultural hotspots (see the rise of the Creative Class), not too different than biodiversity hotspots, and this urbanization will continue, mostly in Asia, as the rural become the urban around the world. Urbanization is in effect an increase in the density of people per unit of area, which leads to lower energy usage per capita and a host of other efficiencies.

Energy: US Major Fuel Transitions (EIA)


The US has undergone a number of energy transitions, from wood to coal to oil, throughout our history, and each of these was one of increasing density. Wood (16.2 MJ/kg) was replaced by coal (24 MJ/kg), a 1.5x increase in energy density. Coal was then replaced by oil, which was refined into gasoline (46 MJ/kg), leading to a 1.91x increase in density. Recently, methane (55.6 MJ/kg) or natural gas has passed coal. Methane is technically denser by mass than both coal and oil, but storing large amounts of gas in a confined space has its challenges (i.e., it requires extremely high pressures or cold temperatures).

Looking at trends this way can become a good filter. For example, ethanol at 25.65 MJ/liter compared to gasoline at 34.2 MJ/liter doesn’t look like such a great improvement. Hydrogen at 123 MJ/kg and uranium at 83,140,000 MJ/kg would be logical next steps, though. We are a long way from hydrogen-powered cars, and the development of nuclear power has been all but halted due to the recent accidents in Japan, however. Still, it’s interesting to note that each major transition over the last 200 years has been one to higher energy density.

Technology: Moore’s Law


Lastly, we come to the one everyone knows — Moore’s Law, which states that every two years, the number of transistors on an integrated circuit will double. This increase in density is what has given us the Internet, mobile phones, and even solar panels (as costs have dropped due to similar production techniques). What’s interesting about this trend is the magnitude of it — in the last 40 years, computers have become 500,000x more dense. There appears to be no end in sight as just when a physical limit appears to be reached, a new technology emerges again. Ultimately, we may find ourselves with quantum or DNA computers, both of which could lead to further increases in density.

Observations and Questions

1. Trends: What’s amazing to me looking at these charts is how smooth they are. Those lines represent the culmination of technology over decades, and yet they are clear, consistently escalating trends. These are trends that you can depend on, that are investible, and that you should be aware of. If you’re starting a business, you need to think about where you’re going to be when you go to market, not just today.

2. Transitions: There are times in each of these trends when there is a major technological shift or leap. And in fact, I think we’re in the midst of one right now with farming as we move towards more controlled indoor environments. These are step changes where there is opportunity and where wealth gets created, but investing alongside incremental changes is a tough business — the solar industry has seen one company after another go out of business as they pursue small incremental changes in panel efficiency.

3. Normal vs. Log: While the lines may look similar, the technology chart is logarithmic. Every unit is a 10x increase as opposed to a 1x increase for the corn chart. This is an enormous difference: in agriculture, a gain of 2-3x over 50 years is huge, yes, but in technology, the gain may be 500,000x over the same period. The physical world behaves differently — has different constraints — than the world of software.

4. Next: You would think there have to be limits to these trends, and we may in fact eventually witness some such barriers, but the trends in yield, the trends in urbanization, the transition to methane, and the trends in technology (chips, solar, sequencing) all seem intact for the foreseeable future. These are all good things — we’ll produce more food with fewer resources, we’ll live on less land, we’ll use more efficient fuels, and we’ll have even more powerful computers in our pockets.