Friday, November 24, 2006

The Web 3.0 Manifesto - The Knowledge Doubling Curve

[This is part I in a multi-part series titled "The Web 3.0 Manifesto"]

PREFACE: I use the term Human Computing Layer or
HCL in this article. What is the HCL? Us. It’s the oldest computing system on the planet and has been here since we began as a species. Read my previous Web 3.0 article that discusses Amazon’s Mechanical Turk service to understand how the HCL is becoming a functional and integral part of the Web and for some ideas on how the integration of the HCL and the Web will take form.

The Knowledge Doubling Curve

To begin this multi-part series on Web 3.0 I want to talk about the phenomenon that is driving the breathtaking increase in the rapid rate of technological innovation. Several decades ago I read about a study where researchers decided to measure the time it took for the amount of knowledge in the world to double. They called it the “The Knowledge Doubling Curve“. (Note: If anyone knows where I can find this article I would really like to know.) They came up with their own measuring system that, if I remember correctly, consisted of measuring the total number of items in print at periodic intervals in recent history. They then graphed that figure over time to see how long it took for the amount of knowledge to double. Finally, they projected the graph into the future. By looking at the graph it became possible to see the rate at which knowledge was doubling over time. In the article they showed a graph like the one below:

figure 1

I don’t remember the exact dates or the exact quantities for the number of items in print at each time event, so I can’t label the X and Y axes of the graph properly. Roughly, the bend in the graph corresponds to a decade somewhere near or after the 1960’s. However, despite the lack of exact figures the radical conclusion represented by this graph still holds:

The time it takes for knowledge to double started out as a linear rate and is now progressing at an exponential rate.

As you can see the graph is asymptotic, that is, the rate continually approaches infinity but never reaches it. This curve of course tracks closely the rate of technological progress in modern civilization. (Note: There are some who would say that the Singularity occurs somewhere near the top of the graph. For a fascinating look at how the rate of technological innovation tends to increase exponentially, read any of the latest books or lectures by renowned futurist, Ray Kurzweil.)

The amount of technological progress made in the last hundred or so years, far outstrips all that made since modern man first appeared on this planet. It took centuries for the wheel, the spear, the bow and arrow, paper, etc. to become commonplace. But since Samuel Morse sent the first modern telegram in 1844, we’ve seen the light bulb, cars, the telephone, the atomic bomb, jets, radio, television, space travel, computers, nanotechnology, and the Internet. There has been such an astonishing increase in the speed of invention and innovation that the list I just gave you is woefully and radically incomplete.

What changed in the last 150 years or so? It is certainly not us. Being an American, I have read writings made by the forefathers of our country, who gave us one of the most eloquent and powerful documents of our time, The Declaration Of Independence. They were brilliant men who would easily tower, intellectually, head and shoulders above most our contemporary politicians. Therefore the change is not related to an evolution of our DNA or a giant jump in the intelligence of mankind.

The Knowledge Duplication Curve

To understand the answer let me share an enlightenment that I had about The Knowledge Doubling curve. Look at the curve I plotted in the figure below:

figure 2

As you can see, the curve is a mirror image of The Knowledge Doubling Curve if you flipped it vertically. This curve is not based on any study; it is an intuitive explanation of the mechanics behind The Knowledge Doubling Curve. This curve shows that as technology advanced, the amount of time wasted by humanity in creating duplicate solutions was reduced linearly and that after the bend, the rate of reduction became exponential.

When a caveman solved the problem of painting on a cave wall, the ability for him to transfer that solution to others was limited to a small geographical area around him. Paper was a giant leap forward because solutions could now be written down, copied, and spread to others far and wide. The printing press accelerated that spread by making the copying of written works, and therefore the solutions contained, much faster. But in the last 150 years, the speed of distribution and replication of solutions has reached a breakneck pace never seen before in the history of civilization.

Even in my short lifetime, I have gone from having to go to the local library to find a solution, or having to find and contact an expert via the telephone, to being able to download instantaneously an entire software package that is a complete solution to a problem or need I have. For example, if you want to have your own discussion forum all you have to do is download and take a few minutes to install a free forum software package like phpBB.


Now, what is driving the rapid decrease in the Knowledge Duplication Curve? Connectivity. The more connected we are the less time we waste duplicating solutions. Let’s highlight particular members from the list of recent technological advancements I made before, specifically, the modern telegraph, the telephone, radio, television, and now the Internet. Each of these was a quantum leap in connectivity, leading correspondingly to a quantum decrease in the duplication of effort. I included radio and television because links do not have to be both ways. Any efficient broadcast technology, even if it is one-way, increases our connectivity.

Web Versioning Defined

Here is the definition for what constitutes a new version of the Web:

Any technological change that is a quantum leap in our ability to rapidly share solutions over the Web by providing modular reusable building blocks of functionality constitutes a version change.
  • Web 1.0 - Connected computers together using a set of standardized protocols invented by Internet pioneer Vint Cerf.

  • Web 2.0 - Marked by the appearance of Web Services which are modular solutions to complex problems, made available over the Web to external developers via an application program interface (API).

  • Web 3.0 - The marriage of artificial intelligence and The Human Computing Layer (HCL) and their subsequent integration into the Web, making powerful pattern recognition solving capabilities widely available to web surfers and developers alike.

Web 1.0 allowed us to share files, data, and software over the Internet.

Web 2.0 allowed us to share modular programming solutions to common problems, available via web interface API calls. This allowed and allows outside developers to build software applications on top of these services without having to download or integrate foreign code libraries into their own software, greatly increasing the ease and the pace of creating new software applications.

Web 3.0 will allow us to share an entirely new class of solutions over the web, both by developers and directly by users (web surfers) to build larger more complex applications. Most importantly, these shareable solutions, with the help of artificial intelligence and the integration of The Human Computing Layer, will allow us to cooperatively solve a class of problems normally reserved for specialized applications found in the areas of complex pattern recognition and high level semantic analysis.


I will close this article with a word of hope and anticipation for the future. If you take a high energy beam of ordinary light and shine it at a thick piece of steel you get a nice reflection. When you take that same light and align the photons so they move together in lock step, they form a laser beam and you can burn a hole through that same steel. I leave you with this exciting question. What happens when we, the most powerful computing beings on the planet working together in superhuman harmony, turn our combined attention to the monumental problems that, to date, have evaded solution?

Coming soon…

In future articles in the Web 3.0 Manifesto series I will discuss further the shape and substance of Web 3.0, especially in regards to how artificial intelligence and The Human Computing Layer will cooperate and integrate with the Web. Thank you for reading this far and sharing some of your time with me.

For more thoughtful commentary on Web 2.0, Web 3.0 and the Semantic Web I strongly recommend reading Dion Hinchcliffe's recent blog post "Going Beyond User Generated Software: Web 2.0 and the Pragmatic Semantic Web". Pay special attention to his comments regarding "recombinant, self-assembling software that exploits collective intelligence". He does point out that the companies he mentions involved in this line of research are using good old Web 2.0 techniques, but I feel that this field of research will play a big part in shaping Web 3.0.

Bookmark This Post!

BlinkList | | Digg it | ma.gnolia | RawSugar |
reddit | Shadows | Simpy | Spurl | Yahoo MyWeb


At 2:27 PM, Anonymous Anonymous said...

I have been puzzling over what web 3.0 will look like too. I think it will be a bit like 2.0 where several ideas all come together to create a whole new world. You can see my idea for the user interface of Web 3.0 at

At 7:09 PM, Anonymous Anonymous said...

I am not quite sure how you draw the conclusion that since the forefathers of America were of extraordinary intelligence, wisdom and common sense, the “change” is not due to a dramatic change to out intelligence. I had to read that section twice, and I am still puzzled with your reasoning.

At 7:46 AM, Anonymous Mike Bergman said...

Very interesting posting; I would like to see the reference to the knowledge doubling myself.

I thought you might be interested in one of my blog postings from July that shows EXACTLY the same curve for trends in wealth.

Could it be that knowledge and wealth are inextricably linked?

At 6:35 AM, Blogger Paul said...

I'm also looking for the research you indicated and am really having a very difficult time finding it. I do hope another reader can help us locate that research.

To your point, I don't want to duplicate it!

At 2:17 AM, Blogger zakkal said...

Do we really believe that with all the blogging and posting going on, there's now less duplication than ever before?

And is the number of words in print, or the number of webistes, etc, a measure of knowledge?

At 11:29 AM, Blogger roschler said...

I can't find the reference to the original knowledge doubling article since it came from a print article in the 1960/70's time period.

In reference to the "less duplication ever before" comment, the actual point I make is that there is more solution reuse than ever before, hence less duplication in the critical areas of problem solving like programming, engineering, etc. I am not referring to the obvious rampant duplication that comes from general text creation.

At 12:42 PM, Blogger Diego Pimentel said...

Dear Roschler,
You cand find this information about it online on:
by Theodore Modis. And obviously taken by the Futurist's Guru Ray Kurzweil at: Ray Kurzweil, The Singularity Is Near : When Humans Transcend Biology, Viking Adult, 2005, ISBN 0670033847.

Regards from Argentina,


At 9:57 AM, Anonymous Anonymous said...

I think it's "Fuller's Curve" from Buckminster Fuller's 1982 book, "Critical Path."

At 3:39 PM, Blogger MDH said...

I first heard about the doubling of the "universal body of human knowledge" sometime in the 1980s. I found the following information on Peter Russell's webpage

Russell wrote:
Estimating the rate of growth in our collective knowledge is a difficult task. One inspired attempt has been that of the French economist Georges Anderla for the Organization for Economic Cooperation and Development. He takes the known scientific facts of the year AD1 to represent one unit of collective human knowledge. Assuming that our collective learning began with language, it had taken approximately fifty thousand years for humanity to accrue that first unit

According to Anderla’s estimates, humanity had doubled its knowledge by a.d.1500. By 1750 total knowledge had doubled again; and by 1900 it had become 8 units. The next doubling took only fifty years, and the one after that only ten years, so that by 1960 humanity had gathered 32 units of knowledge. It then doubled again in the next seven years, and again in the following six years, taking us to 128 units in 1973, the year of Anderla’s study.

There is no indication that this acceleration has slowed since then. It has almost certainly continued to increase ever-more rapidly. Today, with the advent of the information revolution, human knowledge is estimated to be doubling once every eighteen months.


Post a Comment

<< Home