Singularity Institute

Why Work Toward the Singularity?

If you travelled backward in time to witness a critical moment in the invention of science, or the creation of writing, or the evolution of homo sapiens, or the beginning of life on Earth, no human judgment could possibly encompass all the future consequences of that event, and yet there would be the feeling of being present at the dawn of something worthwhile. The most critical moments of history are not the closed stories, like the start and finish of wars, or the rise and fall of governments. The story of intelligent life on Earth is made up of beginnings.

Imagine traveling back in time to witness a critical moment in the dawn of human intelligence. Suppose that you find an alien bystander already on the scene, who asks: "Why are you so excited? What does it matter?" The question seems almost impossible to answer; it demands a thousand answers, or none. Someone who valued truth and knowledge might answer that this was a critical moment in the human quest to learn about the universe, in fact, the beginning of that quest. Someone who valued happiness might answer that the rise of human intelligence was a necessary precursor to vaccines, air conditioning, and the many other sources of happiness and solutions to unhappiness that have been produced by human intelligence over the ages. There are people who would answer that intelligence is meaningful in itself; that "It is better to be Socrates unsatisfied than a fool satisfied; better to be a man unsatisfied than a pig satisfied." A musician who chose that career believing that music is an end in itself might answer that the rise of human intelligence mattered because it was necessary to the birth of Bach; a mathematician could single out Euclid; a physicist might cite Newton or Einstein. Someone with an appreciation of humanity, beyond the individual humans, might answer that this was a critical moment in the relation of life to the universe, the beginning of humanity's growth,of our acquisition of strength and understanding, eventually spreading beyond Earth to the rest of the galaxy and the universe.

The beginnings of human intelligence, or the invention of writing, probably went unappreciated by the individuals who were present at the time. But such developments do not always take their creators unaware. Francis Bacon, one of the critical figures in the invention of the scientific method, made astounding claims about the power and universality of his new mode of reasoning and it's ability to improve the human condition; claims which, from the perspective of a 21st century human, turned out to be exactly right. Not all good deeds are unintentional. It does occasionally happen that humanity's victories are won not by accident but by people making the right choices for the right reasons.

Why is the Singularity worth doing? The Singularity Institute for Artificial Intelligence can't possibly speak for everyone who cares about the Singularity. We can't even presume to speak for the volunteers and donors of the Singularity Institute. But it seems like a good guess that many supporters of the Singularity have in common a sense of being present at a critical moment in history; of having the chance to win a victory for humanity by making the right choices for the right reasons. Like a spectator at the dawn of human intelligence, trying to answer directly why superintelligence matters chokes on a dozen different simultaneous replies; what matters is the entire future growing out of that beginning.

But it is still possible to be more specific about what kinds of problems we might expect to be solved. Some of the specific answers seem almost disrespectful to the potential bound up in superintelligence; human intelligence is more than an effective way for apes to obtain bananas. Nonetheless, modern day agriculture is very effective at producing bananas, and if you had advanced nanotechnology at your disposal, energy and matter might be plentiful enough that you could produce a million tons of bananas on a whim. In a sense that's what nanotechnology is, good old fashioned material technology pushed to the limit. This only begs the question of So what?, but the Singularity advances on this question as well; if people can become smarter, this moves humanity forward in ways that transcend the faster and easier production of more and more bananas. For one thing, we may become smart enough to answer the question So what?

In one sense, asking what specific problems will be solved is like asking Benjamin Franklin in the 1700s to predict electronic circuitry, computers, Artificial Intelligence, and the Singularity on the basis of his experimentation with electricity. Setting an upper bound on the impact of superintelligence is impossible; any given upper bound could turn out to have a simple workaround that we are too young as a civilization, or insufficiently intelligent as a species, to see in advance. We can try to describe lower bounds; if we can see how to solve a problem using more or faster technological intelligence of the kind humans use, then at least that problem is probably solvable for genuinely smarter than human intelligence. The problem may not be solved using the particular method we were thinking of, or the problem may be solved as a special case of a more general challenge; but we can still point to the problem and say: "This is part of what's at stake in the Singularity."

If humans ever discover a cure for cancer, that discovery will ultimately be traceable to the rise of human intelligence, so it is not absurd to ask whether a superintelligence could deliver a cancer cure in short order. If anything, creating superintelligence only for the sake of curing cancer would be swatting a fly with a sledgehammer. In that sense it is probably unreasonable to visualize a significantly smarter than human intelligence as wearing a white lab coat and working at an ordinary medical institute doing the same kind of research we do, only better, in order to solve cancer specifically as a problem. For example, cancer can be seen as a special case of the more general problem The cells in the human body are not externally programmable. This general problem is very hard from our viewpoint; it requires full scale nanotechnology to solve the general case, but if the general problem can be solved it simultaneously solves cancer, spinal paralysis, regeneration of damaged organs, obesity, many aspects of aging, and so on. Or perhaps the real problem is that the human body is made out of cells or that the human mind is implemented atop a specific chunk of vulnerable brain, although calling these problems raises philosophical issues not discussed here.

Singling out cancer as the problem is part of our culture's particular outlook and technological level. But if cancer or any generalization of cancer is solved soon after the rise of smarter than human intelligence, then it makes sense to regard the quest for the Singularity as a continuation by other means of the quest to cure cancer. The same could be said of ending world hunger, curing Alzheimer's disease, or placing on a voluntary basis many things which at least some people would regard as undesirable: illness, destructive aging, human stupidity, short lifespans. Maybe death itself will turn out to be curable, though that would depend on whether the laws of physics permit true immortality. At the very least, the citizens of a post Singularity civilization should have an enormously higher standard of living and enormously longer lifespans than we see today.

What kind of problems can we reasonably expect to be solved as a side effect of the rise of superintelligence; how long will it take to solve the problems after the Singularity; and how much will it cost the beneficiaries? A conservative version of the Singularity would start with the rise of smarter than human intelligence in the form of enhanced humans with minds or brains that have been enhanced by purely biological means. This scenario is more conservative than a Singularity which takes place as a result of brain computer interfaces or Artificial Intelligence, because all thinking is still taking place on neurons with a characteristic limiting speed of 200 operations per second; progress would still take place at a humanly comprehensible speed. In this case, the first benefit's of the Singularity probably would resemble the benefit's of ordinary human technological thinking, only more so. Any given scientific problem could benefit from having a few Einsteins or Edisons dumped into it, but it would still require time for research, manufacturing, commercialization and distribution.

Human genius is not the only factor in human science, but it can and does speed things up where it is present. Even if intelligence enhancement were treated solely as a means to an end, for solving some very difficult scientific or technological problem, it would still be worthwhile for that reason alone. The solution might not be rapid, even after the problem of intelligence enhancement had been solved, but that assumes the conservative scenario, and the conservative scenario wouldn't last long. Some of the areas most likely to receive early attention would be technologies involved in more advanced forms of superintelligence: broadband brain computer interfaces or full fledged Artificial Intelligence. The positive feedback dynamic of the Singularity, smarter minds creating still smarter minds, doesn't need to wait for an AI that can rewrite it's own source code; it would also apply to enhanced humans creating the next generation of Singularity technologies.

The Singularity creates speed for two reasons: First, positive feedback, intelligence gaining the ability to improve intelligence directly. Second, the shift of thinking from human neurons to more readily expandable and enormously faster substrates. A brain computer interface would probably offer a limited but real version of both capabilities; the external brainpower would be both fast and programmable, although still yoked to an ordinary human brain. A true Artificial Intelligence, or a human scanned completely into a sufficiently advanced computer, would have total self access. At this point one begins to deal with superintelligence as the successor to current scientific research, the global economy, and in fact the entire human condition; rather than a superintelligence plugging into the current system as an improved component. At this point human nature sometimes creates an Us Vs. Them view of the situation; the instinct that people who are different are therefore on a different side, but if humans and superintelligences are playing on the same team, it would be straightforward for the most advanced mind at any given time to offer a helping hand to anyone lagging behind; there is no technological reason why humans alive at the time of the Singularity could not participate in it directly. In our view this is the chief benefit of the Singularity to existing humans; not technologies handed down from above but a chance to become smarter and participate directly in creating the future.

One idea that is often discussed along with the Singularity is the proposal that, in human history up until now, it has taken less and less time for major changes to occur. Life first arose around three and half billion years ago; it was only eight hundred and fifty million years ago that multi celled life arose; only sixty five million years since the dinosaurs died out; only five million years since the hominid family split off within the primate order; and less than a hundred thousand years since the rise of homo sapiens in it's modern form. Agriculture was invented ten thousand years ago; Socrates lived two and half thousand years ago; the printing press was invented five hundred years ago; the computer was invented around sixty years ago. You can't set a speed limit on the future by looking at the pace of past changes, even if it sounds reasonable at the time; history shows that this method produces very poor predictions. From an evolutionary perspective it is absurd to expect major changes to happen in a handful of centuries, but today's changes occur on a cultural timescale, which bypasses evolution's speed limit's. We should be wary of confident predictions that transhumanity will still be limited by the need to seek venture capital from humans or that Artificial Intelligences will be slowed to the rate of their human assistants (both of which I have heard firmly asserted on more than one occasion).

We can't see in advance the technological pathway the Singularity will follow, since if we were that smart ourselves we'd already have done it. But it's possible to toss out broad scenarios, such as A smarter than human AI absorbs all unused computing power on the then existent Internet in a matter of hours; uses this computing power and smarter than human design ability to crack the protein folding problem for artificial proteins in a few more hours; emails separate rush orders to a dozen online peptide synthesis labs, and in two days receives via FedEx a set of proteins which, mixed together, self assemble into an acoustically controlled nanodevice which can build more advanced nanotechnology. This is not a smarter than human solution; it is a human imagining how to throw a magnified, sped up version of human design abilities at the problem. There are admittedly initial difficulties facing a superfast mind in a world of slow human technology. Even humans, though, could probably solve those difficulties, given hundreds of years to think about it. And we have no way of knowing that a smarter mind can't find even better ways.

If the Singularity involves not just a few smarter than usual researchers plugging into standard human organizations, but the transition of intelligent life on Earth to a smarter and rapidly improving civilization with an enormously higher standard of living, then it makes sense to regard the quest to create smarter minds as a means of directly solving such contemporary problems as cancer, AIDS, world hunger, poverty, et cetera. And not just the huge visible problems; the huge silent problems are also important. If modern day society tends to drain the life force from it's inhabitants, that's a problem. Aging and slowly losing neurons and vitality is a problem. In some ways the basic nature of our current world just doesn't seem very pleasant, due to cumulative minor annoyances almost as much as major disasters. This may usually be considered a philosophical problem, but becoming smarter is something that can actually address philosophical problems.

The transformation of civilization into a genuinely nice place to live could occur, not in some unthinkably distant million year future, but within our own lifetimes. The next leap forward for civilization will happen not because of the slow accumulation of ordinary human technological ingenuity over centuries, but because at some point in the next few decades we will gain the technology to build smarter minds that build still smarter minds. We can create that future and we can be part of it.

If there's a Singularity effort that has a strong vision of this future and supports projects that explicitly focus on transhuman technologies such as brain computer interfaces and self improving Artificial Intelligence, then humanity may succeed in making the transition to this future a few years earlier, saving millions of people who would have otherwise died. Around the world, the planetary death rate is around fifty five million people per year (UN statistics) 150,000 lives per day, 6,000 lives per hour. These deaths are not just premature but perhaps actually unnecessary. At the very least, the amount of lost lifespan is far more than modern statistics would suggest.

There are also dangers for the human species if we can't make the breakthrough to superintelligence reasonably soon. Albert Einstein once said: "The problems that exist in the world today cannot be solved by the level of thinking that created them." We agree with the sentiment, although Einstein may not have had this particular solution in mind. In pointing out that dangers exist it is not our intent to predict a dystopian future; so far, the doomsayers have repeatedly been proven wrong. Humanity has faced the future squarely, rather than running in the other direction as the doomsayers wished, and has thereby succeeded in avoiding the oft predicted disasters and continuing to higher standards of living. We avoided disaster by inventing technologies which enable us to cope with complex futures. Better, more sustainable farming technologies have enabled us to support the increased populations produced by modern medicine. The printing press, telegraph, telephone, and now the Internet enable humanity to apply it's combined wisdom to problem solving. If we'd been forced to move into the future without these technologies, disaster probably would have resulted. The technology humanity needs to cope with the coming decades may be the technology of smarter than human intelligence. If we have to face challenges like basement laboratories creating lethal viruses or nanotechnological arms races with just our human intelligence, we may be in trouble.

Finally, there is the integrity of the Singularity itself to safeguard. This is not necessarily the most difficult part of the challenge, compared to the problem of creating smarter than human intelligence in the first place, but it needs to be considered.It is possible that the integrity of the Singularity needs no safeguarding; that any human from Gandhi to Stalin, if enhanced sufficiently far beyond human intelligence, would end up being wiser and more moral than anyone alive today; that the same holds true for all minds in general from enhanced chimpanzees to arbitrarily constructed Artificial Intelligences. But this is not something we know in advance. Since we don't know how many moral errors persist in our own civilization, safeguarding the integrity of the Singularity, in our view, consists more of ensuring the will and ability to grow wiser with increased intelligence than of trying to find perfect candidates for human intelligence enhancement. An analogous problem exists for Artificial Intelligence, where the task is not enforcing servitude on the AI or coming up with a perfect moral code to hardwire, but rather transferring over the features of human cognition that let us conceive of a morality improving over time (see the section on Friendly Artificial Intelligence for more information).

Safeguarding the integrity of the Singularity is another reason for facing the challenge of the Singularity squarely and deliberately. It may be that human intelligence enhancement will turn out well regardless, but there is still no point in taking unnecessary risks by driving the projects underground. If human intelligence enhancement is banned by the FDA, for example, this just means that the first experiments will take place outside the US, slightly later than they otherwise would have; increasing the possible risks, delaying the possible benefit's. If human intelligence enhancement is banned by the UN this means the experiments will take place offshore, out of the public eye, and perhaps sponsored by groups that we would prefer not be involved, although there is a significant chance it would turn out well regardless. In the case of Artificial Intelligence there are certain specific things that must be done to place the AI in the same moral frame of reference as humanity, to ensure the AI absorbs our virtues, corrects any inadvertently absorbed faults, and goes on to develop along much the same path as a recursively self improving human altruist. Friendly Artificial Intelligence is not necessarily more difficult than the problem of AI itself, but it does need to be handled along with the creation of Artificial Intelligence. In both cases, we can best safeguard the integrity of the Singularity by confronting the Singularity intentionally and with full awareness of the responsibilities involved.

What does it mean to confront the Singularity? Despite the enormity of the Singularity, sparking the Singularity, creating the first smarter than human intelligence, is a problem of science and technology. The Singularity is something that we can actually go out and do, not a philosophical way of describing something that inevitably happens to humanity. It takes the sweep of human progress and a whole technological economy to create the potential for the Singularity, just as it takes the entire framework of science to create the potential for a cancer cure, but it also takes a deliberate effort to run the last mile and fulfill that potential. If someone asks you if you're interested in donating to AIDS research, you might reply that you believe that cancer research is relatively underfunded and that you are donating there instead; you would probably not say that by working as a stockbroker you support the world economy in general and thereby contribute as much to humanity's progress toward an AIDS cure as anyone. In that sense, sparking the Singularity is no different from any other grand challenge; someone has to do it.

At this moment in time, there is a tiny handful of people who realize what's going on and are trying to do something about it. It is not quite true that if you don't do it, no one will, but the pool of other people who will do it if you don't is smaller than you might think. If you're fortunate enough to be one of the few people who currently know what the Singularity is and would like to see it happen, even if you learned about the Singularity just now, we need your help because there aren't many people like you. This is the one place where your efforts can make the greatest possible difference, not just because of the tremendous stakes, though that would be far more than enough in itself, but because so few people are currently involved.

The Singularity Institute exists to carry out the mission of the Singularity aware to accelerate the arrival of the Singularity in order to hasten it's human benefit's; to close the window of vulnerability that exists while humanity cannot increase it's intelligence along with it's technology; and to protect the integrity of the Singularity by ensuring that those projects which finally implement the Singularity are carried out in full awareness of the implications and without distraction from the responsibilities involved. That's our dream. Whether it actually happens depends on whether enough people take the Singularity seriously enough to do something about it, whether humanity can scrape up the tiny fraction of it's resources needed to face the future deliberately and firmly.

We can do better. The future doesn't have to be the dystopia promised by doomsayers. The future doesn't even have to be the flashy yet unimaginative chrome and computer world of traditional futurism. We can become smarter. We can step beyond the millennia old messes created by human level intelligence. Humanity can solve it's problems both the huge visible problems everyone talks about and the huge silent problems we've learned to take for granted. If the nature of the world we live in bothers you, there is something rational you can do about it. We can do better with your support.

Don't be a bystander at the Singularity. You can direct your effort at the point of greatest impact, the beginning.

Vernor Vinge

Department of Mathematical Sciences

San Diego State University

(c) 1993 by Vernor Vinge

(This article may be reproduced for noncommercial purposes if it is copied in it's entirety, including this notice.)

The original version of this article was presented at the VISION-21 Symposium sponsored by NASA Lewis Research Center and the Ohio Aerospace Institute, March 30-31, 1993. A slightly changed version appeared in the Winter 1993 issue of Whole Earth Review.

Abstract

Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.

Is such progress avoidable? If not to be avoided, can events be guided so that we may survive? These questions are investigated. Some possible answers (and some further dangers) are presented.

What is The Singularity?

The acceleration of technological progress has been the central feature of this century. I argue in this paper that we are on the edge of change comparable to the rise of human life on Earth. The precise cause of this change is the imminent creation by technology of entities with greater than human intelligence. There are several means by which science may achieve this breakthrough (and this is another reason for having confidence that the event will occur):

There may be developed computers that are awake and superhumanly intelligent. (To date, there has been much controversy as to whether we can create human equivalence in a machine. But if the answer is yes, we can, then there is little doubt that beings more intelligent can be constructed shortly thereafter.)

Large computer networks (and their associated users) may wake up as a superhumanly intelligent entity.

Computer human interfaces may become so intimate that users may reasonably be considered superhumanly intelligent.

Biological science may provide means to improve natural human intellect.

The first three possibilities depend in large part on improvements in computer hardware. Progress in computer hardware has followed an amazingly steady curve in the last few decades [17]. Based largely on this trend, I believe that the creation of greater than human intelligence will occur during the next thirty years. (Charles Platt [20] has pointed out that AI enthusiasts have been making claims like this for the last thirty years. Just so I'm not guilty of a relative time ambiguity, let me more specific: I'll be surprised if this event occurs before 2005 or after 2030.)

What are the consequences of this event? When greater than human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent entities, on a still shorter time scale. The best analogy that I see is with the evolutionary past: Animals can adapt to problems and make inventions, but often no faster than natural selection can do it's work, the world acts as it's own simulator in the case of natural selection. We humans have the ability to internalize the world and conduct what if's in our heads; we can solve many problems thousands of times faster than natural selection. Now, by creating the means to execute those simulations at much higher speeds, we are entering a regime as radically different from our human past as we humans are from the lower animals.

From the human point of view this change will be a throwing away of all the previous rules, perhaps in the blink of an eye, an exponential runaway beyond any hope of control. Developments that before were thought might only happen in "a million years" (if ever) will likely happen in the next century. (In [5], Greg Bear paints a picture of the major changes happening in a matter of hours.)

I think it's fair to call this event a singularity (the Singularity for the purposes of this paper). It is a point where our old models must be discarded and a new reality rules. As we move closer to this point, it will loom vaster and vaster over human affairs till the notion becomes a commonplace. Yet when it finally happens it may still be a great surprise and a greater unknown. In the 1950s there were very few who saw it: Stan Ulam [28] paraphrased John Von Neumann as saying:

One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.

Von Neumann even uses the term singularity, though it appears he is thinking of normal progress, not the creation of superhuman intellect. (For me, the superhumanity is the essence of the Singularity. Without that we would get a glut of technical riches, never properly absorbed (see [25]).)

In the 1960s there was recognition of some of the implications of superhuman intelligence. I.J. Good wrote [11]:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an intelligence explosion, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the _last_ invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. ...It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make.

Good has captured the essence of the runaway, but does not pursue it's most disturbing consequences. Any intelligent machine of the sort he describes would not be humankind's tool, any more than humans are the tools of rabbit's or robins or chimpanzees.

Through the '60s and '70s and '80s, recognition of the cataclysm spread [29] [1] [31] [5]. Perhaps it was the science fiction writers who felt the first concrete impact. After all, the hard science fiction writers are the ones who try to write specific stories about all that technology may do for us. More and more, these writers felt an opaque wall across the future. Once, they could put such fantasies millions of years in the future [24]. Now they saw that their most diligent extrapolations resulted in the unknowable... soon. Once, galactic empires might have seemed a Post Human domain. Now, sadly, even interplanetary ones are.

What about the '90s and the '00s and the '10s, as we slide toward the edge? How will the approach of the Singularity spread across the human world view? For a while yet, the general critics of machine sapience will have good press. After all, till we have hardware as powerful as a human brain it is probably foolish to think we'll be able to create human equivalent (or greater) intelligence. (There is the far fetched possibility that we could make a human equivalent out of less powerful hardware, if we were willing to give up speed, if we were willing to settle for an artificial being who was literally slow [30]. But it's much more likely that devising the software will be a tricky process, involving lots of false starts and experimentation. If so, then the arrival of self aware machines will not happen till after the development of hardware that is substantially more powerful than humans' natural equipment.)

But as time passes, we should see more symptoms. The dilemma felt by science fiction writers will be perceived in other creative endeavors. (I have heard thoughtful comic book writers worry about how to have spectacular effects when everything visible can be produced by the technologically commonplace.) We will see automation replacing higher and higher level jobs. We have tools right now (symbolic math programs, cad/cam) that release us from most low level drudgery. Or put another way: The work that is truly productive is the domain of a steadily smaller and more elite fraction of humanity. In the coming of the Singularity, we are seeing the predictions of _true_ technological unemployment finally come true.

Another symptom of progress toward the Singularity: ideas themselves should spread ever faster, and even the most radical will quickly become commonplace. When I began writing science fiction in the middle '60s, it seemed very easy to find ideas that took decades to percolate into the cultural consciousness; now the lead time seems more like eighteen months. (Of course, this could just be me losing my imagination as I get old, but I see the effect in others too.) Like the shock in a compressible flow, the Singularity moves closer as we accelerate through the critical speed.

And what of the arrival of the Singularity itself? What can be said of it's actual appearance? Since it involves an intellectual runaway, it will probably occur faster than any technical revolution seen so far. The precipitating event will likely be unexpected, perhaps even to the researchers involved. (But all our previous models were catatonic! We were just tweaking some parameters....) If networking is widespread enough (into ubiquitous embedded systems), it may seem as if our artifacts as a whole had suddenly wakened.

And what happens a month or two (or a day or two) after that? I have only analogies to point to: The rise of humankind. We will be in the Post Human era. And for all my rampant technological optimism, sometimes I think I'd be more comfortable if I were regarding these transcendental events from one thousand years remove...instead of twenty.

Can the Singularity be Avoided?

Well, maybe it won't happen at all: Sometimes I try to imagine the symptoms that we should expect to see if the Singularity is not to develop. There are the widely respected arguments of Penrose [19] and Searle [22] against the practicality of machine sapience. In August of 1992,Thinking Machines Corporation held a workshop to investigate the question How We Will Build a Machine that Thinks [27]. As you might guess from the workshop's title, the participants were not especially supportive of the arguments against machine intelligence. In fact, there was general agreement that minds can exist on nonbiological substrates and that algorithms are of central importance to the existence of minds. However, there was much debate about the raw hardware power that is present in organic brains. A minority felt that the largest 1992 computers were within three orders of magnitude of the power of the human brain. The majority of the participants agreed with Moravec's estimate [17] that we are ten to forty years away from hardware parity. And yet there was another minority who pointed to [7] [21], and conjectured that the computational competence of single neurons may be far higher than generally believed. If so, our present computer hardware might be as much as _ten_ orders of magnitude short of the equipment we carry around in our heads. If this is true (or for that matter, if the Penrose or Searle critique is valid), we might never see a Singularity. Instead, in the early '00s we would find our hardware performance curves beginning to level off, this because of our inability to automate the design work needed to support further hardware improvements. We'd end up with some _very_ powerful hardware, but without the ability to push it further. Commercial digital signal processing might be awesome, giving an analog appearance even to digital operations, but nothing would ever wake up and there would never be the intellectual runaway which is the essence of the Singularity. It would likely be seen as a golden age...and it would also be an end of progress. This is very like the future predicted by Gunther Stent. In fact, on page 137 of [25], Stent explicitly cites the development of transhuman intelligence as a sufficient condition to break his projections.

But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the threat and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of a machine in the likeness of the human mind [13]. In fact, the competitive advantage, economic, military, even artistic, of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first.

Eric Drexler [8] has provided spectacular insights about how far technical improvement may go. He agrees that superhuman intelligences will be available in the near future, and that such entities pose a threat to the human status quo. But Drexler argues that we can confine such transhuman devices so that their results can be examined and used safely. This is I.J. Good's ultraintelligent machine, with a dose of caution. I argue that confinement is intrinsically impractical. For the case of physical confinement: Imagine yourself locked in your home with only limited data access to the outside, to your masters. If those masters thought at a rate, say, one million times slower than you, there is little doubt that over a period of years (your time) you could come up with helpful advice that would incidentally set you free. (I call this fast thinking form of superintelligence weak superhumanity. Such a weakly superhuman entity would probably burn out in a few weeks of outside time. Strong superhumanity would be more than cranking up the clock speed on a human equivalent mind. It's hard to say precisely what strong superhumanity would be like, but the difference appears to be profound. Imagine running a dog mind at very high speed. Would a thousand years of doggy living add up to any human insight? (Now if the dog mind were cleverly rewired and _then_ run at high speed, we might see something different....) Many speculations about superintelligence seem to be based on the weakly superhuman model. I believe that our best guesses about the post Singularity world can be obtained by thinking on the nature of strong superhumanity. I will return to this point later in the paper.)

Another approach to confinement is to build _rules_ into the mind of the created superhuman entity (for example, Asimov's Laws [3]). I think that any rules strict enough to be effective would also produce a device whose ability was clearly inferior to the unfettered versions (and so human competition would favor the development of the those more dangerous models). Still, the Asimov dream is a wonderful one: Imagine a willing slave, who has 1000 times your capabilities in every way. Imagine a creature who could satisfy your every safe wish (whatever that means) and still have 99.9% of it's time free for other activities. There would be a new universe we never really understood, but filled with benevolent gods (though one of _my_ wishes might be to become one of them).

If the Singularity can not be prevented or confined, just how bad could the Post Human era be? Well... pretty bad. The physical extinction of the human race is one possibility. (Or as Eric Drexler put it of nanotechnology: Given all that such technology can do, perhaps governments would simply decide that they no longer need citizens!). Yet physical extinction may not be the scariest possibility. Again, analogies: Think of the different ways we relate to animals. Some of the crude physical abuses are implausible, yet.... In a Post Human world there would still be plenty of niches where human equivalent automation would be desirable: embedded systems in autonomous devices, self aware daemons in the lower functioning of larger sentients. (A strongly superhuman intelligence would likely be a Society of Mind [16] with some very competent components.) Some of these human equivalents might be used for nothing more than digital signal processing. They would be more like whales than humans. Others might be very human like, yet with a one sidedness, a _dedication_ that would put them in a mental hospital in our era. Though none of these creatures might be flesh and blood humans, they might be the closest things in the new environment to what we call human now. (I.J. Good had something to say about this, though at this late date the advice may be moot: Good [12] proposed a Meta Golden Rule, which might be paraphrased as Treat your inferiors as you would be treated by your superiors. It's a wonderful, paradoxical idea (and most of my friends don't believe it) since the game theoretic payoff is so hard to articulate. Yet if we were able to follow it, in some sense that might say something about the plausibility of such kindness in this universe.)

I have argued above that we cannot prevent the Singularity, that it's coming is an inevitable consequence of the humans' natural competitiveness and the possibilities inherent in technology. And yet... we are the initiators. Even the largest avalanche is triggered by small things. We have the freedom to establish initial conditions, make things happen in ways that are less inimical than others. Of course (as with starting avalanches), it may not be clear what the right guiding nudge really is:

Other Paths to the Singularity: Intelligence Amplification

When people speak of creating superhumanly intelligent beings, they are usually imagining an AI project. But as I noted at the beginning of this paper, there are other paths to superhumanity. Computer networks and human computer interfaces seem more mundane than AI, and yet they could lead to the Singularity. I call this contrasting approach Intelligence Amplification (IA). IA is something that is proceeding very naturally, in most cases not even recognized by it's developers for what it is. But every time our ability to access information and to communicate it to others is improved, in some sense we have achieved an increase over natural intelligence. Even now, the team of a PhD human and good computer workstation (even an off net workstation!) could probably max any written intelligence test in existence.

And it's very likely that IA is a much easier road to the achievement of superhumanity than pure AI. In humans, the hardest development problems have already been solved. Building up from within ourselves ought to be easier than figuring out first what we really are and then building machines that are all of that. And there is at least conjectural precedent for this approach. Cairns-Smith [6] has speculated that biological life may have begun as an adjunct to still more primitive life based on crystalline growth. Lynn Margulis (in [15] and elsewhere) has made strong arguments that mutualism is a great driving force in evolution.

Note that I am not proposing that AI research be ignored or less funded. What goes on with AI will often have applications in IA, and vice versa. I am suggesting that we recognize that in network and interface research there is something as profound (and potential wild) as Artificial Intelligence. With that insight, we may see projects that are not as directly applicable as conventional interface and network design work, but which serve to advance us toward the Singularity along the IA path.

Here are some possible projects that take on special significance, given the IA point of view:

Human computer team automation: Take problems that are normally considered for purely machine solution (like hill climbing problems), and design programs and interfaces that take a advantage of humans' intuition and available computer hardware. Considering all the bizarreness of higher dimensional hill climbing problems (and the neat algorithms that have been devised for their solution), there could be some very interesting displays and control tools provided to the human team member.

Develop human computer symbiosis in art: Combine the graphic generation capability of modern machines and the esthetic sensibility of humans. Of course, there has been an enormous amount of research in designing computer aids for artists, as labor saving tools. I'm suggesting that we explicitly aim for a greater merging of competence, that we explicitly recognize the cooperative approach that is possible. Karl Sims [23] has done wonderful work in this direction.

Allow human computer teams at chess tournaments. We already have programs that can play better than almost all humans. But how much work has been done on how this power could be used by a human, to get something even better? If such teams were allowed in at least some chess tournaments, it could have the positive effect on IA research that allowing computers in tournaments had for the corresponding niche in AI.

Develop interfaces that allow computer and network access without requiring the human to be tied to one spot, sitting in front of a computer. (This is an aspect of IA that fits so well with known economic advantages that lots of effort is already being spent on it.)

Develop more symmetrical decision support systems. A popular research product area in recent years has been decision support systems. This is a form of IA, but may be too focused on systems that are oracular. As much as the program giving the user information, there must be the idea of the user giving the program guidance.

Use local area nets to make human teams that really work (ie, are more effective than their component members). This is generally the area of groupware, already a very popular commercial pursuit. The change in viewpoint here would be to regard the group activity as a combination organism. In one sense, this suggestion might be regarded as the goal of inventing a Rules of Order for such combination operations. For instance, group focus might be more easily maintained than in classical meetings. Expertise of individual human members could be isolated from ego issues such that the contribution of different members is focused on the team project. And of course shared data bases could be used much more conveniently than in conventional committee operations. (Note that this suggestion is aimed at team operations rather than political meetings. In a political setting, the automation described above would simply enforce the power of the persons making the rules!)

Exploit the worldwide Internet as a combination human machine tool. Of all the items on the list, progress in this is proceeding the fastest and may run us into the Singularity before anything else. The power and influence of even the present-day Internet is vastly underestimated. For instance, I think our contemporary computer systems would break under the weight of their own complexity if it weren't for the edge that the USENET group mind gives the system administration and support people! The very anarchy of the worldwide net development is evidence of it's potential. As connectivity and bandwidth and archive size and computer speed all increase, we are seeing something like Lynn Margulis' [15] vision of the biosphere as data processor recapitulated, but at a million times greater speed and with millions of humanly intelligent agents (ourselves).

The above examples illustrate research that can be done within the context of contemporary computer science departments. There are other paradigms. For example, much of the work in Artificial Intelligence and neural nets would benefit from a closer connection with biological life. Instead of simply trying to model and understand biological life with computers, research could be directed toward the creation of composite systems that rely on biological life for guidance or for the providing features we don't understand well enough yet to implement in hardware. A longtime dream of science fiction has been direct brain to computer interfaces [2] [29]. In fact, there is concrete work that can be done (and is being done) in this area:

Limb prosthetics is a topic of direct commercial applicability. Nerve to silicon transducers can be made [14]. This is an exciting, near term step toward direct communication.

Direct links into brains seem feasible, if the bit rate is low: given human learning flexibility, the actual brain neuron targets might not have to be precisely selected. Even 100 bit's per second would be of great use to stroke victims who would otherwise be confined to menu driven interfaces.

Plugging in to the optic trunk has the potential for bandwidths of 1 Mbit second or so. But for this, we need to know the fine scale architecture of vision, and we need to place an enormous web of electrodes with exquisite precision. If we want our high bandwidth connection to be _in addition_ to what paths are already present in the brain, the problem becomes vastly more intractable. Just sticking a grid of high-bandwidth receivers into a brain certainly won't do it. But suppose that the high bandwidth grid were present while the brain structure was actually setting up, as the embryo develops. That suggests:

Animal embryo experiments. I wouldn't expect any IA success in the first years of such research, but giving developing brains access to complex simulated neural structures might be very interesting to the people who study how the embryonic brain develops. In the long run, such experiments might produce animals with additional sense paths and interesting intellectual abilities.

Originally, I had hoped that this discussion of IA would yield some clearly safer approaches to the Singularity. (After all, IA allows our participation in a kind of transcendence.) Alas, looking back over these IA proposals, about all I am sure of is that they should be considered, that they may give us more options. But as for safety ... well, some of the suggestions are a little scary on their face. One of my informal reviewers pointed out that IA for individual humans creates a rather sinister elite. We humans have millions of years of evolutionary baggage that makes us regard competition in a deadly light. Much of that deadliness may not be necessary in today's world, one where losers take on the winners' tricks and are coopted into the winners' enterprises. A creature that was built _de novo_ might possibly be a much more benign entity than one with a kernel based on fang and talon. And even the egalitarian view of an Internet that wakes up along with all mankind can be viewed as a nightmare [26].

The problem is not simply that the Singularity represents the passing of humankind from center stage, but that it contradicts our most deeply held notions of being. I think a closer look at the notion of strong superhumanity can show why that is.

Strong Superhumanity and the Best We Can Ask for

Suppose we could tailor the Singularity. Suppose we could attain our most extravagant hopes. What then would we ask for: That humans themselves would become their own successors, that whatever injustice occurs would be tempered by our knowledge of our roots. For those who remained unaltered, the goal would be benign treatment (perhaps even giving the stay-behinds the appearance of being masters of godlike slaves). It could be a golden age that also involved progress (overleaping Stent's barrier). Immortality (or at least a lifetime as long as we can make the universe survive [10] [4]) would be achievable.

But in this brightest and kindest world, the philosophical problems themselves become intimidating. A mind that stays at the same capacity cannot live forever; after a few thousand years it would look more like a repeating tape loop than a person. (The most chilling picture I have seen of this is in [18].) To live indefinitely long, the mind itself must grow... and when it becomes great enough, and looks back... what fellow feeling can it have with the soul that it was originally? Certainly the later being would be everything the original was, but so much vastly more. And so even for the individual, the Cairns-Smith or Lynn Margulis notion of new life growing incrementally out of the old must still be valid.

This problem about immortality comes up in much more direct ways. The notion of ego and selfawareness has been the bedrock of the hardheaded rationalism of the last few centuries. Yet now the notion of self awareness is under attack from the Artificial Intelligence people (selfawareness and other delusions). Intelligence Amplification undercuts our concept of ego from another direction. The post Singularity world will involve extremely high bandwidth networking. A central feature of strongly superhuman entities will likely be their ability to communicate at variable bandwidths, including ones far higher than speech or written messages. What happens when pieces of ego can be copied and merged, when the size of a selfawareness can grow or shrink to fit the nature of the problems under consideration? These are essential features of strong superhumanity and the Singularity. Thinking about them, one begins to feel how essentially strange and different the Post Human era will be, _no matter how cleverly and benignly it is brought to be_.

From one angle, the vision fits many of our happiest dreams: a time unending, where we can truly know one another and understand the deepest mysteries. From another angle, it's a lot like the worst case scenario I imagined earlier in this paper.

Which is the valid viewpoint? In fact, I think the new era is simply too different to fit into the classical frame of good and evil. That frame is based on the idea of isolated, immutable minds connected by tenuous, low bandwith links. But the post Singularity world _does_ fit with the larger tradition of change and cooperation that started long ago (perhaps even before the rise of biological life). I think there _are_ notions of ethics that would apply in such an era. Research into IA and high-bandwidth communications should improve this understanding. I see just the glimmerings of this now [32]. There is Good's Meta Golden Rule; perhaps there are rules for distinguishing self from others on the basis of bandwidth of connection. And while mind and self will be vastly more labile than in the past, much of what we value (knowledge, memory, thought) need never be lost. I think Freeman Dyson has it right when he says [9]: "God is what mind becomes when it has passed beyond the scale of our comprehension."

[I wish to thank John Carroll of San Diego State University and Howard Davidson of Sun Microsystems for discussing the draft version of this paper with me.]

17.42 // Causes and Prevention of War // MIT //Stephen Van Evera MIT OpenCourseWare

17.42 Causes and Prevention of War Spring 2009

VII. SOCIAL SCIENCE AND THE CAUSES OF WAR

A. Testing social science theories.

1. Observation vs. experimentation.

2. Large n (statistical) method vs. case study method.

B. Criticisms of social science.

1. Accidents drive history, butterflies cause hurricanes in history, hence general theories cannot explain much. When Annie Oakley was on tour in Europe before 1914 she had a shot at Kaiser Wilhelm. What if....?

2. Each historical event is unique; hence generalization is futile, even misleading, a claim made by historians. Implied: politics has no laws of motion.

3. Human will defeats our effort to generalize about human conduct.Once we know what people will do, they'll change their minds. But will they?

4. Social data is bad, hence social science has no reliable empirical basis. This is sometimes true but often not. Compare social science with paleontology: we can interview our subjects and read their memoirs; physical scientists can't. So let's stop whining.

C. Controversies in social science about how to do it.

1. Social scientists waste much energy debating what is the best way to measure the association between measures of variables. Some claim that large n methods are inherently stronger than case methods. This claim is false. Neither method is inherently stronger. Rather, the stronger method is the one that suits the data available. This varies with the subject being studied.

2. In the history field the study of international history and military history is being destroyed and replaced by a hegemonic study of race, class and gender issues. Questions of race, class and gender are very important, but so are international and military history! It's demise is harming our ability to study the causes and prevention of war.