The purpose of his paper is to ask what happens when AI systems are given goals (like “Make humans happy”) and also given the ability to modify their own design … including the ability to modify the goals themselves.

In the following analysis I want to focus on the opening definition offered in that paper, and its relationship to everything that comes after it.

Goertzel begins by asking “What does it mean for system S to possess goal G over time interval T?”.

His suggested answer to this question makes the assumption that there exists an “…observer O, who is hopefully a smart guy”. He proposes the following definition, couched in terms of that observer:

“S possesses goal G over time interval T if, to O, it appears that the actions S takes during T are significantly more probable to lead to the maximization of G than random actions S would be able to take.”

Let’s analyze this.

The definition says that if we compare S’s actions (in time interval T) - call this set of actions A[T] - with some randomly chosen set of alternative actions - call this set A’[T] - and if the comparison shows that A[T] are significantly more likely to lead to a maximization of G than A’[T], then the system will be deemed to have possessed goal G.

But this entirely depends on the intellect and subjective judgement of the observer O. So much so, that we might as well say that S has the goal G if the observer believes that S has the goal G - an empty definition, to say the least.

More specifically, there are at least five points on which the definition depends on subjectivity in the observer:

1) The actions A[T] have to be enumerated and compared with a random set A’[T] …. but ‘actions’ are not quantum entities, nor are they always obvious to an observer. What if the observer does not notice that a particular action occurred, either because it was too subtle or too abstract?  If King Henry says “Who will rid me of this meddlesome priest?”, does that count as an action on his part, or was he just thinking aloud to himself?  Ordinarily, Henry can say things and the mere use of words causes things to happen in the world, so his words are, in a real sense, his actions, but in this cases who is going to make the call on whether this was an action or an unfortunate soliloquy?

2) What is the meaning of a “significant” difference between the outcomes of the two sets of actions? Just how significant does something have to be, to be labelled ’significant’ by our observer?  What if the only action I take in time T is to toss a coin once and say ‘heads’, and I actually do get heads?  If the other randomly chosen actions are to toss the coin in different ways, then is there a significant difference between the result I got that first time (a solid, indisputable, triumphant heads) and all the other times, which on average come out 50-50?  What I am getting at with this coin example is that the process of seeing a ’significant’ difference depends on exactly how the observer chooses to model the situation.

3) What exactly is the “maximization” of a goal? The best possible achievement of the goal in all possible worlds? Some goals can be stated only in qualitative terms, not quantitative terms, so the observer may be in a position of having to make subtle judgements about the relative merits of different kinds of goal achievement. Is it better to maximize happiness by making absolutely sure that no person ever experiences a degree of happiness that deviates from the average by more than 0.01%, on a particular measure? Or would maximization of happiness allow greater variance with a higher average? The more abstract the goal, the more meaningless it is to speak of its maximization.

4) And the actions must be judged, by the observer, to “lead” to the maximization of the goal. Who is to say how actions are causally connected to outcomes? Does a reduction in income tax for the wealthiest people lead to a ‘trickle-down’ increase in the overall wealth of the poor, or does it just lead to greater income disparity?

5) When exactly are the effects of the actions allowed to come into play? If the actions A[T] do not have a significant effect during time T, but then have a massive effect at [T + delta], does the observer ignore that fact and say that the system did not have goal G during time interval T (even though system S may verbally declare that it did have the goal, but did not expect it to have any result until later)?

Now, if the subsequent analysis offered by Goertzel, in this paper, were to result in a clarification of all these subjective aspects of the definition, then we might hope that the subjectivity was on the way to being reduced or eliminated. It would be okay, I think, to start off with some vagueness in your definition if the math that comes later is designed to eliminate that vagueness in a believable way.

But this is not what happens. Rather, the ideas embedded in the above definition (like what counts as a goal, and what counts as its maximization) are just left as primitives. The observer O is a crucial character in this paper, because O is supposed to be … you and me!  We are being asked to step into O’s shoes and buy the idea that we all agree roughly what it means for a system to have a goal.

But nothing could be further from the truth: primitive ideas like “goal” and “maximization” are a million miles away from being cast in objective terms, and so the mathematical analysis that comes later is built on nothing firmer than quicksand. The primitive terms in that opening definition beg so many questions that the later analysis cannot be said to go anywhere at all.

Now, to be fair, Goertzel is sanguine about how much he has achieved in this paper, saying quite honestly that “Essentially nothing has been resolved in the above discussion. What I hope is that I have raised some interesting questions.”

However, he then goes on to say that “My central goal here has been to replace vague conceptual questions about goal preservation in self-modifying systems with semantically similar questions that are at least somewhat more precise”. With this I respectfully but firmly disagree: I believe that he started with concepts that were so subjective and vague as to be of no use at all, then built a mathematical apparatus on top of that insecure foundation.

In my book the first thing you do is sort out your definitions. Then you do the math. Not the other way around.
Sadly, the very existence of the mathematical apparatus that Goertzel proposes will serve to disguise the fact that all of our attention, right now, should be directed at the insecure foundations. In the literature as a whole, the concept of a “goal” is bandied about as if everyone understood that this was a moderately well-defined concept. In fact it is anything but. It is all well and good to have philosophical discussions in which we take a kind of Turing-esque, hands-off approach and say that a goal is just what a “reasonably smart guy” would judge to be a goal, but this kind of philosophical handwaving is not going to cut the mustard if real systems need to be designed to do real intelligent things.

We still await a calculus of goals and motivation that is founded on basic concepts that are not defined in terms of subjective observers or homunculi.

When trying to decide if a given system is complex, it is important to be clear about some of the distinctions I made in the definition of complexity (yesterday’s post).

First, the strict definition of a complex system is that it has some observable behavior that can only be explained by a theory that is too large for us to discover (and possibly there is no explanation at all, except for simulating the entire system).  So the most basic criterion for complexity is the size of the theory that “explains” the system’s behavior.

Now before you jump on this idea, I need to add that in general you cannot know if a given system is complex!  To know for sure, you have to know the size of the theory that explains the system, but it only takes a moment’s thought to realize that this reduces to two possibilities, one of which is trivial and one of which is impossible.  If the system is understood (somebody has previously explained how the behavior is related to the mechanisms that drive the system), then we don’t even need to look at the size of the theory, because the very fact that theory exists tells us that the system is not complex.  That is trivial.  But if we do not have a theory of the system yet (if we cannot explain how the high-level behavior is explained by the low-level mechanisms), then we are completely in the dark.  We don’t know if a good theory is going to turn up tomorrow, or if scientists will struggle to understand this system for the next million years, and still not get anywhere.  This, of course, is an impossibly difficult situation to be in: strictly speaking, we can’t say anything because we do not know the size of the theory that explains the system (you can’t say anything about the size of something that you haven’t discovered yet).

So if someone brings up an example of a real world system (the F-14 fighter jet, or a craps table, for example) and wants to know if this fits my definition of “complexity”, my first reaction is to say we already understand how the system works, then the answer is a trivial “no”.  But if we do not understand (cannot explain) the system, then the strict answer is “we don’t know”.

That is not, of course, the end of the story (I would not talk about complex systems so much if it were), but it is very important to be clear about that first step.  Without that idea, endless confusion can result.  The definition of “complex” is an abstract one, and when it is taken in isolation it leads nowhere — the only person who can know for sure which systems are complex and which are not is someone outside the universe (the Divine Mother, as I put it before) who knows the size of all theories.

For this reason, then, I prefer not to be asked whether System X is an example of something that is complex, because if the person asking the question has not already taken on board the above concept then any attempt by me to answer the question can lead to tangled confusion.

Okay, now I am going to move on and assume that the basic concept is understood.

Focus on the ‘regularity’

I need to confront a small side issue that has to do with which part, or aspect of a system you want to ask questions about.

It is almost always possible to drill down into a system and find some complexity somewhere, so it is very important to focus on the “behavior” (or, in the terminology that I prefer, the “regularity”) of the system that is important to you.  What exactly is it, in this system, that is supposed to be in need of explanation?  Once you focus on that, it becomes easier to ask whether that behavior is understood (and, if it is not understood, whether it is complex).

For example, in the question about throwing dice at a craps table, are we asking for an explanation of exactly how the dice fall on a particular occasion, or are we asking about some statistical bias that we observe in many throws of the dice?  If we want to explain a particular throw, this is not really a regular behavior that is of any interest … the system has not shown a pattern of behavior that is non-random, because we are talking about a single event, and a single event is not a pattern.  Put it this way:  would a curious scientist say that the single event was a fascinating, inexplicable thing, and would she then stop and devote a couple of decades of serious research to trying to “explain” this single event?  Of course not:  she would just say that the trajectory of that one throw was too hard to measure, and she would add that there is nothing interesting in a single event anyway.

I hope this makes it a little clearer why we need to be careful to say exactly what is the regularity that we suspect might be complex.  Not the system in general, but the regularity (the behavior) that the system is exhibiting.  The complex system concept is about whether the things we observe can be explained

To put it more vividly, it does not make sense for someone to hold up an apple and say “Explain this.”  That is not a regularity or a behavior, it is a thing.  On the other hand, it does make perfect sense to hold up an apple and say “Explain how this can grow from a seed.”  In a similar fashion, it does not make much sense to present a system of some kind and simply ask if the system is a complex system.

The example of an F-14 fighter jet suffers from a similar problem.  At a low level, there are aspects of the flight behavior of this jet that are extremely unstable and, quite possibly, “complex” in the sense that they cannot be explained fully.  But what happens in an F-14 is that the computer works extremely hard to control the intrinsic instability of the machine.  Crucially, however, we notice that the engineers who designed the jet were able to write the software well enough to compensate for the instability (the complexity) of the underlying system.  So whatever that complexity was, it was simple and predictable enough that the control software could actually be written and the complexity could be cancelled out.  That means that the complexity was not really an important part of the design of the plane, it was more like a noise signal that the engineers managed to cancel out.  And, of course, the resultant system of [plane] plus [control software] was not a complex system at all:  the complete system was understood well enough to make it fly straight.  If it is understood, it is not complex.  One aspect of it might be complex, but the complexity in this case was deliberately nullified.

The lesson to be learned from this F-14 example is that there are two perspectives on the system.  The high level involves no complexity.  Down at a deeper level there are things going on that are possibly complex.  Which of these answers you give depends on which aspect of the system you want to discuss.

The only problem with this careful distinction I have just made between a system and a regularity displayed by the system, is the fact that I (like everyone else) will violate the distinction all the time and talk about “the system” when what I really mean is a particular regularity displayed by the system.  Unfortunately, we often need to use the shorthand form and just say “system”.  Just bear in mind that his almost always means a particular regularity (a particular behavior) of the system.

Systems that are not yet understood

Now back to the dilemma I laid out at the beginning.  If a system (which means a particular behavior of that system) is not already understood, how do we decide whether or not the system is complex?

What I am going to do at this point is to simplify the problem drastically.  If the system is on the borderline, I am just not interested. There is no point arguing about whether a fringe case is complex or not.  It only makes sense to target the really extreme cases, because there the situation is clear enough that we can say something meaningful.

This is why I talked about properties of systems that involve “untouchable” mathematics.  If the mechanisms that drive the system are so grossly ugly that they are not just intractable, but utterly unpleasant to a mathematician’s delicate sensibilities, then we can make a hypothesis that these mechanisms may never be solved.

So, for example, we can point to these characteristics:

- Memory.  Does the mechanism use stored information about what it was doing fifteen minutes ago, when it is making a decision about what to do now?  An hour ago?  A million years ago?  Whatever:  if it remembers, then it has memory.

- Development.  Does the mechanism change its character in some way over time?  Does it adapt?

- Identity.  Do individuals of a certain type have their own unique identities, so that the result of an interaction depends on more than the type of the object, but also the particular individuals involved?

- Nonlinearity.  Are the functions describing the behavior deeply nonlinear?

These four characteristics are enough. Go take a look at a natural system in physics, or an engineering system, and find one in which the components of the system interact with memory, development, identity and nonlinearity.  You will not find any that are understood.

Now go and make up some artificial systems that have all these properties, and look at their overall behavior.

Notice that it is not always random.  Notice that sometimes it is highly structured.  Notice that sometimes the stability (in some sense of ’stability’) is dependent on the local mechanisms, but we cannot explain exactly how.

Notice, above all, that no engineer has ever tried to persuade one of these artificial systems to conform to a pre-chosen overall behavior.  This is widely thoght to be an insanely impossible thing to try to do.

Then, finally, notice that the one example that we have of a really bad, seriously overachieving example of a system with all these properties is these things that we call “intelligent systems”.

And I ask you the question:  do we have any good reason to believe that, even though all four of these mischievous properties are present in intelligent systems, they are nevertheless not complex, and that therefore we can just treat them as if they are ordinary engineering systems like all the others that do not have these nasty properties?  I don’t want a hunch that everything is okay, I want a good reason why we can be sure that complexity is not, in fact, present in sufficient quantities to cause trouble.

I propose the following as a general definition (or explanation-plus-definition) of “complexity”, where that term is intended in the sense of “complex system”.

1) All systems in the world consist of a number of elements that interact with one another in some way (call these interactions the “local mechanisms”), and as a result the system as a whole has observable characteristics (also called the “global behavior” of the system).

2) Science is about finding systems, observing their global behavior, then trying to discover the local mechanisms that explain that global behavior. [Example: Newton observes the planets following elliptical paths, then explains this by proving that an inverse-square force of gravitation predicts those elliptical paths].

3) As far as we understand it today, the process of finding a scientific explanation is not a formal mathematical process, it is an activity of human minds. (Whether it will ever become a formal procedure in the future is a moot point, in the context of this definition).

4) One aspect of the scientific process is that some explanations are larger than others, in the sense that they take longer to discover, take more space to write down, and take longer to understand. As history progresses, the size of explanations do tend to shrink, but as far as we know these explanations will not all collapse to the same size (quantum electrodynamics will always take longer to understand than universal gravitation), nor will they ever become so small that (for example) human toddlers with today’s brain design will routinely learn QED a couple of weeks after learning to talk. I will refer to this subjective notion of the size of scientific explanations as “theory size”. (The fact that this idea of theory size is subjective, not formalized and measurable, turns out later to be a crucial part of the definition, not just an act of laziness).

5) It is almost tautologous to say that in the history of science so far, theory sizes have not been larger than one-lifetime size when they were initially discovered. But that prompts an interesting question: are there systems for which the simplest explanation is massively larger than that? For example, is there any system that would require a scientist to work full-time for a million years before she could discover the explanation for that system? And, by extension, we can also ask if there are systems for which the theory size is effectively infinite: systems in which there simply is no explanation except the low-level mechanisms?

6) A system is deemed “complex” if the smallest size of a theory that will explain that system is so large that, for today’s human minds, the discovery of that theory is simply not practical. Notice that this definition does not imply that there any such systems in the real world, it just says that if the theory size were ever to go off the scale then the system would (by definition) be complex.

7) The practical problem of deciding whether or not a given system is complex is not trivial. In fact, it is quite easy to show that there cannot be a practical decision procedure that will predict whether a given system is complex. Nevertheless, we can look at the real systems in the world and make two interesting observations. The first is that the vast majority of natural systems are governed by extremely simple local mechanisms, and as practicing scientists we have come across very few cases in which a system could not be analyzed by some means or other, so as to yield a satisfying explanation for its global behavior in terms of its local mechanisms.

The second observation is that we can easily construct, in computers, systems that have local mechanisms with a dramatically different character than the mechanisms seen in low-level aspects of the natural world. These bizarre characteristics can be summarized by the shorthand term “tangled and nonlinear”, but what this means is a combination of nonlinear, memory-dependent, developmental, identity-dependent, and so on (all these terms need to be discussed in more depth elsewhere, but the details are not important at this point). What is interesting about these artificial systems is that their global behavior is sometimes interesting and structured (i.e. not just random mush), but we have no idea, at the moment, how to analyze the local mechanisms to explain their global behavior. Indeed, the experience of most mathematicians is that we may never be able to analyze the local mechanisms to predict the global behavior: these types of system are so far outside the normal realms of mathematical analysis that our success rate on them is almost zero, and most mathematicians would regard them as virtually “untouchable” (my term).

9) Combining the empirical observation about “untouchable” systems with the previous abstract definition of “complex” systems, we come to the following hypothesis:

Hypothesis 1: Systems that are mathematically “untouchable”, and which have global behavior that is structured enough to demand an explanation, are overwhelmingly likely to be “complex”.

This concludes the definition of complexity. In speaking loosely about complex systems, we tend to make an immediate identification between untouchable systems and complex systems — calling them both complex systems — but the strict definition is that these are two separate ideas. It is just that in practice the above hypothesis makes it convenient to conflate the two.

As a final note, it is worth saying that the point of this analysis is that intelligent systems have many characteristics that make us suspect that they can only be built in such a way as to have some degree of untouchability in them, so we can go on to state a second hypothesis:

Hypothesis 2: Intelligent systems contain so many mechanisms that have untouchable aspects, that the problem of explaining intelligent systems (psychology) and the parallel problem of building an intelligent system (artificial intelligence) must both be tackled on the assumption that intelligent systems are complex. (The “Complex Systems Problem”)

Finally, a couple of points about how this relates to other definitions of complexity, and what it means for a system to be partially complex.

The notion of complexity defined here is not identical or derivable from the Chaitin-Kolmogorov notion of complexity, because the latter is based on a strictly formal notion of algorithm complexity, whereas the present definition is about scientific theory size. Part of the reason that these two are not identical is that a self-consistent consequence of the notion of complexity given here is that because scientific discovery is a process carried out by human brains, that process itself is likely to inherit sufficient aspects of complexity to mean that any definition of “theory size” that attempts to be precise and formal would render the present definition self-contradictory. (If you cannot analyze complex systems, how can you assume that the complexity-infected process of scientific discovery can be analyzed to yield a formal definition of “theory size”?).

Can a system be “partially” complex? Of course: systems often have many components and many different types of global behavior that (quite often) can be explained separately. It would be no surprise at all if some systems were a mixture of complex and non-complex. (To take a simple example, many characteristics of fluids can be explained in a non-complex way, but some aspects — such as Reynolds Number and the nature of vortices — appear to be a consequence of complex interactions.

But if a system can be partially complex, how do we measure the “amount” of complexity in the system? The short answer is that don’t: we have no reason to suppose that measuring the exact amount of complexity will ever be feasible. If we were able to plug the specification of a system into some algorithm, and the algorithm were to output a number describing the precise amount of complexity to be expected, or if it were to predict exactly where the complexity was going to manifest itself, the very existence of that algorithm would be a contradiction of the definition of a complex system. The very best we can do is to make an empirical observation about the amount of untouchableness in the characteristics of the system: if the local mechanisms look untouchable, then tread very, very carefully.

This is extremely frustrating, of course. But the frustration should not make us do anything foolish, like pretending that the issue might go away if we deny it.

You have heard of ordinary Artificial Intelligence, and you probably already understand that Artificial General Intelligence is a breakaway field that has split from AI. The AGI subfield is about trying to build complete thinking systems, whereas conventional AI stopped trying to do that a long time ago.

I am going to explain just two facts about AGI: why it has the potential to succeed where old AI did not, and why AGI, if it does succeed, is going to be an industry that makes the internet look as technologically cutting-edge as Grandma’s Olde Fashioned Gooseberry Marmalade.

First, then, why will AGI succeed where AI did not? To understand this you have to bear in mind the way that academic scientists and engineers work. To make it big in the academic world you have to publish papers, and what matters in these papers is not the quality of their content, but the sheer quantity of papers you produce. Of course, quality does matter to some extent, but very often “quality” means doing a polished piece of work that fits the mold that everyone has come to expect in your particular area. The worst thing you can do in the modern academic environment is to be brilliant, but publish only one epoch-making book every decade. That one book might count for only one point, whereas someone else who publishes 25 papers a year (be they ever so trashy) will be collecting 25 points a year. Guess who’s going to get the promotion, the research grants, the new job at a prestigious department?

What does this have to do with AI and AGI? Well, AI has always been a very tough frontier field where it is not easy to make noticeable progress, so if you are going to generate those 25 papers a year you had better give up any hope of actually thinking about the core issues that might make a difference. Instead, your best strategy is to solve a few mathematical puzzles, write a little code and then spin the work to make it look like it has something to do with intelligent systems. This is easy once you get the knack, and if you know the trendy issues well enough you can make a pretty good living that way.

The most disastrous strategy, if you are an AI professor in search of grants and a smoothly-greased career path, is to go for the brass ring and try to actually build an intelligent machine - a real, honest-to-goodness thinking computer of the sort that they promised in the early days of AI. Too big. Too risky. No guaranteed 25-papers-a-year payoff.

This has been going on for a long time in old-fashioned AI, but unlike other sciences (where the same publish-or-perish rules apply), in AI this has resulted in such dismal progress that people have had to go looking for excuses to explain why nothing seems to be happening. As a result, some people have come right out and said that AI is not really “about” building complete intelligent systems at all! People in the field have openly given up, and have started using the words “artificial intelligence” to mean something different than what it meant at the outset. For the last couple of decades (at least) the words have been used to mean “slightly fancy type of computer program”, plus there are a lot of excuses about why it is okay to use the words that way.

People in the new Artificial General Intelligence community are reacting to this abysmal trend. This fact should be obvious enough if you know the first thing about AGI, but what is more important, from the investors’ point of view, is that the AGI people are consciously breaking the mold, defying old taboos and thinking for themselves. In scientific and technological research, there is nothing so vibrant, so fertile, and so likely to cause a dramatic breakthrough as a gang of hot-headed revolutionaries.

This is why you should invest in AGI. The old AI folks gave up years ago, but the AGI community is lean, mean, hungry and hellbent on getting results.

Now to my second point, which is about what would happen if AGI succeeded. You’ve probably heard this story before, but it doesn’t hurt to tell it one more time.

If anyone can build a full-up, human level AGI, that system will be able to invent new knowledge by itself - new technology, new medicines, and, of course, new types of AGI that can function more quickly than the original AGI. If you can build one AGI that is as smart as the best medical researcher on the planet, you can duplicate that machine with all of its knowledge intact … something that has never been possible with human experts. And if you use AGI systems to develop better AGI systems, you could produce new systems that generate new discoveries much faster than we do. If, for example, the new systems function at one thousand times the speed of a human researcher, this would mean that new discoveries would start arriving at a rate of one thousand years of new science and technology per year.

This is wild, wild stuff, but (please!) don’t blame the wildness on pie-eyed AGI researchers, because this is just a regular part of the territory. AGI is just like that.  It is not like any other technology in history: a better mousetrap does not have the side effect of making everything else in the world happen faster, but a better AGI does exactly that.

But now when all is said and done, and you have listened to all this special pleading, you the investor still need to decide whether you might be wasting your money if you dumped $10 million on one of these renegade, revolutionary AGI outfits. Suppose the team you decide to back is a bunch of idiots? What if they burn all the cash and produce nothing?

You know what? It won’t matter to you at all.

Why not? Because at this stage of the game what your investment will do is to get the AGI buzz into high gear, and this buzz will soon lead to the birth of hundreds of AGI companies. And if there are that many, and just one of them connects with the ball, you personally will get the benefit of the flood of inventions and medicines that will come on stream a short time later.

Would you really care if it was somebody else’s team that made it big, if your investment indirectly caused the entire planet to get all of the technology of the year 3000, tomorrow? By throwing a modest chunk of cash at one of these upstart AGI companies, you help to stimulate an industry, put egg on the collective face of the old-fashioned AI people, and wake the world up to some dramatic new possibilities for the future.

But, heck, don’t take any notice of what I say.  Whatever you do, don’t invest in any AGI companies. Too risky by half. Waste of money. Give it to General Electric instead - they do mousetraps and missiles, which are the key to a better future. Don’t listen to me.

P.S. Do you think that AGI could actually have bad side effects? If you do, you need to contact me to schedule an interview. There are ways to do AGI (probably the only ways to do AGI) that would ensure that this technology is safer than Grandma’s Olde Fashioned Gooseberry Marmalade.

Hey, I’m serious: give me a call.

The purpose of the argument

My purpose is to explain that if the task of building an artificial intelligence involves trying to engineer a “complex system”, then we are in big trouble because all the methods currently used by AI researchers depend on the fact that intelligent systems are not complex systems.

Step 1 of the argument

Scientists have been trying to explain various aspects of the world for quite some time, and they have noticed that some explanations are longer and more difficult to write down than others. That may seem like an obvious point, but some people find it surprising.

For example, explaining how the wheel works is simpler than explaining why some things float, which is simpler than explaining Newton’s theory of gravitation, which is simpler than quantum electrodynamics, which is simpler than string theory… and so on.  Scientists have an informal idea of “theory size” - they don’t bother to talk about it much because it is usually of no importance, but the idea is there nonetheless. Roughly speaking, the size of a theory is the amount of paper that the average person would have to read and understand in order to get from no knowledge of any science, up to the level where they could understand the theory.

But this prompts a question that a scientist might ask. Could it be that there are systems in the universe that can only be explained by a theory that is so huge that we can never discover it? Or, could there be systems in the universe that we cannot explain because no theory can ever be found (of any size whatsoever) that could explain that system? We have encountered many things in the universe that we can explain, but by itself that means nothing: do we have any reason to believe that all systems are explicable in a reasonable amount of human time?

Keep that question in mind while we go to step 2.

Step 2 of the argument

Imagine that one day you go up to a mathematician and tell her that you just constructed a “system” (a bunch of objects that are having effects on one another), and that each of the objects in the system is following a rule that you invented, which you have written down on a single piece of paper. Before handing over the piece of paper, you ask the mathematician if she will be able to predict the behavior of the whole system, given only the description of the object-behavior that you wrote on the piece of paper. You specifically want to know if there is a pretty good chance that she will be able to predict the system, no matter how complicated and ugly the rule is.

The mathematician will laugh and say “Heck no! Unless your rule happens to be really simple, I wouldn’t even touch it”.

You are a bit discouraged, but you press the point. You ask what would happen if you got all the mathematicians in the world, and they did nothing but work on your rule for a thousand years. Would they then be able to say that no matter what the rule was, they’d have a pretty good chance of solving it (i.e. figuring out what the system as a whole would do)?

But her answer is just the same; “No way. The vast majority of rules that you could write down would be so ugly that no mathematician would even want to touch them. We don’t know for sure that they are unsolvable, but probably not.” She would go on to explain that if mathematicians were not allowed to choose their own problems to work on, but instead they were given problems by some outsider who made up systems at random, probably none of these systems would yet have been solved. There might be solutions to some of them, but they would be impossibly difficult to discover, or (more likely) most of the systems would not have any solutions.

But now you have another question. If we start building examples of these “untouchable” systems and actually observe how they behave, wouldn’t most of them just be behaving randomly, with nothing interesting to explain? The answer would again be negative: “Sorry, but no. That would be true of some of them, but you could easily find systems that had untouchable rules and very interesting behavior when you looked at the system as a whole.”

She would add that there is a name for systems that have untouchable rules but which nevertheless have “interesting” behavior: they are called complex systems.

Step 3 of the argument (the last step)

Back to the scientists we left in Step 1, who were wondering if there might be some things in the world that could only be explained by a theory of enormous (and overwhelming) size. A theory so big that we could never, in practice, discover it or write it down.

Now that you have had your chat with the mathematician, you go up to the scientists and say “Hey, there is no need to worry about the possibility of something having a humungous theory-size. I mean after all, you have studied lots of systems since the beginning of modern science, and even though the objects that make up those systems are sometimes following rules that those pessimistic mathematicians don’t like - the horribly nonlinear, tangled, “untouchable” rules - you scientists have still been able to explain how those systems worked. You must have come across plenty of untouchable-rule systems in your time, and if you can work with those systems even when the mathematicians have given up on them, surely you don’t have to worry.”

And then the scientists look at you strangely. “Uh… no, actually not. The overwhelming majority of the systems that we have scientifically explained have been nice and elegant, not “untouchable” at all. Down at the lowest levels, the universe seems to be made only of things that follow reasonably elegant rules. Or, if not elegant, then at least only slightly ugly.”

“So, what” you ask “would count as a really ugly kind of system?”

“Well, imagine that electrons were not all the same, but each one had its own unique characteristics and when it hit some other particle it reacted in a way that depended on which other particles it had hit recently, and what other particles happened to be in the neighborhood right now. If electrons had memory, and if they had individual preferences for particular other electrons which developed over time, that would be seriously ugly. If that were the way the universe worked then we’d probably have to go to the mathematicians and ask them if they have changed their mind about those really nasty, untouchable systems. And if they are right about how bad those untouchable systems are, then we would have to give up: this would be a real, honest-to-goodness example of chasing a theory that was too big to ever discover. Probably there would be no theory at all to explain particle physics, if electrons had complicated memories and preferences, and if they developed over time.”

You try to cheer the scientists up again: “But you have never come across a system like that, so maybe you never will, and there won’t be any theories that are too big. Why worry about something that will probably never happen?”

And then one scientist starts to look worried “Well, there is one example of a system like that.”

“What?”

“An artificial intelligence system. An AI is supposed to be made up of particles called “concepts” or “symbols”, and these symbols interact with each other in ways that seem to be exactly like those hypothetical electrons with memory and preferences, and which develop over time. If that is the way that intelligence works, then those AI folks are probably in deep trouble.  It might be impossible to build a theory that explains why a given set of low level mechanisms must give rise to a system that is intelligent.  And if they don’t have such a theory, and can never work towards one, I don’t know how they think they are choosing the low level mechanisms that they think will work…”

Unfortunately, the new area called “Artificial General Intelligence” - which is generally quite critical of old-style AI, and which is all about returning to the original goals of AI and trying to build a complete, human-level thinking machine - is using the same research techniques, so it is vulnerable to exactly the same problem.

The assumption that almost everyone makes is that intelligent systems are not “complex systems.” [The term “complex system” has a specific technical meaning that I will come to in a moment]. I should explain right away that not everyone would say out loud that intelligent systems are not complex systems, but regardless of what they say they are behaving as if this were the case. People are using research methods that can only work if there is no significant amount of complexity in intelligent systems, so whether they know it or not they are implicitly making this assumption.

The problem, though, is that there are substantial grounds for believing that this assumption is just plain wrong; that any system that claims to be a general, human-level artificial intelligence must contain a significant amount of complexity.

In this post I am going to explain what a complex system is and what it means to say that all AI systems must contain a significant amount of complexity. Then I am going to look at the implications of this idea. My conclusion is that people working in the AI field have severely underestimated the impact of complexity on the way that AI research is carried out, and that the task of building a real AI (a human-level system capable of general intelligence) will be virtually impossible unless we have a drastic revolution in the way that people do research in this field.

For people who like to watch the spectacle of a knock-down fight between two competing paradigms, the next few years should furnish some interesting entertainment. The “drastic revolution” that I think is necessary is something that is deeply repugnant to many people now working in the field, and it appears that the old school is going to fight this one right down to the wire. Already there are signs that some people will do anything and say anything to denigrate the idea that there is such a thing as a “complex systems problem.” Interesting times indeed.

Complex systems

What does it mean to say that intelligent systems must be “complex systems,” or that intelligence always involves an unavoidable amount of “complexity”?

As you might imagine, this is not about things being just complicated (as in, the opposite of simple): the term “complex system” has a specific technical meaning that refers to systems with a peculiar property. You can assume from now on that whenever I use the word complex, or speak of complexity, it always refers to this technical meaning.

The idea of a complex system is controversial and frustratingly difficult to pin down. Even the people who claim to be doing complex systems research do not agree on what the exact definition of a complex system should be. And there are some people outside the field who argue that it is all nonsense; that there is no such thing as a “complex system” at all, only a bunch of irresponsible scientists playing games with a vague idea.

I believe that in spite of all this controversy and confusion, it is possible to come up with a clean and very powerful definition of complexity. That is the good news. The bad news is that this definition contains some very bizarre characteristics that cut right to the heart of what it means to do science.

Rather than jump straight to the hardcore definition, I am going to work up to the full version in a couple of stages: first an informal illustration of what complexity means, then a definition, then a critical look at the definition to bring its strangeness out into the open.

An Informal Illustration of Complexity

Imagine that we were to put together a large network of computers and connect them in such a way that each one can only see a few of its close neighbors.

Figure 1: Network of computers with a small number of local connections per node (the connections from only one node are shown).

Now suppose that there is a set of rules (an algorithm) that describes what each computer is doing, and that all the computers follow the same algorithm. What does the algorithm do? Well, each node has a bunch of variables inside it (U, V, W, X, Y …), and what the algorithm does is to modify these variables in some way, as a result of what the node can see in its neighbors’ variables. Each node can only see its own variables and those of its connected neighbors.

The interesting thing is what happens when we make different choices for the algorithm that each node is following. And the most interesting thing is what happens when we choose some horrible, ugly algorithm that includes delays, gradients, peculiar cross-references between variables, and so on.

On the face of it you might think that if we use a messy algorithm the result will just be a lot of messy randomness in the U, V, W, X and Y variables. Sometimes this does happen, but not always.

In fact, if we experiment with many examples of different messy algorithms we eventually find that we can make two observations:

1) Sometimes, when we look at the overall behavior of the system, we see “regularities.” For example, if we take the U value at each node and use it to control the brightness of one pixel in an image, and if there are enough nodes (computers) to fill all the pixels in a generous-sized image, we might notice that as time progresses there are (e.g.) waves of brightness moving around in the image. We might find that when two of these waves hit each other a vortex is created for (say) exactly 20 seconds, then it stops. I am making up this wave+vortex behavior, of course, but that is the kind of weird thing that can happen.

2) The other observation is about the messy algorithms themselves. We know that these kinds of algorithm are so pathological that we cannot do any math to analyze and predict the behavior of the system. In other words (and this is a crucial point) there are absolutely no techniques that will allow us to look at the algorithm and predict that the system will show those waves and vortices.

Just to reinforce the point, it really is childishly easy to write down algorithms that are so intractable that no mathematician would want to touch them. For example, we are talking about algorithms that involve weird little dependencies like:

“Pick two neighbors at random, then pick two variables at random from each of these, and for the next 10 seconds try to make one of my variables (chosen at random, again) follow the average of those two as they were exactly 20 minutes ago, EXCEPT when neighbors 5 and 7 both show the same value of the V parameter, in which case drop this algorithm for the next 73 seconds and instead follow the substitute algorithm B….”

What is the conclusion to be drawn from this? The conclusion is that there are some systems that have interesting “global behavior” (the waves and vortices that we observe), but we cannot explain this global behavior in any rigorous way by looking at the “local mechanisms” (the messy algorithm inside each node) that drive the system. We can simulate the systems and notice the global stuff, but we cannot look at the algorithms first and derive a prediction that tells us what the global behavior will be. Systems that have this disconnect between the local and the global are complex systems.

How Does This Relate To Intelligence?

I will have a great deal more to say about how this connects to intelligence in a later post, but it might help if I give a quick example of how this illustration of a complex system maps onto the problem of building an AI.

In the example I just gave there were waves and vortices visible at the global level. If these waves and vortices correspond to the processes that we want to get up and running in our AI system (reasoning and learning processes, for example), then we might have a problem. We know what we want to have happen (we know what global behavior we want to see), but going from this desired behavior to the underlying mechanisms is difficult or impossible if the system is complex.

Many people react to this suggestion with skepticism. They would point out that an intelligence mechanism like reasoning is obviously not just the result of a mysterious mechanism underneath, so the comparison with my complex system is just not believable. Logical reasoning is not voodoo, it is more like a mechanism, and we can see a great deal of the structure that makes it work.

I am willing to grant this, to a certain extent, but it is important to understand that there is another, much more subtle and pernicious way to compare my example complex system with the mechanisms inside an AI. Imagine a variation on the system I described before, where it is not the waves and vortices themselves that we should be focussing on, but the stability of these waves and vortices. You see, the stability of the system’s global behavior can also count as a “global regularity” that we cannot explain in terms of the local mechanisms.

I will make this explicit with an example. Suppose we set up the same system that I described earlier, but this time we see waves and vortices that we can partially understand. We look at the underlying algorithm, and we find that this is not an example of the worst sort of complex system, because we can see a partial explanation for why the waves and vortices are occurring. But just when we start to believe that maybe this system is not complex at all, and that we might be able to build a complete, regular-science explanation for what is going on, we discover something annoying about the system. If we fiddle with some parts of the algorithm, the waves and vortices have a tendency to fade out after a few minutes, and the system descends into boring randomness. When we make the right choice for the algorithm design, we get waves and vortices that last indefinitely, but as soon as we modify the algorithm a little the system changes its character and the interesting global behavior only lasts for a few minutes and then goes away.

Most importantly, I want you to imagine that this sensitivity to the exact design of the algorithm is “complex”, because this one aspect of the behavior is completely inexplicable. We cannot see any way to predict what is the right choice of algorithm to get stability, we only know that the correct choice works and others do not.

This is a much more subtle type of complex system, notice. Lots of roughly-predictable behavior (the waves and vortices), but then some bits that are impossible to understand. This is a system that is partially complex.

You might be tempted to say, at this point, that this situation is easy enough to deal with: just make sure that we do lots of experimentation with the algorithm design, to find the right choice that makes the waves and vortices stable. Unfortunately, this way of thinking is upside down and backwards. There are going to be incalculable numbers of ways to build the part of the algorithm that makes the waves and vortices appear (the part that we partially understand), and also an incalculable number of ways to build the part of the algorithm that is responsible for the stability of the system - and it may well be that there is only one choice out of each of these combinations that gives stability! So you might spend the first ten billion years of your AI project trying out different mechanisms that produce the waves and vortices that you want, but for all of your choices in that first ten billion years, none of them can be made stable by any choice for the other part of the algorithm.

Imagine the frustration. You pick one choice for the part of the algorithm that gives some waves and vortices, then you start trying all the possible choices for the other part of the algorithm, the part that could in theory make it stable. But since your first choice is a dead end choice (because there is no choice for the other part that leads to stability), all your testing eventually comes to nothing. So you move on to the next choice for the first part, and the same thing happens. Ten billion years later you get lucky: for today’s choice there really is a design for the other component that will make the whole thing stable, and after fiddling with the parameters for a long time (a century?) you find one that works.

The problem is that if you pick an algorithm that looks like it might give rise to the waves and vortices that you want the system to show, you have no idea whether this will be the right one, and no idea what the right choice for the other component is. The complexity of the second component “infects” the first component, which otherwise looked like it was going to be a good bet.

Notice that this exactly describes what has been happening in AI for the last fifty years. People choose some mechanisms that almost make the system behave intelligently, but then they find that the mechanisms only work on toy problems, or they only work for a limited time and then degenerate, or it turns out to be difficult to find learning mechanisms that will allow the system to acquire its own knowledge without having to spoon-fed by programmers, or people have difficulty getting the parts of an intelligent system to coordinate with one another. Always, initial success followed by messy problems and stagnation. This is like my complex system example: it is easy enough to cobble together a rough algorithm that will make the waves and vortices appear for a short time, but making them stable just seems impossible.

So the actual progress made in the field of Artificial Intelligence is very much like he progress we would expect if the complex systems problem were real.

Much more could be said about how the complex systems idea relates to the methodology of AI, but what I have said so far should at least give you a feel for where this argument is going.

Most Systems Are Not Complex

Perhaps the most intriguing aspect of complex systems is that almost every system that has been studied by science over the past three hundred years is not complex. (Or, to be more careful, most natural systems do not have a significant degree of complexity in them; they are not dominated by complexity). When scientists started trying to understand the world they found that there are huge numbers of “regularities” out there that they could explain by finding local mechanisms and doing some mathematical analysis to show that the local mechanisms imply the global regularities.

The classic example of this is what Newton did: he noticed that the planets moved in ellipses and followed Kepler’s Laws – this was the global regularity to be explained. He then suspected that gravitational attraction might explain this, and was eventually able to show (after inventing the calculus!) that if all the matter in the universe followed an inverse square law of gravitation, this would imply that the planets had to move on elliptical orbits and follow Kepler’s Laws.

This dazzling chain of scientific thought (describe the global regularity, then suggest a possible local mechanism to explain it, then use mathematics to prove that the local mechanism does indeed explain the global regularity) has been repeated countless times in the history of science. It is an almost peculiar and wonderful fact about the universe, that so much of what we observe can be explained by this method. In other words, it is amazing (but true) that most things we see are actually not complex systems.

The big question then becomes:  what do we do, as scientists, if we come across a situation where the system we want to explain, or the system we want to build, is actually a big, hairy complex system?  What methods do we use if the overall behavior of the system cannot be mathematically explained in terms of its local mechanisms?  Will the scientific method work under those circumstances?  Can we engineer a system in which, for reasons we have no control over, the overall behavior of the target system is not mathematically explicable in terms of the mechanisms that drive it?

You might think that this kind of question had come up before in science or engineering, and that someone would have figured out some good answers to these questions.  Not so.  I believe that AI is the first occasion when engineers have confronted the issue (and, by extension, cognitive psychology is the first occasion when scientists have confronted the issue).

Before going on to hammer out an answer to these “What should we do?” questions, I want to stop and make the definition of complex system a little more water tight.  If you don’t care about this cleaning up operation you can skip to the next section.

A Better Definition

Suppose that it is possible to take every scientific or mathematical explanation that has ever been discovered and write it out in standard form.  The kind of standard form that I have in mind would involve something like a written account that started with the most basic facts you learn in grade school and including all the material needed to follow the explanations up to the point where the theory itself is completely described.  Even though this would be impossible to do in practice, we will suppose that the Divine Mother is able to look down on everything and can normalize every scientific explanation for the purpose of comparing the size of different explanations.

From that perspective, complex systems are just system in which explanations of their global regularities are so huge that we have no practical hope of ever discovering them.  And that includes cases where the explanation simply does not exist, so the size of the theory is undefined/infinite.

Most real explanations (Newton’s gravitational theory, etc.) have a theory-size that is pretty close to zero, in the sense that we can easily read all the books we need to in order to understand them.  The point of calling some systems “complex” is that they happen to have theory-sizes that are somewhere up in the billions-of-books range (or worse).  So, yes, there may one day be an explanation for why a particular complex system shows the overall pattern of behavior that it does, but for all practical purposes that explanation is out of our reach.  All we can do is observe the regularity and perhaps pseudo-explain it by making a simulation of the system and observing that the simulation and the real system do roughly the same thing.

Notice that although the cluster of computers that I described above would be a wicked example of a complex system, the biggest supercomputer in the world would not be complex at all if it followed a nice, well behaved algorithm.  So this is not about how complicated (in the sense of the opposite of simple) a thing is, because that supercomputer could be programmed with algorithms that most of us would describe as profoundly non-simple, but so long as the algorithms were (n some sense) well behaved, we could predict the overall behavior quite nicely.

In the real world, of course, systems are mixtures of complex and not-complex.  It may not be possible to quantify the amount of complexity, but there is a clearly a sense in which some systems are dominated by complexity, whereas others have just a touch of complexity in them (a few cases where the behavior is regular but inexplicable) while at the same time being mostly quite normal.

So, finally, here is a better definition of complexity.  A system contains a certain amount of complexity in it if it has some regularities in its overall behavior that are governed by mechanisms that are so tangled that, for all practical purposes, we must assume that we will never be able to find a closed-form explanation of how the global arises from the local.

The key idea is that we must assume that we cannot explain the regularities.  Some systems that we think are complex may turn out in the long run to have explanations.  This is fine:  we would expect this.  But the point of defining the concept of a complex system is that we need to handle a broad class of systems in which, on average, the theory sizes (as measured by the Divine Mother, or whoever else has all-knowing perspective on these things) must be assumed to be off the scale.

One last comment about why this definition is so unnatural.  Notice that if the idea of complex system is a valid one (if there really are systems whose explanation is off the scale, or infinite), then we will probably not be able to prove the existence of this class of systems, or prove any theorems that allows us to measure the “amount” of complexity in a given system.  Why not?  Because all such properties of complex systems actually fall, themselves, into the category of “global regularities that cannot be explained”.  So if complex systems really do exist, then by definition we will be unable to make many rigorously provable statements about these systems.

We may be able to observe characteristics of complex systems in general, and these observations may capture facts that (from the “Divine Mother perspective”) really are true, but we will not be able to know for sure that these really are true facts.

This topic of the godelian nature of complex systems is fascinating, but it will have to be postponed for another day.

the big lesson we need to take home from all of this is that we cannot know exactly how much of an impact complexity will have because we have no way to measure the “amount” of complexity, nor any way to say how much impact we get from a given amount of complexity. So if a system gives us reason to suspect that it is complex, we are very much in the dark and the best we can do is to tread carefully and make the default assumption that it really is complex.  The system itself may never give us definitive proof that it is complex – in fact, it may tease us by hovering on the brink of wicked complexity and tempt us to think that maybe we can get away with treating it as non-complex.  We may get into a situation where, lacking any convincing proof that complexity is playing a big role in the system’s behavior, we pretend that the complexity is not a problem, but then when we try to understand the system we may find it peculiarly difficult.

And the worst part of that trap, where a complex system teases us into thinking that perhaps it can be treated as non-complex, is that we might think that all of our problems with the system are purely the result of trying to deal with a non-complex system that just happens to be very complicated (very difficult to understand).  This would be a disastrous situation to get into, because we would run around in circles trying to find “normal” explanations for the system’s behavior, or trying to engineer the system using “normal” engineering techniques, while all the time these techniques can never give us closure.

Are Intelligent Systems Complex?

Now we come to the crucial question:  do we have reason to suspect that a significant amount of complexity is involved in all intelligent systems?

I could present lots of reasoning here, but instead I will resort to quoting AGI researcher. Ben Goertzel, in a message he posted to the AGI discussion list on 12/6/07:

“There is no doubt that complexity, in the sense typically used in dynamical-systems-theory, presents a major issue for AGI systems.”

(www.mail-archive.com/agi@v2.listbox.com/msg09053.htm)

I would like to take it as understood that researchers in Artificial General Intelligence accept the existence of complexity in AGI systems.

Next time I will take the argument from there and try to look at how big an impact this might have on the methods we use to try to build intelligent systems.

So, for example:

1. Instead of being the most dangerous technology ever created, it will be the safest by a very wide margin. The degree of safety will go so far beyond that of any technology created before that it will redefine the very meaning of the word ‘safe’.

2. Many people assume that if a machine could ever be made to think, this would somehow devalue us and allow scientists to say that our own minds are nothing but machines. Quite the reverse: I believe that science is about to realize that there is, after all, something special and inexplicable about the human mind, and that when an intelligent machine is built it becomes another mind, just like our own, with its own special and inexplicable properties. Building an artificial mind does not devalue us, it just adds to the number of conscious minds in the universe.

3. A similar, but subtly different point: there are also many people who believe that it will never be possible to build a computer that can really think the way we do. According to this way of thinking, all future robots will be limited by their programming and will only mimic human thought, not actually do the real, creative, original thinking that we do. This idea is simply wrong, and is caused by a misunderstanding about the way artificial intelligence systems are built. We have every reason to believe that future AI systems will think in ways that are no less valid than our own.

4. Perhaps the most damaging misunderstanding is that AI systems will have the potential to be just as aggressive, selfish, greedy, jealous (etc.) as humans. People tend to assume that if someone builds an artificial intelligence then the mere fact that it is intelligent will mean that it could easily do what a human child can do: it can appear very innocent at first, but then later on develop into a psychopathic monster. This is completely false. An intelligent machine could (and almost certainly would) be built in such a way that it was incapable of feeling any of the destructive motives and emotions that humans feel. An AI of this sort could no more develop into a monster than you could spontaneously grow a new leg.

5. Another, similar assumption is that AIs would develop desires and aspirations that would bring them into conflict with us. For example, it is sometimes said that the “natural” desire of an AI would be to acquire more computing power, and that it would push us to one side in its quest to get what it wanted. This assumption is just as mistaken as the previous one, for exactly the same reasons: we who design the machine in the first place are the ones who decide what it “desires”, and we could choose to make the AI completely empathic with our own aspirations. In other words, such an AI would want to help us and would shudder at the thought of competing with us in any way.

6. The next mistaken assumption is that when AI systems are built, there will be large numbers of different individuals and different types, and that eventually these AIs will “evolve” in some way. The idea behind this assumption is that no matter how much trouble we take to ensure that the first AI is well behaved, the process will be out of our hands from that point on, so eventually there will be a time when some other, more dangerous type of AI is built. The problem with this assumption is that there is not the slightest reason why there should be more than one type of AI, or any competition between individual AIs, or any evolution of their design. So after the first safe AI is built, the situation will stabilize completely and any further change will always occur in a controlled way that is consistent with the original design.

7. It is worth noting another idea that is just a variant of the above mistake – that life could remain just the way it is now, but with robots going about their daily life alongside us humans and competing with us for food, energy and jobs. There is no reason why this would happen, for exactly the same reasons stated in the previous paragraph. It is important to emphasize that this means the AIs would not quietly push the human race to one side, behaving like a new species and relegating humanity to extinction. There is no parallel whatsoever between the human dominance of other animal and plant species, and the situation of these robots with respect to us.

8. It is often assumed that there will be large numbers of robots, but they will all be controlled by different governments or corporations, and used as instruments of power. The main argument against this idea is that it would require an extremely unlikely combination of circumstances for this kind of situation to become established. The first artificial intelligence would have to be both smart and designed to be aggressive, but this combination would be almost impossible to pull off, even for a military organization. The long version of the argument against this idea is too long to summarize in one paragraph, but the bottom line is that even though this seems like a reasonable and plausible possibility for the future, it turns out to be deeply implausible when examined carefully.

9. Even when people make the most charitable assumptions about the actual AIs themselves, there is still a tendency to assume that the future will in some sense be non-human and mechanized. There is no reason for this. In fact, quite the opposite: there would be every reason to expect that most of the world would become very much less technological and much more natural than it is now. The effect of artificial intelligence will be to green the planet, not mechanize it.
This is my list of the nine most glaring discrepancies between what people think AI is about and what it is really about. It is not supposed to be a comprehensive list, but it is does give some idea of the scale of the problem.

Of course, the above list is just a very brief declaration of the difference between assumption and reality, not a complete explanation of why each issue. I have not said exactly why each of the assumptions is wrong, I have really only declared that they are wrong. In the future I propose to take each of these issues and write a more expansive essay that gives a detailed explanation of what real artificial intelligence is and what it will do to the world.

Before I get around to undoing all the black propaganda about artificial intelligence, however, I will be moving on to something that I consider to be a much more important topic in my next post. This has to do with what I said in the opening words of this one: I claimed that the misunderstanding of AI was one of the biggest problems facing the world today, and I really meant that. My point is that if the street-level understanding of AI were not so tainted by these negative ideas, more effort would be made to build safe and friendly AI systems that could help the world out of the mess that it is in right now. The list of misunderstandings above is one of two immense blockages that stop the world from realizing how important it is to get all hands on deck and start figuring out how to build a SAFAI.

The other immense blockage is the question of what approach to take when trying to actually build a safe and friendly AI. When somebody looks back on this time from the vantage point of a few decades from now, I think the most significant event they will write about is the day when someone finally wakes up and realizes that artificial intelligence research is being done in a ridiculous way at the present time, and that there is an alternative.

That alternative will be the subject of tomorrow’s post.

Rather than employ an architecture that has little in common with the way that thinking and reasoning occurs in the human mind, as most AI researchers tend to do, we are trying to stay as close as possible to the design of the human cognitive system.

There is a reason for this. We believe that a strong case can be made that all intelligent systems (natural and artificial) must contain a significant degree of complexity in them - where the term ‘complexity’ is used in the sense of ‘complex system’, rather than just ‘complicated’.

If this is correct then the methodology of both artificial intelligence and cognitive psychology is flawed. Specifically, the methods traditionally used by AI and cognitive psychology researchers will eventually get them into a diminishing-returns trap, with more and more effort being expended to make less and less progress, and with a serious possibility of never being able to reach the goal of building a human-level AI or developing a complete theory of the human mind.

This is a strong claim, but we believe it to be valid. The methods used in all other sciences depend for their effectiveness on the fact that most natural systems are not dominated by complexity. There is a strong probability that intelligent systems are indeed dominated by complexity, but this would mean that the AI/cognitive psychology field is in the unique position of being the only science or engineering domain in which complexity is dominant. This leads to a conclusion that is hard to swallow: some of the most fundamental methods of science and engineering may not work in this area.

What to do about this problem?

Well, solving this problem is what Surfing Samurai Robots is all about. SSR is developing a radical alternative to the methodologies currently used in AI and cognitive psychology. The new approach is referred to here as ‘theoretical psychology’, and one of its main components is the idea that AI can only succeed if it stays as close as possible to the design of the human mind. So our decision to build human-like AI systems is not just driven by an urge to explore a new avenue, it is driven by necessity: if the complex systems problem is real then this is the only way to build a complete, human-level artificial intelligence.

The Possibility of Rapid Progress

We also believe that theoretical psychology will be the start of a new era of rapid progress in AI.

The reason for this optimism is that something similar to the TP approach has been tried before, when cognitive scientists and AI researchers embraced the new philosophy known as ‘connectionism’ or ‘parallel distributed processing’ (PDP) back in the mid-1980s. At that time there was a sudden burst of rapid progress, which then flattened out as the new ideas became more mature. From the point of view of theoretical psychology this sudden burst of progress would make sense: the elements in common between PDP/connectionism and theoretical psychology were strong in the early days, but these common elements were then weakened and diluted as those doing the research tried to push the field away from complex systems and back toward conventional, non-complex science.

If the progress made in the early days of the PDP/connectionist revolution was an indication of what happens when complexity is allowed to play a role in the construction of cognitive models and AI systems, then what we need is a more wholehearted and comprehensive move toward more complexity. This is exactly what theoretical psychology is all about. One reason we are exploring this new approach so vigorously is that we believe it could unlock the floodgates and bring a new period of rapid progress in AI.

In fact, it may turn out that the goal of building a complete, human-level AI is not, after all, an extremely difficult problem that will take decades or centuries to solve. Perhaps it only looked difficult for as long as we were using methods and techniques that could never have worked.

It could be that a viable human-level AI is within easy reach, but located around a corner where nobody ever thought to look before.

Safe and Friendly Artificial Intelligence

One of the other implications of the theoretical psychology approach is that it opens up the real possibility that AI systems can be built in such a way that there will be virtually no danger that they will ever get out of control.

The meaning of this statement may not be altogether clear, so here is a more specific version. Imagine that you have just collected together every science fiction story in which the robots get out of control, or have some kind of accident, or commit some act that goes against the wishes of the majority of ‘reasonable’ people. Now add to this collection all of the nonfiction articles and documentaries in which people have described the many ways in which artificial intelligence could go wrong. Now imagine that the chance of anything resembling one of these nightmare scenarios is reduced to such a low level that even if these AI systems were to stay around for billions of years, the probability would be so low as to be insignificant. That is the meaning of ‘virtually no danger that they would get ever out of control’.

From now on, any phrase that says something like “This type of AI will not do X…” should be taken to mean the same as “The probability that this type of AI will do X is so low that it is not worth bothering with…”. The word ‘will’ in these contexts is just being used as a shorthand for the long and cumbersome (but more accurate) phrasing that talks about vanishingly small probabilities.

This is not to be confused with the idea of proving that the AI will not do anything dangerous, where ‘prove’ means reducing the probability to a strict zero. Nothing in the real world is provable at that level. Even a provably correct algorithm has to be run on a real computer which could get hit by an inconvenient cosmic ray.

There are two aspects to the idea of not posing a danger: safety and friendliness. An AI is safe if it will not commit accidental damage. An AI is friendly if it will not intentionally do anything to harm or seriously annoy us. The two concepts are orthogonal: in theory an AI could be friendly but accident-prone, or it could be immune to accidents but unfriendly. What we want, of course, is a way to build AI systems that are both safe and friendly.

To wrap all of this up, the term ‘Safe and Friendly AI’ (or SAFAI) will henceforth signify a type of AI that conforms to the strict standards of non-dangerousness described above, including both the safety and the friendliness aspects.

Of course it is one thing to define the meaning of SAFAI, quite another to show that such a thing is possible. There is not enough space in this introductory post to explain the mechanics of achieving SAFAI, but it is nevertheless important to declare that this is one of the most important features of the theoretical psychology approach to AI.

The reason that SAFAI is possible is that the behavior of this type of AI is governed by large numbers of distributed constraints, rather than by the localized (and vulnerable) control system that is usually assumed to be the controller of a conventional AI system. Any system governed by large numbers of constraints is more stable, other things being equal, and the safety and friendliness of our preferred type of AI is, at root, derived from this same general principle.

What is Surfing Samurai Robots Up To?

At the moment, the primary goal of SSR is to build a software development environment to explore the properties of AGI systems that follow the theoretical psychology approach.

This development environment goes by the name of SAFAIRE (which stands for ‘Safe and Friendly AI Research Environment’, and is pronounced sapphire).

SAFAIRE is currently scheduled for general release in 2010.

Although the easiest way to describe SAFAIRE is to call it a software tool, it is also an AGI system in its own right. The distinction between tool and actual AGI is sufficiently blurred that it makes no sense to have two names. In the future, of course, other AGI instantiations will be built using SAFAIRE, but for the time being the one term will be used for both tool and (initial) target system.

The SSR Blog

The function of this blog will be to describe aspects of the work done at SSR, and to answer questions and comments about topics related to this work.

Comments and questions are welcome, within the usual limits of constructive engagement .