The Complex Systems Problem (Part 1)
Posted on April 10, 2008
Filed Under Artificial Intelligence, General |
I believe that the field of Artificial intelligence is in serious trouble, because the research techniques used by people working in that field are all based on an assumption that is not valid.
Unfortunately, the new area called “Artificial General Intelligence” - which is generally quite critical of old-style AI, and which is all about returning to the original goals of AI and trying to build a complete, human-level thinking machine - is using the same research techniques, so it is vulnerable to exactly the same problem.
The assumption that almost everyone makes is that intelligent systems are not “complex systems.” [The term “complex system” has a specific technical meaning that I will come to in a moment]. I should explain right away that not everyone would say out loud that intelligent systems are not complex systems, but regardless of what they say they are behaving as if this were the case. People are using research methods that can only work if there is no significant amount of complexity in intelligent systems, so whether they know it or not they are implicitly making this assumption.
The problem, though, is that there are substantial grounds for believing that this assumption is just plain wrong; that any system that claims to be a general, human-level artificial intelligence must contain a significant amount of complexity.
In this post I am going to explain what a complex system is and what it means to say that all AI systems must contain a significant amount of complexity. Then I am going to look at the implications of this idea. My conclusion is that people working in the AI field have severely underestimated the impact of complexity on the way that AI research is carried out, and that the task of building a real AI (a human-level system capable of general intelligence) will be virtually impossible unless we have a drastic revolution in the way that people do research in this field.
For people who like to watch the spectacle of a knock-down fight between two competing paradigms, the next few years should furnish some interesting entertainment. The “drastic revolution” that I think is necessary is something that is deeply repugnant to many people now working in the field, and it appears that the old school is going to fight this one right down to the wire. Already there are signs that some people will do anything and say anything to denigrate the idea that there is such a thing as a “complex systems problem.” Interesting times indeed.
Complex systems
What does it mean to say that intelligent systems must be “complex systems,” or that intelligence always involves an unavoidable amount of “complexity”?
As you might imagine, this is not about things being just complicated (as in, the opposite of simple): the term “complex system” has a specific technical meaning that refers to systems with a peculiar property. You can assume from now on that whenever I use the word complex, or speak of complexity, it always refers to this technical meaning.
The idea of a complex system is controversial and frustratingly difficult to pin down. Even the people who claim to be doing complex systems research do not agree on what the exact definition of a complex system should be. And there are some people outside the field who argue that it is all nonsense; that there is no such thing as a “complex system” at all, only a bunch of irresponsible scientists playing games with a vague idea.
I believe that in spite of all this controversy and confusion, it is possible to come up with a clean and very powerful definition of complexity. That is the good news. The bad news is that this definition contains some very bizarre characteristics that cut right to the heart of what it means to do science.
Rather than jump straight to the hardcore definition, I am going to work up to the full version in a couple of stages: first an informal illustration of what complexity means, then a definition, then a critical look at the definition to bring its strangeness out into the open.
An Informal Illustration of Complexity
Imagine that we were to put together a large network of computers and connect them in such a way that each one can only see a few of its close neighbors.
Figure 1: Network of computers with a small number of local connections per node (the connections from only one node are shown).
Now suppose that there is a set of rules (an algorithm) that describes what each computer is doing, and that all the computers follow the same algorithm. What does the algorithm do? Well, each node has a bunch of variables inside it (U, V, W, X, Y …), and what the algorithm does is to modify these variables in some way, as a result of what the node can see in its neighbors’ variables. Each node can only see its own variables and those of its connected neighbors.
The interesting thing is what happens when we make different choices for the algorithm that each node is following. And the most interesting thing is what happens when we choose some horrible, ugly algorithm that includes delays, gradients, peculiar cross-references between variables, and so on.
On the face of it you might think that if we use a messy algorithm the result will just be a lot of messy randomness in the U, V, W, X and Y variables. Sometimes this does happen, but not always.
In fact, if we experiment with many examples of different messy algorithms we eventually find that we can make two observations:
1) Sometimes, when we look at the overall behavior of the system, we see “regularities.” For example, if we take the U value at each node and use it to control the brightness of one pixel in an image, and if there are enough nodes (computers) to fill all the pixels in a generous-sized image, we might notice that as time progresses there are (e.g.) waves of brightness moving around in the image. We might find that when two of these waves hit each other a vortex is created for (say) exactly 20 seconds, then it stops. I am making up this wave+vortex behavior, of course, but that is the kind of weird thing that can happen.
2) The other observation is about the messy algorithms themselves. We know that these kinds of algorithm are so pathological that we cannot do any math to analyze and predict the behavior of the system. In other words (and this is a crucial point) there are absolutely no techniques that will allow us to look at the algorithm and predict that the system will show those waves and vortices.
Just to reinforce the point, it really is childishly easy to write down algorithms that are so intractable that no mathematician would want to touch them. For example, we are talking about algorithms that involve weird little dependencies like:
“Pick two neighbors at random, then pick two variables at random from each of these, and for the next 10 seconds try to make one of my variables (chosen at random, again) follow the average of those two as they were exactly 20 minutes ago, EXCEPT when neighbors 5 and 7 both show the same value of the V parameter, in which case drop this algorithm for the next 73 seconds and instead follow the substitute algorithm B….”
What is the conclusion to be drawn from this? The conclusion is that there are some systems that have interesting “global behavior” (the waves and vortices that we observe), but we cannot explain this global behavior in any rigorous way by looking at the “local mechanisms” (the messy algorithm inside each node) that drive the system. We can simulate the systems and notice the global stuff, but we cannot look at the algorithms first and derive a prediction that tells us what the global behavior will be. Systems that have this disconnect between the local and the global are complex systems.
How Does This Relate To Intelligence?
I will have a great deal more to say about how this connects to intelligence in a later post, but it might help if I give a quick example of how this illustration of a complex system maps onto the problem of building an AI.
In the example I just gave there were waves and vortices visible at the global level. If these waves and vortices correspond to the processes that we want to get up and running in our AI system (reasoning and learning processes, for example), then we might have a problem. We know what we want to have happen (we know what global behavior we want to see), but going from this desired behavior to the underlying mechanisms is difficult or impossible if the system is complex.
Many people react to this suggestion with skepticism. They would point out that an intelligence mechanism like reasoning is obviously not just the result of a mysterious mechanism underneath, so the comparison with my complex system is just not believable. Logical reasoning is not voodoo, it is more like a mechanism, and we can see a great deal of the structure that makes it work.
I am willing to grant this, to a certain extent, but it is important to understand that there is another, much more subtle and pernicious way to compare my example complex system with the mechanisms inside an AI. Imagine a variation on the system I described before, where it is not the waves and vortices themselves that we should be focussing on, but the stability of these waves and vortices. You see, the stability of the system’s global behavior can also count as a “global regularity” that we cannot explain in terms of the local mechanisms.
I will make this explicit with an example. Suppose we set up the same system that I described earlier, but this time we see waves and vortices that we can partially understand. We look at the underlying algorithm, and we find that this is not an example of the worst sort of complex system, because we can see a partial explanation for why the waves and vortices are occurring. But just when we start to believe that maybe this system is not complex at all, and that we might be able to build a complete, regular-science explanation for what is going on, we discover something annoying about the system. If we fiddle with some parts of the algorithm, the waves and vortices have a tendency to fade out after a few minutes, and the system descends into boring randomness. When we make the right choice for the algorithm design, we get waves and vortices that last indefinitely, but as soon as we modify the algorithm a little the system changes its character and the interesting global behavior only lasts for a few minutes and then goes away.
Most importantly, I want you to imagine that this sensitivity to the exact design of the algorithm is “complex”, because this one aspect of the behavior is completely inexplicable. We cannot see any way to predict what is the right choice of algorithm to get stability, we only know that the correct choice works and others do not.
This is a much more subtle type of complex system, notice. Lots of roughly-predictable behavior (the waves and vortices), but then some bits that are impossible to understand. This is a system that is partially complex.
You might be tempted to say, at this point, that this situation is easy enough to deal with: just make sure that we do lots of experimentation with the algorithm design, to find the right choice that makes the waves and vortices stable. Unfortunately, this way of thinking is upside down and backwards. There are going to be incalculable numbers of ways to build the part of the algorithm that makes the waves and vortices appear (the part that we partially understand), and also an incalculable number of ways to build the part of the algorithm that is responsible for the stability of the system - and it may well be that there is only one choice out of each of these combinations that gives stability! So you might spend the first ten billion years of your AI project trying out different mechanisms that produce the waves and vortices that you want, but for all of your choices in that first ten billion years, none of them can be made stable by any choice for the other part of the algorithm.
Imagine the frustration. You pick one choice for the part of the algorithm that gives some waves and vortices, then you start trying all the possible choices for the other part of the algorithm, the part that could in theory make it stable. But since your first choice is a dead end choice (because there is no choice for the other part that leads to stability), all your testing eventually comes to nothing. So you move on to the next choice for the first part, and the same thing happens. Ten billion years later you get lucky: for today’s choice there really is a design for the other component that will make the whole thing stable, and after fiddling with the parameters for a long time (a century?) you find one that works.
The problem is that if you pick an algorithm that looks like it might give rise to the waves and vortices that you want the system to show, you have no idea whether this will be the right one, and no idea what the right choice for the other component is. The complexity of the second component “infects” the first component, which otherwise looked like it was going to be a good bet.
Notice that this exactly describes what has been happening in AI for the last fifty years. People choose some mechanisms that almost make the system behave intelligently, but then they find that the mechanisms only work on toy problems, or they only work for a limited time and then degenerate, or it turns out to be difficult to find learning mechanisms that will allow the system to acquire its own knowledge without having to spoon-fed by programmers, or people have difficulty getting the parts of an intelligent system to coordinate with one another. Always, initial success followed by messy problems and stagnation. This is like my complex system example: it is easy enough to cobble together a rough algorithm that will make the waves and vortices appear for a short time, but making them stable just seems impossible.
So the actual progress made in the field of Artificial Intelligence is very much like he progress we would expect if the complex systems problem were real.
Much more could be said about how the complex systems idea relates to the methodology of AI, but what I have said so far should at least give you a feel for where this argument is going.
Most Systems Are Not Complex
Perhaps the most intriguing aspect of complex systems is that almost every system that has been studied by science over the past three hundred years is not complex. (Or, to be more careful, most natural systems do not have a significant degree of complexity in them; they are not dominated by complexity). When scientists started trying to understand the world they found that there are huge numbers of “regularities” out there that they could explain by finding local mechanisms and doing some mathematical analysis to show that the local mechanisms imply the global regularities.
The classic example of this is what Newton did: he noticed that the planets moved in ellipses and followed Kepler’s Laws – this was the global regularity to be explained. He then suspected that gravitational attraction might explain this, and was eventually able to show (after inventing the calculus!) that if all the matter in the universe followed an inverse square law of gravitation, this would imply that the planets had to move on elliptical orbits and follow Kepler’s Laws.
This dazzling chain of scientific thought (describe the global regularity, then suggest a possible local mechanism to explain it, then use mathematics to prove that the local mechanism does indeed explain the global regularity) has been repeated countless times in the history of science. It is an almost peculiar and wonderful fact about the universe, that so much of what we observe can be explained by this method. In other words, it is amazing (but true) that most things we see are actually not complex systems.
The big question then becomes: what do we do, as scientists, if we come across a situation where the system we want to explain, or the system we want to build, is actually a big, hairy complex system? What methods do we use if the overall behavior of the system cannot be mathematically explained in terms of its local mechanisms? Will the scientific method work under those circumstances? Can we engineer a system in which, for reasons we have no control over, the overall behavior of the target system is not mathematically explicable in terms of the mechanisms that drive it?
You might think that this kind of question had come up before in science or engineering, and that someone would have figured out some good answers to these questions. Not so. I believe that AI is the first occasion when engineers have confronted the issue (and, by extension, cognitive psychology is the first occasion when scientists have confronted the issue).
Before going on to hammer out an answer to these “What should we do?” questions, I want to stop and make the definition of complex system a little more water tight. If you don’t care about this cleaning up operation you can skip to the next section.
A Better Definition
Suppose that it is possible to take every scientific or mathematical explanation that has ever been discovered and write it out in standard form. The kind of standard form that I have in mind would involve something like a written account that started with the most basic facts you learn in grade school and including all the material needed to follow the explanations up to the point where the theory itself is completely described. Even though this would be impossible to do in practice, we will suppose that the Divine Mother is able to look down on everything and can normalize every scientific explanation for the purpose of comparing the size of different explanations.
From that perspective, complex systems are just system in which explanations of their global regularities are so huge that we have no practical hope of ever discovering them. And that includes cases where the explanation simply does not exist, so the size of the theory is undefined/infinite.
Most real explanations (Newton’s gravitational theory, etc.) have a theory-size that is pretty close to zero, in the sense that we can easily read all the books we need to in order to understand them. The point of calling some systems “complex” is that they happen to have theory-sizes that are somewhere up in the billions-of-books range (or worse). So, yes, there may one day be an explanation for why a particular complex system shows the overall pattern of behavior that it does, but for all practical purposes that explanation is out of our reach. All we can do is observe the regularity and perhaps pseudo-explain it by making a simulation of the system and observing that the simulation and the real system do roughly the same thing.
Notice that although the cluster of computers that I described above would be a wicked example of a complex system, the biggest supercomputer in the world would not be complex at all if it followed a nice, well behaved algorithm. So this is not about how complicated (in the sense of the opposite of simple) a thing is, because that supercomputer could be programmed with algorithms that most of us would describe as profoundly non-simple, but so long as the algorithms were (n some sense) well behaved, we could predict the overall behavior quite nicely.
In the real world, of course, systems are mixtures of complex and not-complex. It may not be possible to quantify the amount of complexity, but there is a clearly a sense in which some systems are dominated by complexity, whereas others have just a touch of complexity in them (a few cases where the behavior is regular but inexplicable) while at the same time being mostly quite normal.
So, finally, here is a better definition of complexity. A system contains a certain amount of complexity in it if it has some regularities in its overall behavior that are governed by mechanisms that are so tangled that, for all practical purposes, we must assume that we will never be able to find a closed-form explanation of how the global arises from the local.
The key idea is that we must assume that we cannot explain the regularities. Some systems that we think are complex may turn out in the long run to have explanations. This is fine: we would expect this. But the point of defining the concept of a complex system is that we need to handle a broad class of systems in which, on average, the theory sizes (as measured by the Divine Mother, or whoever else has all-knowing perspective on these things) must be assumed to be off the scale.
One last comment about why this definition is so unnatural. Notice that if the idea of complex system is a valid one (if there really are systems whose explanation is off the scale, or infinite), then we will probably not be able to prove the existence of this class of systems, or prove any theorems that allows us to measure the “amount” of complexity in a given system. Why not? Because all such properties of complex systems actually fall, themselves, into the category of “global regularities that cannot be explained”. So if complex systems really do exist, then by definition we will be unable to make many rigorously provable statements about these systems.
We may be able to observe characteristics of complex systems in general, and these observations may capture facts that (from the “Divine Mother perspective”) really are true, but we will not be able to know for sure that these really are true facts.
This topic of the godelian nature of complex systems is fascinating, but it will have to be postponed for another day.
the big lesson we need to take home from all of this is that we cannot know exactly how much of an impact complexity will have because we have no way to measure the “amount” of complexity, nor any way to say how much impact we get from a given amount of complexity. So if a system gives us reason to suspect that it is complex, we are very much in the dark and the best we can do is to tread carefully and make the default assumption that it really is complex. The system itself may never give us definitive proof that it is complex – in fact, it may tease us by hovering on the brink of wicked complexity and tempt us to think that maybe we can get away with treating it as non-complex. We may get into a situation where, lacking any convincing proof that complexity is playing a big role in the system’s behavior, we pretend that the complexity is not a problem, but then when we try to understand the system we may find it peculiarly difficult.
And the worst part of that trap, where a complex system teases us into thinking that perhaps it can be treated as non-complex, is that we might think that all of our problems with the system are purely the result of trying to deal with a non-complex system that just happens to be very complicated (very difficult to understand). This would be a disastrous situation to get into, because we would run around in circles trying to find “normal” explanations for the system’s behavior, or trying to engineer the system using “normal” engineering techniques, while all the time these techniques can never give us closure.
Are Intelligent Systems Complex?
Now we come to the crucial question: do we have reason to suspect that a significant amount of complexity is involved in all intelligent systems?
I could present lots of reasoning here, but instead I will resort to quoting AGI researcher. Ben Goertzel, in a message he posted to the AGI discussion list on 12/6/07:
“There is no doubt that complexity, in the sense typically used in dynamical-systems-theory, presents a major issue for AGI systems.”
(www.mail-archive.com/agi@v2.listbox.com/msg09053.htm)
I would like to take it as understood that researchers in Artificial General Intelligence accept the existence of complexity in AGI systems.
Next time I will take the argument from there and try to look at how big an impact this might have on the methods we use to try to build intelligent systems.
Comments
19 Responses to “The Complex Systems Problem (Part 1)”
Leave a Reply
Great, but isn’t this just a rehash of the old “neats vs scruffies” debate that’s been going on for decades?
Furthermore, I agree that a brain-comparable AGI system will probably be too complex for any one person to understand. But is that a big problem? Windows Vista, for example, is way too complex for any one person to understand when looking at the assembly code. But it still works predictably on the macro scale, and is describable (at least approximately) on intermediate levels.
I hear what you say, but I think this is much more than a rehash of the neats and scruffies debate. This time, there is a specific understanding of the problem, I believe, and a specific prescription for what has to be done (which I have not described yet). Stay tuned: this is only the first part of an extensive paradigm statement.
The problem with a comparison with Vista is that Vista is not dominated by complexity. Far from it: the thing may be ugly as sin, but is predominantly an ordered, structured system that pushes complexity into the dark corners. For this reason, the comparison breaks down completely.
This approach sounds a lot like Wolfram’s “new kind of science”.
It is indeed very similar. Apart from some fine details that are original to my work, the basic argument is just a straightforward statement of the fact that complex systems make life difficult. There is a difference of perspective and goals between what I do and what Wolfram does, but we are kindred spirits.
My motivation for describing all of this is that there are many people who do not have any clear idea of what it might mean for “complexity” to exist in an AI, and what impact that complexity might have. Similarly, I have sometimes tried to explain to a person that Wolfram’s NKS has significant implications for AI, and gotten complete non-comprehension of what that statement is supposed to mean.
you wrote: So, finally, here is a better definition of complexity. A system contains a certain amount of complexity in it if it has some regularities in its overall behavior that are governed by mechanisms that are so tangled that, for all practical purposes, we must assume that we will never be able to find a closed-form explanation of how the global arises from the local.
this is not a “watertight” definition! What on earth do you mean by “tangled”, “closed-form explanation”, “global”?
the best (and probably the only) way to make precise definitions is to write them down in maths, not in english.
For example, Kolmogorov complexity is a mathematically rigorous concept that seems to be touching on the same issues that you touch upon.
Your comment makes no sense. The definition I gave clearly described a concept that is extremely hard to formalize - that is the whole point of the complexity idea, that it defies formalization in ways that have no precedent in science. So the very idea of complex system has “This Cannot Be Completely Formalized with a Mathematical Definition” built into it …. and you find this definition weak *because* it is not a precise, mathematical definition?
Having said that, I probably would have been better off not using the word “watertight”, when what I really meant was that it was immune to a certain type of confusion and ambiguity. My bad.
As for Kolmogorov complexity, it is similar but not the same. The concept of a standard form of scientific explanation is not formalizable in the way that Kolmogorov complexity is, and that is by design (see the above paragraph). At some point I would like to explore the relationship between the two, but in the absence of that longer explanation, I don’t think it is too difficult for someone who understands Kolmogorov to see where it departs from the concepts I have described here.
“that is the whole point of the complexity idea, that it defies formalization in ways that have no precedent in science”
- I think this is an unwise path to go down. If you try to come to conclusions about what is and is not possible in the real world using concepts that are not well defined, you are unlikely to come to the right conclusions. The reason that formal mathematical arguments have been so successful is that, within a formal system, you can be sure that what you are writing down makes sense.
If you start arguing informally using some nebulous, vague conception of “complexity”, I won’t pay much attention to what you say, any more than I would pay attention to someone who started talking about crystal healing or spirits.
I think it would be fruitful to try to mathematically formalize the concept. This will be difficult, but probably worth it.
You say “If you start arguing informally using some nebulous, vague conception of “complexity” … but this betrays the fact that you cannot have read what I said with enough care. I say this to you respectfully: you are making comments about a description of “complexity” that was not the one I gave.
The concept is not nebulous or vague at all. The way that I put it, it is as precise as it is ever going to get, and in this form it has enough precision to have devastating implications.
Consider: suppose that there was a completely objective way to measure “size of explanation” that applied not just to abstract mathematical concepts, but to real scientific explanations in the real world. Now consider what situation we would be in if, for some types of system that clearly have interesting regularities in their overall behavior (the kind of regularities that scream “Explain Me!” when a curious scientist looks at the system), the actual size of the simplest explanation is quasi-infinite. Supposing for the moment that such systems did exist, there would be a number of very unusual implications. One of those implications would be that you could never prove that the theory-size for that system actually was out of reach.
In those circumstances, you could spend the whole lifetime of the universe believing that those systems did not exist (i.e. that all theory sizes were within reach of a feasible amount of human effort). You would have no way to establish that such systems really did exist (if you did find a way to prove their existence, that proof would negate their existence). The problem is that in such a universe it makes a great deal of difference whether you acknowledge the possibility that such systems exist, or pretend that they do not. If you pretend that they do not, you can waste a lot of time chasing theories that you will never find. If you acknowledge their possibility, you can adjust your strategy to cope. There is a big difference between the consequences of these two courses of action, but you have to make your choice whilst being blind: you can never prove that the theory sizes really are out of reach.
All of your comments about “vagueness” of this concept miss the point. The description I just gave, in these last few paragraphs, is complete, and not vague in any way.
“The description I just gave, in these last few paragraphs, is complete, and not vague in any way.”
I disagree. Firstly, you are assuming that it makes sense to speak formally about “real world scientific explanations”; I don’t think that this makes sense. It probably would make sense to speak of explanations relative to some fixed computer language, in which case you would have the kolmogorov complexity. Then the “size of explanation” would be the kolmogorov complexity of a a bitstring.
Secondly, what do you mean by “quasi infinite”? This is a serious problem. I guess you mean “very large”, but one has to be rigorous about these things: notions like “large” are hard to make precise in a useful way, and before I accept your argument as holding water, I would want to know exactly what you mean.
This argument is about the practical strategy to be adopted by a scientist or engineer trying to understand or build systems. Do you take the position that, because scientists cannot formally quantify the size of explanations, anytime they talk about some explanations being larger than others, their talk is strictly meaningless? If a physicist says that an explanation of quantum electrodynamics is larger than an explanation of newtonian gravitation, would you say that they are opening and closing their mouths without saying anything?
The problem is that there is no mathematical definition of “scientific explanation” that, in general, a scientist would say maps exactly onto the concept that use when they discuss the relative sizes of different explanations. You may hope that in the future there will be such a thing, but that is beside the point: we cannot wait for some future formalization of the concept of a theory-size. So, absent that future formalization, what do you suggest, as a practical strategy, that scientists do when confronted with the possibility that there are some systems in which the (informally defined) theory-size is so large as to beyond the reach of current human intellect. Say, something that could only be written down in a billion books?
I submit that your demand for a formalization of the scientists’ informal use of the concept of a theory-size is meaningless (because such a thing may never be possible) and contributes nothing to the practical problem of dealing with systems that may have unmanageable theory-size.
If you disagree, can you suggest a strategy that should be adopted by scientists faced with such a possibility? Analyzing the informal concept of ‘theory-size’ as well as it can be analyzed, and then using the analysis to come up with a strategy, is exactly what I am in the process of doing.
P.S.
“Quasi-infinite” should be obvious in the context: larger than is practicable. If the smallest theory of a trivial complex system is a billion times greater than the size of QED, this is for all practical purposes up at infinity. And some complex systems may have no theory at all, not of any size.
So, absent that future formalization, what do you suggest, as a practical strategy, that scientists do when confronted with the possibility that there are some systems in which the (informally defined) theory-size is so large as to beyond the reach of current human intellect.
I think by introducing a confused notion of theory size upon a confused notion of “explanation”, you have created a problem where there isn’t one.
Consider, for example, how difficult newton would have considered a problem in modern physics which requires, say methods like the path integral formalism, or the principle of least action. Newton’s conceptual language was so impoverished that even to write down the problem would have taken hundreds of pages. I believe he did all his calculus using greek-style geometry arguments.
Or, for a better example, consider how massively difficult the very first computer programmers [e.g. lady lovelace] would have considered the problem of writing a modern computer program like microsoft word. They thought of everything in terms of rigid input-output rules. Today we have class hierarchies, polymorphic functions, object oriented programming, etc.
As science advances, its conceptual language changes. Layers of abstraction get built up, one on top of the other. Concepts which would have been inexplicable 200 years ago become explicable, which is why you shouldn’t use words like “quasi infinite” - you should simply say “large”, in which case it becomes clear that a set of data whose simplest explanation relative to one language is large, might have an explanation relative to another language which is small.
To put this more strongly, you must not base impossibility proofs on ill-defined notions.
Consider the following argument (which would have sounded convincing circa 1890) that powered flight is impossible for humans to develop:
1. Powered flight requires a quasi infinite precision-of-design
2. humans can only ever build fairly clunky machines, with bounded precision-of-design.
3. Therefore, powered flight may be forever out of our reach.
The reason that this argument fails is that if you do define precision-of-design in some precise way (for example in terms of thrust, lift and weight), you realize that there is no reason that human ability to design things very precisely is limited, and that quasi infinite really means finite. Thrust to weight ratios continue to improve, and at some point we reach cross-over.
The same goes for understanding intelligence. If you define “complex system” and “explanation” precisely, you’ll realize that our ability to write down regularities of increasingly complex systems will continue to improve as we invent richer, more abstract conceptual languages. You’ll realize that quasi infinite complexity really means finite complexity, levels of abstraction and generality of our conceptual languages continue to increase, and that at some point we reach cross-over.
I’m sorry, but you are saying things that mathematicians would dismiss as naive. Please take a trip over to DAMTP (Department of Applied Mathematics and Theoretical Physics) and find someone who understands the tractability of deeply nonlinear equations, and then try to get a grasp of just how silly it is to assume that all such equations will eventually have closed-form solutions that can be discovered and written down in a humanly feasible amount of time.
All of the examples you cite of physical systems being understood by human persistence are examples of manifestly non-complex systems. If you need a longer explanation of this point, you can find one in the paper on Complex Systems and AI that is on my publications page. For that reason, none of your examples have any bearing on the argument.
As for your powered flight example, you are grasping at a straw man: nobody would be so dumb as to make the assertion in step 1, and nothing that I have said implies that my argument maps onto such an assertion.
You have also avoided my direct question (”Do you take the position that, because scientists cannot formally quantify the size of explanations, anytime they talk about some explanations being larger than others, their talk is strictly meaningless?”) by throwing out several red herrings. Go to the folks at (e.g.) DAMTP and ask them if they think it is meaningful to say that QED will always be a theory that requires more explanation than newtonian gravitation, and then ask if they believe it possible that some future theories of physical phenomena could be so large that even their simplest expression would be so large that it could take a billion years to simply write the whole theory out. Or ask them if it is possible that some physical systems may have no theory whatsoever that will explain them (if we do not count a simulation as a theory). They will tell you that this is possible, and that in those circumstances they would not expect any human ever to be able to discover or write down such a theory.
If you don’t like their answer, please take it up with them. At the moment you are trying to tell me that their informal notion of theory-size is incoherent and meaningless …. but instead of complaining to them about it, you are accusing me of peddling in confused concepts.
You have also avoided my direct question (”Do you take the position that, because scientists cannot formally quantify the size of explanations, anytime they talk about some explanations being larger than others, their talk is strictly meaningless?”)
It has meaning in an informal context, but, until you define exactly what you mean by theory size, you cannot use it to make formal arguments. I’ve never heard any academic say “theory size”, although in an informal context they will frequently say “this piece of physics is easier than this one”.
One might compare this with the way that people will informally say that a man is bald, and that adding one hair to a bald man’s head will not change the fact that he is bald, all without giving a formal definition of “bald”. These kinds of rules of thumb are fine for informally reasoning about day-to-day life. But if you carelessly try to push informal, ill-defined concepts out of their everyday usage into a proper logical argument, you end up with things like :
Theorem: All men are bald
Proof: Clearly a man with no hair on his head is bald. Adding one hair doesn’t change this, so for any n, a man with n hairs is bald. QED
Basically, until I see a mathematical definition of complexity, you’re not going to convince me.
Not that I want to kill your argument though: I think there may be a lot of mileage in using Kolmogorov-Chatin complexity. You should also see Chaitin’s incompleteness theorem.
http://en.wikipedia.org/wiki/Kolmogorov_complexity#Chaitin.27s_incompleteness_theorem
You still haven’t answered the question that I posed, but avoided it with more red herrings (your baldness theorem, for example, is usually cited as a reason why it is pointless to insist on a strict, formal definition of informal terms such as “bald” … but you seem unaware of this!).
I will ask the question again, though this is the last time. Scientists face practical, real world problems every day that cannot be defined mathematically. Biologists worked on the idea of “genes” for a long time without having any clear and formal definition of what a “gene” actually was. This is especially true of the methodology of science, which is not a formal mathematical process at all. This argument that I am presenting is all about the methodology of science. So now I say to you again: scientists face a dilemma, in that there appears to be something that could be called “theory size” and there appears to be evidence that some systems have a theory size so massive that scientists cannot expect to deal with those systems in the same way that they deal with other systems. They need a strategy, and they cannot stop their search for a strategy because people like you are insisting that the concept of “theory size” has not been defined mathematically.
So this is the question: do you accuse these practicing scientists of being fools, because they are trying to deal with this methodological dilemma in the way that they usually do - without being able to get a precise mathematical definition of “theory size” from the outset? Please answer the question directly, because so far all of your comments have given the implicit answer “Yes, I believe that what they are doing is incoherent and meaningless.” Is this really what you believe?
Thank you for the links to Kolmogorov-Chaitin complexity, but there is no need: I am fully aware of that literature. KC cannot be used in this argument, as I said before, and your comment “Basically, until I see a mathematical definition of complexity, you’re not going to convince me” betrays the fact that you do not understand this at all, because when you make that claim, you are asking me to provide you with something that is a blatant self-contradiction. In my opinion, anyone who demands a self-contradiction without understanding that he is doing so is putting himself in a rather embarrassing position, and it is not my job to convince him of anything.
So now I say to you again: scientists face a dilemma, in that there appears to be something that could be called “theory size” and there appears to be evidence that some systems have a theory size so massive that scientists cannot expect to deal with those systems in the same way that they deal with other systems.
Sure, if you think that you have a new method which will solve previously unsolved problems, then go for it. The same goes for any other scientist. As you say, science is all about following your nose, trying things that may or may not work. I don’t think it’s incoherent to follow a hunch, but I do think it’s naughty to present a hunch as if it is a logical argument.
You did not answer the question again, but just tried to deflect attention with some more allegations. Enough already!