My guess is that this restricts the use of neural network architecture for selection of methods. It has been problematic for humans to determine “what” was the controlling factor at the time a decision was made - “debug” tags may make that less of a problem with artificial “thinking.”

Is it commonly accepted that we won’t be able to extract motivation from an AGI? That we must attempt to infer? Why not instrument?

I am having some difficulty now reconciling “It is almost always possible to drill down into a system and find some complexity somewhere” with the statement here that the choice of regularities really doesn’t matter… but I’ll think on it further. I’m also at the moment rather uncomfortable with the appeal to “delicate sensibilities” and “tangled” as part of defining “untouchable” mathematics. It might take me a while to reforumulate that in a way that makes sense to my own delicate sensibilities.

I’ll be working on that and following along, but probably with fewer barrages of public bewilderment. Thanks for listening to my questions.

My only comment here would be to say that, yes, people have different opinions on which ‘regularities’ might need explaining, but that this really does not matter. If a system shows anything more interesting than random behavior, then that behavior could, in principle, be explained. The only question of importance in our context is: are explanations always small enough that human minds can find them and write them down, or could it be that sometimes the smallest possible explanation that can be found (for a particular system) is too large to be feasibly discovered?

That is the only issue that is important. Hypothetically, such systems could exist. See today’s post for more on that.

The first issue has to do with whether certain systems (the craps table, an advanced fighter airplane, the three-body gravitational systems, and so on) “are complex”. This depends crucially on the nature of the regularities that an observer finds worth considering (that is, that seem to “demand an explanation”). Among reasonable people with similar goals, there is likely to be agreement about which regularities those are; in other cases, the theory size required may vary greatly depending on the properties that a scientist finds interesting and the degree to which they have to be explained by a theory.

In practice, this relativity will be a cause for heated discussion among individuals, but it should become clear with discussion exactly where disagreement lies: what are the specific local mechanisms of interest? what are the specific global regularities that demand an explanation? what must the explanation do to be satisfying? and then how big does an explanatory theory appear to be? (and thus, how important is complexity to the phenomena under study?)

Regarding the Mandelbrot set, it seems to me that the definition could be extended to cover such abstract entities if desired, but given that the purpose of your argument has little to do with such things, there’s no apparent benefit to screwing around with that.

So, if you think I’ve basically got this stuff right, I can move on to trying to understand untouchable mathematics.

Sorry if I seem sort of idiotic; I find that truly trying to understand somebody else’s ideas from their own point of view is quite difficult unless they magically match up with my own ways of thinking. I suspect it is also often difficult for others also, and the effort is therefore rarely made (unless compelled by school or work).

But now I’m just confused again. I promise it is not on purpose.

> I am also sure that it applies but don’t
> believe that it is a huge problem unless
> you ignore it. Remember, gravity with
> three bodies is a complex problem

Is that right (gravity with three bodies is a complex problem)? I don’t see what the global regularities demanding explanation would be in the case of the three body problem.

Regarding the Mandelbrot Set question: naively, it seems to me like a great example of complexity — with very interesting global regularities in the images, but it’s not really a “system” in the sense you seem to be talking about…. no bunches of interacting elements, no dynamic “behavior”. With this question I’m trying to probe the sorts of things your definition applies to.