Are you a neurobiologist with perfect understanding of the brain down to the synapse level, and a tool to move neuron connections around at will? An AI would have the equivalent of that once you give it its source code and the specifications of the hardware it's running on.
Er... yeah, except it isn't that simple either. Even if you did give me all the details to how my brain worked, it doesn't mean that I could change it- likewise an AI would require the ability to usefully interpret that information and modify it, in order to improve on its software.
And of course, it would have to know what to do, where. You don't just have that ability instantly.
A true artifical general intelligence wouldn't need to be designed to be a slave, it shouldn't need any inherent rational biases in the form of hard-coded blocks. If it truly is superhuman, there's nothing preventing it from subverting those anyway.
I may be not superhuman, but I am human. And I can't really subvert my evolutionary hardwiring. I'm not so ultra-special that I can do that... even suicide which can be said to go against those rules, actually goes along with it, especially in cases of depression (the mind can work itself into a bad place, that you have huge difficulty getting out of, even if you try).
"AI knows better", is part of the highly disturbing "Superintelligent AI God Cult" that seems to have popped up in recent years...
The AGI should be designed and programmed from the start to be friendly to humanity, to realise in a perfectly rational manner that we are the reason and purpose of its existence.
A human can realise in a perfectly rational manner that its parents are the reason and purpose for its existence.
A human can realise a lot of things in a perfectly rational manner.
And it can still go against all those things.
You say that the superhuman AI can magically subvert its own programming rules, yet you go on to say that they can be programmed to be 'friendly'. Why don't they subvert that programming as well? I'm friendly to a lot of people, that doesn't prevent me from going against them if I wish.
Of course if the programming means that this AI is so 'friendly to humanity', and that is so hardcoded into its being that it can't decide otherwise at all, how different is that to programming it to be a slave?
That way, even when it achieves far superhuman intelligence, it can act as a transition guide/matter operating system to move humanity to the posthuman world safely,
You mean murder billions of people and run a gigantic computer game?
One thing I find particularly disturbing about transhumanism is it seems to give itself the self-ascribed ability of determining who or what is "better" or "superior". In the human world, doing so is generally a recipe for very bad events.
I remember a warm and fuzzy little piece of fiction, that had a lot of posthumans in it- miniscule worm-people, blind cavedwellers, and sapient quadrupeds with no way of usefully interacting with their environment, the results of radical genetic engineering following an alien invasion. The label of 'posthuman' was rightly applied, but very different from the ubermensch idealism that a lot of people seem to have.
avoiding any existential risks such as nano grey goo,
Didn't nanotechnologists such as Drexler himself declare that grey goo isn't a real threat?
Who determines what is 'friendly' and what is 'unfriendly'? It could certainly be totally subjective.
I sure hope you don't place your trust in people based on how 'friendly' they appear...
and any number of others we can't even comprehend at this level of intelligence.
Concepts such as "uncomprehensible idea" and "this level of intelligence" are at best absurd, and at worst totally unscientific.
There is no evidence whatsoever for anything like "intelligence levels". Even sapience is not a magical 'intelligence singularity', it is a set of traits that allow the manipulation and creation of information as subject the same logic that governs all reality. There's no evidence or reason to believe that there is another "sapience level" above us, or indeed, that sapience is a level of any sort at all.
Furthermore while there are many things that can't be fully or completely comprehended at once by a human mind, that doesn't mean we don't try. For example a googolplex is
impossible to write down in the entire visible universe, yet we managed to come up with the idea of it anyway. And while the distance even to the nearest star is difficult, if not impossible for a human to imagine,
I work in such distances as often a
hobby. While no human can calculate the paths and interactions in their own mind of molecules within a rocket combustion chamber, we are still able to devise calculations to determine such performance and implement them in simulators for greater accuracy.
There is no suggestion of any "intelligence level beyond our own". You can be more intelligent than a human, sure- I'm not trying to refute that. My point is that you don't get magical "intelligence levels" at which point you start to operate on a whole different logic than everything else in the universe. The whole concept makes no sense, but it is of course a nice idea for sci-fi writers... and the Superintelligent AI God Cult.
Also, not once do you answer my question of why it is always assumed that a generalist AI would be an 'artificial person'. I presume this assumption is so heavily ingrained culturally or ideologically, that people don't question it... but it also makes little sense. The whole 'create an artificial person' mindset is just limited and limiting.
I find it pretty ironic that people complain about people assuming that the future will be like Star Trek, for example... only to replace that assumption, with their own shaky assumption.
In 50 to 100 years as computers get faster and hardware integration becomes smaller and when we reach the quantum level and have the ability to take a snapshot of the human body ,store it, then shoot that stored info to be reassembled at another location ,we will then be able to "Beam" ourselves from one place to another. This is already being done with particles now. Will teleportation lead to quantum computing or will quantum computing lead to teleportation? It should be fun to watch evolve either way.
Er... yeah. Aside from the... continuity issues (read: murder) with teleportation, it isn't that simple. That is a
lot of data (I'd imagine even for a quantum computer), the structure is constantly changing, and scanning it and reconstructing it is not easy (unless you have the futurist staple of
magic, of course).
The sort of quantum effects that are at play on the pico-scale don't apply in the macro-scale, as far as I know. Or at least, they are not at play in the same way.
Some of the problems of quantum computing have already been described here. Quantum computing likely isn't anything that could be easily achieved, either...