Steve ask the question "Can friendly AI evolve?".
I would like to comment on that, lengthly, so I decided to set a topic here, as he is around and this seems to be an Eclectic Thinkers topic.
First of all, the "us vs them " attitude is somehow overdone. There are people willing to create AIs with build-in evolution potential allowing even for confrontation with NIs (for Natural Intelligences). Else, the discussion would be inexistent. The segmentation between NI and AI is done essentially to give credit to the parents of AIs, in a world where technological achievements are mostly made for glory and/or money. At least for power.
So, let's keep in mind that there may be a secondary segmentation between NIs, a minority pro-AIs, those driven by Asimov's Frankenstein syndrome, afraid from machines, and all the others who haven't' even take time to think about it.
Self-preservation is build even in simple, non-intelligent systems, and this is due to human willing to preserve their realizations. Back-up (clone, mirror, whatever) is essential so one wouldn't have to re-iterate the same work a second time. I am sometimes puzzled from people who don't do that. I always considered this as being one of the first elements to build in any application. Thus it seems obvious that AIs will/should be self preserving.
This doesn't mean that they will have to compete with NIs.
If you are an IBM engineer we may be willing to see Deepest Blue win a chess tournament against a human Chess Master. That doesn't mean that Deepest Blue wanted to win the game. But then DB isn't intelligent either.
Competitions present at least two aspects: I am better then you and I own more then you. Both may be present in a single competition, say with a great material prize to win for the best competitor.
In a any case, power is what the best gain. Either directly as money to spend to control their environment, or indirectly, as they can influence the very same environment using their fame.
Do the NIs and AIs have to share the same environment? If yes, there will be some competition. Even if obviously there isn't any need to share a parcel, the power gained by the others will drive competition. The Google vs Microsoft vs Yahoo! situation is an example. Competition between Yahoo! and Google seems natural, as they share the same space. The entry of Microsoft in the domain is driven essentially by the will to preserve influence over internautes and don't let new actors gain enough power and be able to drive the public opinion, thus ensuring that the public opinion will be favorable to Microsoft.
Being selfish or selfless is a quite simple or complicated matter.
If you do have clearly defined goals and introduced your self-value it is easy to see if you should act egoistically or altruistically, depending of the overall gain. I would accept to die instantly if I had the assurance that every single HIV virions would disappear and don't be replaced by something worse.
I suppose that AIs would be at least during the first times simpler and thus more efficient then the actual NIs. Thus more selfless, understanding more clearly the need for personal sacrifice to improve the species survival.
As a matter of fact, most of the selfish attitudes inscribed in a social behavior tends to be based on the idea that the self-value is important for the community (which is not necessarily truth). I recently named this behavior the "Neo complex": I am the One, even if it isn't clear even for me how I will manage to save the World. Let ME see were the white rabbit goes...
I suspect that AIs will be able to develop such attitudes :-)
Now, let me know what do you think about those matters dear NI-fellows.
