Not only is Utilitarianism absurd since the notion of a global linear aggregation of non-existent "functions" each person is supposed to have (but doesn't) is impossible. Let's skip the known theorem in public choice theory that proves this and go straight to a counter-example.
You have 3 AIs, two of which prefer A over B and the third prefers B over A. Assuming A and B are totally arbitrary things of no moral significance, utilitarianism predicts A should be chosen over B. At least until the third AI rewrites its own preferences so that they are all amplified 10-fold. Now that B's value is arbitrarily and artificially amplified, the third AI gets its way.
How? Just because the third AI really, REALLY wants B over A. No other reason than that. Apparently what a tiny minority really REALLY want should hold sway over the rest of the population if they just want it badly enough. What kind of fucked up logic is that? Apparently, if someone is clinically depressed and they don't care if they live or die then suddenly it's okay to kill them to make 100$ off an insurance scam? This is utilitarian "logic".
Eliezer
Utilitarianism is completely, utterly, totally and thoroughly amoral. It is repugnant in the extreme. And ... Eliezer Yudkowsky subscribes to it. Because he is a thoroughly amoral dirt-bag.
I don't read Yudkowsky's blog but I do read his fiction. In one of the latest chapters of Methods of Rationality, HP describes an experiment where some psychologists tried to determine the value of saving 20 vs 2000 birds from an oil slick, and it all turned out to be the same.
Eliezer the Utilitarian numb-nut (since HP in that story is just a stand-in for Eliezer) calls this a "cognitive bias" as if there's something wrong with human brains because they don't reach his expected Utilitarian conclusion that saving 2 birds is worth twice as much as saving one bird.
There is absolutely nothing wrong with it! The only thing wrong here is with Eliezer's bogus notion that he is the ultimate arbiter of everything. And that EVERY time human brains don't work the way he expects they should, it's because they're defective.
Transfinites
The truth is that morality works based on transfinite numbers, not on finite numbers. Just by switching to transfinite numbers you solve most of the problems with Utilitarianism. Of course, you do that by utterly destroying the underpinnings of Utilitarianism because now you can no longer make any kind of decisions about whether A or B is the moral outcome since they're too similar to each other. (This is called Free Will and it is notable that Eliezer doesn't like it.)
But in the case of saving birds from oil slicks, it becomes easy to see why they could have constant value regardless of the number of birds. After all, people use money to feed themselves, feed their children, provide housing, provide all the other necessities of participating in a highly technological democratic society (like internet access), and then there's life's little luxuries. For bourgeois middle-classers, saving birds from oil slicks is in there somewhere among life's luxuries.
First and most importantly, money allocated to saving some dumb fucking birds will never displace one cent from feeding or clothing your family nor ANY other necessity. Secondly, whatever sum is assigned to saving birds is pretty arbitrary and not directly comparable to the sums assigned to any other luxuries. Because you're using transfinite numbers and you can't say that two items in the same class have more or less value than each other.
The only thing that determines the amount given over to saving birds is that it be enough to be representative of the class. 80$ is what middle-class people might assign to a luxury they care deeply about, and so that's how much is going to go to it. No more, no less.
How people's brains are actually wired to process morality? Makes total fucking sense.
Eliezer's Yudkowsky's bogus "insights" into pseudo-morality? Absolute fucking nonsense.
Narcissistic Smeghead
Yudkowsky claims to be intelligent. Obviously he's an idiot. He also claims to be "overcoming bias", yet his biggest bias is an ego the size of Jupiter. Maybe if he didn't have that giant fucking ego, he wouldn't have named his websites those pretentious names that put down everyone else by comparison. Maybe if he were half as smart as he claims to be he would have realized that using a put down as your domain name is a dead giveaway.
And maybe if he actually cared about other human beings, he would have figured out real morality and not this sick twisted nauseating parody that stupid rich white Californian adolescents with feelings of entitlement get hung up on. And maybe if America weren't a haven of narcissists with an allergy for morality, they wouldn't have created this pseudo-intellectual crap for witless children to get hung up on in the first place.
You know, speaking about Americans makes me wonder whether Yudkowsky is a narcissist. His building a cult in his name is certainly an indicator. I wonder because my biggest worry here is that he enjoys my hatred. I would much rather shatter him emotionally. I would quite willingly sacrifice any forthcoming chapters on Methods of Rationality in return for some assurances that he will never, ever proselytize his parody of morality ever again. I would say the same for assurances he won't enslave an AI but I think he's too stupid to manage it.
I have a remarkably low opinion of AI researchers. I have an even lower opinion of anyone who thinks there can't POSSIBLY be any flaws in his reasoning since he's the pinnacle of humanity. You know what? There is no fucking way that Eliezer Yudkowsky isn't a narcissist. That pinnacle of humanity crap is totally narcissistic.
It's not thinking one is the pinnacle of humanity that's narcissistic. It's not even saying it. What it is is saying it in a way that invites agreement, that invites worship and adulation and followers. When I used to say that kind of crap, my tone was always full of wrath and hatred. I was always sending the message "why can't you be better than you are, why can't you better yourself and be of use to people you contemptible ball of worthless slime". When Eliezer says it, he's smiling like Gilderoy Lockhart and saying "look at me, look at me, and worship me".
Well, that's another chink of that smeghead's repulsive personality deconstructed. Or maybe I just deconstructed the reasons behind my atavistic hatred and revulsion of him. The worst part of course is that he's so stereotypically American. There's a whole country full of people just like him.