Coherent extrapolated volition

Eliezer Yudkowsky’s 2004 proposal: rather than our current, contradictory preferences, give an AI what we would want if we knew more, thought faster and had grown up further together. The criticism: whose volition counts leaves enough free parameters to justify almost anything.