hyperion, vr, higher-order desires, and more

so i’ve been reading dan simmons’ hyperion lately (thank you joao!!) and hoooly shit it’s a masterpiece. peak of the golden age of scifi. anyways one of the ideas they used in it is called the stimsim. in hyperion, stimsim lets people experience artificial realities directly through their nervous systems. the obvious way to think about this is as extremely good vr, but i think that actually undersells it.

imagine that the technology gets arbitrarily good, so good it plugs right into ur nerves and your brain. you want to travel around the world, so it generates the experience. you want to live in a mansion, become a famous musician, have amazing sex, explore another planet, or spend a decade living inside some elaborate fantasy society, and it can generate all of that too. at some point, a huge fraction of material scarcity becomes irrelevant, since you do not need ten million physical mansions if ten million people can each experience living in one.

nick bostrom talks abt a related idea in deep utopia, where he argues that what happens after technology makes human effort unnecessary for almost every instrumental purpose, calling this the “post-instrumental” condition. the world is not merely rich enough that we do not have to work very much, and there is increasingly nothing practical that requires us to work at all. utopia for those optimistic + extremely bullish on agi!! (although agi approaches this slightly differently). stimsim takes this one step further, in that if what you ultimately wanted from some activity was the resulting experience, why bother with the activity?

the obvious objection being, maybe i don’t just want to experience something, maybe i want to actually accomplish it. this is basically robert nozick's experience machine. nozick asks us to imagine a machine capable of giving us whatever experiences we want. you could experience writing a great novel, falling in love, or saving the world without any of those things actually happening. he thought our reluctance to plug into that showed that we care about more than how life feels from the inside, and there’s something intrinsic abt us wanting to do things and be certain kinds of people, not merely experience ourselves doing and being them. that seems like a pretty strong objection to stimsim….. but what if you make the machine a little better? what if the simulation contains actual societies? suppose stimsim can generate whole worlds, which lets you live in a city containing millions of simulated people, who can all react to you, disagree with you, remember things you did ten years ago, fall in love, form political movements, make art and develop grudges. you can try to become famous and succeed/fail. you can start a company. you can make friends who surprise you.

at that point, the line between “simulation” and “reality” gets blurry, or at least not super useful. david chalmers argues something close to this in reality+, where he says sufficiently rich virtual worlds are not necessarily fake worlds at all, and that virtual objects can be real virtual objects, and a person could in principle live a genuinely meaningful life in virtual reality. the physical substrate is not what determines whether the relationships and events inside that world matter.

so it seems that nozick's objection only works against a particular kind of machine, one that feeds you hallucinations while nothing corresponding to them actually occurs. well fine, we can give the simulation persistent causal structure, and inhabitants with autonomy, maybe even “consciousness.” and now you have something much harder to dismiss as fake. but what if i care that i am in physical reality? this is the coolest part of the thought experiment imo. why can't stimsim change that preference too? satisfying your preferences isn’t the machine’s only option… it can also just edit them. economics normally starts with some version of

you have a utility function ($u$), representing what you care about, and the problem is to find some state of the world ($x$) that scores highly according to it. technological progress mostly expands the set of ($x$)'s we can reach. but an advanced enough stimsim system gives us control over both sides:

it can change your world, and it can change what you want from the world. suppose you desperately want to be admired, the machine can generate a society that admires you. suppose you object that their admiration is simulated, well the machine can change your psychology so that substrate no longer matters to you. suppose you object that you do not want your psychology manipulated, well it can alter that preference too so that you don’t care abt that. and suppose you complain that this new person is no longer really you. well, it could presumably change how you think about personal identity. there’s something of a recursive closure here, in that for almost any dissatisfaction, the system gets two possible interventions:

or

once both are easy, “how do we satisfy human preferences?” is now like the preliminary question to the one that matters more, which is “which human preferences should prevail” or better yet “which version of you gets to answer that question?” i did a quick google search, and apparently philosophers have been thinking abt versions of this problem for a long time. there’s a paper by l. a. paul literally called “whose preferences?”

frankfurt and “higher-order desires”

harry frankfurt made an influential distinction between ordinary desires and what he called higher-order desires. you can want something

and then also have preferences about those preferences:

an addict, for example, might want a drug while desperately wishing that he did not want it. frankfurt thought something important about personhood and freedom lives at this higher level, in that a person can reflect on the motives pushing them around and care about which motives actually become their will. this gives us an initially appealing rule for stimsim - “do not ask what someone happens to desire, ask what desires they endorse.” but this is too easy to break!!

imagine that before entering stimsim, i think spending my entire life staring happily at a virtual wall would be horrifying (which is fairly accurate). then the machine modifies me, and then afterward, we get:

i love staring at the wall.

and:

i am extremely happy that i am the sort of person who loves staring at the wall.

my first-order desire has changed, but so has my higher-order desire. i wholeheartedly endorse what happened. it sounds a bit ridiculous, but we could even climb another level:

do i want to want to want to stare at the wall?

the machine can change that too, and there’s no obvious stopping point. this is one reason later theories of autonomy have worried about manipulation, as merely checking whether your desires fit neatly together cannot tell us whether those desires are really “yours.” a completely manipulated person could be internally harmonious! so perhaps we have to look backward.

christman and how you became this person

john christman and other historical theories of autonomy move attention from the final preference to the process that created it. this makes sense intuitively - our preferences obviously come from somewhere. parents, friends, books, advertising, love, college, all of that shapes us. nobody is born with a fully formed utility function, so the relevant distinction cannot just be

(there is no uninfluenced person.) so instead, historical approaches ask whether the processes through which someone's preferences developed were compatible with self-government, or whether the person could reflect on that history without becoming alienated from the resulting values, whether the process bypassed their capacities for reflection, and so on.

this gives us a much better distinction between three versions of stimsim. think abt it this way: in version 1, someone secretly rewires you. in version 2, you know exactly what the system is going to change and consent to it. in version 3, you deliberately design the person you want to become. the final brain state could theoretically be identical in all three cases, while the third feels much more “autonomous” than the first even if it really isn’t. and yes, you could probably argue this ties a little into the free will stuff. what i’m saying is maybe what matters is present-you authored the transition as opposed to just future-you endorsing the transformation. so something like

that is much more satisfying, but then unforch it creates another problem.

what if future-you changes their mind?

derek parfit has a really good example for this. imagine a young russian nobleman who is a socialist when he is young and wants to give away his future inheritance, and he also predicts that when he gets older and richer, his politics will probably change and he will want to keep the money. so he tries to bind his future self in advance and not let himself do so.

young nobleman:

give away my estate. if older me tells you otherwise, ignore him.

years pass.

older nobleman:

i have changed my mind. please give me my property.

who has authority here?? it’s not immediately obvious. parfit uses cases like this as part of a much larger argument that personal identity is less philosophically simple than we normally assume. the twenty-year-old and sixty-year-old are numerically the same human being, but there can still be meaningful psychological distance between them, which then absolutely wrecks the simple “the person who consented to the transformation gets final authority” answer.

imagine i voluntarily enter stimsim and say:

change me permanently into joy₂. under no circumstances should you let joy₂ undo the change.

then joy₂ wakes up and says:

this was a horrible mistake. please change me back.

if autonomy means respecting my original choice, the machine should refuse, but also if autonomy means respecting the wishes of the currently existing person, the machine should reverse it, and we cannot respect both at the same time.

l. a. paul, sometimes choosing changes the person doing the evaluating

l. a. paul's work on transformative experience blew my mind a little bit bc of how well it nails the problem. some experiences don’t just give you new information they also change who you are. her most famous ordinary example is becoming a parent. before having a child, you can learn facts about parenting, ask parents what it is like, estimate costs and benefits, but you do not yet know exactly what it will be like for you to become the sort of person produced by the experience. parenting itself may change your priorities, emotional attachments and sense of self.

so we might have:

before the transformation, and

afterward, but the problem is that ($u_0$) and ($u_1$) are not necessarily the same ruler. paul argues that this makes some transformative decisions difficult to fit into ordinary decision theory, in that the choice partly determines which future preferences will exist to evaluate its outcome.

this is almost like stimsim in miniature, no? the machine is just making the transformation explicit/programmable in that you’re going from “which future will satisfy me the most?” to “which future evaluator should i create?” which is a totally separate kind of optimization problem.

we already face this in medicine

imagine someone says while fully competent, “if i eventually develop severe dementia, i do not want my life prolonged” but then years later, that person develops dementia but seems happy, loves seeing her family, enjoys her days, and wants to keep living.

should the earlier person's values rule, or should the present person's interests rule? ronald dworkin's idea of precedent autonomy gives significant weight to the values and decisions of the earlier competent person, but agnieszka jaworska challenges a simple version of this. she argues that people who have lost some decision-making capacities can nevertheless remain capable of genuinely valuing things. if someone still cares about family, experiences attachments and has meaningful present interests, those values have moral importance rather than simply being erased by an earlier directive. this is literally the stimsim problem without the stimsim, with past-you saying one thing, present-you sincerely saying another, and no “real you” sitting above both of them.

so every rule is wrong…?

you could say “let the latest preference win” but it’s like, change someone until they love their new condition, and apparently you have done nothing wrong. “let the original preference win” but then an eighteen-year-old gets permanent dictatorship over the eighty-year-old person she eventually becomes. “let the higher-order preference win” but the machine can rewrite that too. “respect whatever transformation the person originally consented to” but then a past person can create a future person who desperately wants escape and has no right to get it. you could say “just maximize happiness across the whole life” but paul's problem comes straight back, in that whose conception of happiness are we using? if the intervention itself changes the utility function, our ruler changes while we are measuring with it. so clearly we are probably protecting the wrong thing.

autonomy and becoming someone else

we normally think autonomy means “let me choose what i want.” but in a world where technology can modify what i want, that is a circular definition, so i say we use preference sovereignty as the object here instead.

preference sovereignty would mean openly wanting books to change you in some way, and relationships, and letting experiences you don’t yet even understand let you care about things you do not currently care about. the important thing is preserving your capacity to reconsider the transformation.

think about your future possible selves as a space

where different choices move me around the space. ordinarily we evaluate a choice based on how good the resulting state is

but perhaps we should also value how much meaningful access to future selves remains afterward

where ($\Omega(\mathcal S_{t+1})$) is some measure of future preference option value. btw this does not mean having the maximum possible number of personalities is automatically good, only that irreversible preference changes should carry an additional cost when they destroy the future person's ability to evaluate, reject or revise the transformation that created them.

suppose two procedures both make me enormously happy. procedure a makes me happy while leaving me able to understand my old preferences, compare them with my new ones and voluntarily reverse the procedure. procedure b produces the exact same happiness but deletes my ability to want reversal, destroys my memories of ever objecting, and makes me regard anyone offering to restore me as insane. a normal utility calculation might call these equivalent (even though they’re obviously not), and the preference-sovereignty view i’m talking about will definitely favor a, which preserves the option.

why should it matter, though? if b-me is happy and will never want to go back, the option a preserves is one nobody will ever use. i think the answer is that the value of the option was never about using it, it's about what the option lets us know. whether a transformation was good is exactly the thing we're uncertain about when we choose it. under a, if the change was a mistake, there is still someone positioned to notice. under b, the change is good only if my original guess happened to be right, and the procedure has destroyed the only evidence that could ever show otherwise. b-me's contentment can't count as confirmation, because b-me's contentment was one of the procedure's outputs. it's like a scientist who deletes every dataset that could falsify his theory and then points out that no data contradicts it, or a government that abolishes elections and then notes that nobody has voted it out.

manipulation, what’s real and fake?

i don’t think the problem with extreme stimsim is necessarily that “happiness is fake.” i actually think chalmers is convincing that “virtual” and “fake” should not automatically be treated as synonyms. a rich virtual civilization containing genuine relationships and agency could be meaningful in basically every important sense.

the problem is also not that our preferences change, bc our preferences are changing constantly anyway. i was kind of hinting at this in the past couple sections, but it’s probably more like preference lock-in: preference changes that remove their own possibility of reconsideration.

a preference transformation is self-locking when it simultaneously makes you want ($X$) and makes you unable, unwilling, or epistemically incapable of seriously reconsidering whether wanting ($X$) was a good transformation.

the extreme version looks like:

plus

now future-you cannot function as an independent check on the decision, because part of what the decision did was eliminate the check. this is why “but future-you will be happy about it” is not sufficient. of course they will be happy about it when you designed them to be.

on post-scarcity

initially, the interesting thing about stimsim seemed to be that it could eliminate scarcity. then it seemed like maybe some special things would remain scarce whether it’s real achievement, real relationships, status, authenticity, voluntary recognition. but if we allow arbitrarily sophisticated simulations, autonomous artificial people and psychological modification, even those are not obviously the hard limit.

the fundamental problem is that technological progress eventually stops merely answering human desires and starts participating in the process that creates those desires. bostrom's post-instrumental world asks what humans do when machines can do everything useful for us. and now we can add, “what happens when machines can decide what “useful” feels like?” at that point, a perfectly optimized civilization could consist of billions of beings in extremely simple environments experiencing unimaginable fulfillment. maybe they adore counting virtual pebbles. maybe every one of them, if asked, says, “i would never want to live any other way! i am super happy!” if you’re a pure experientialist, perhaps this really is utopia, but if you’re anyone else, this is really really concerning, and it’s not even that “their experiences aren’t real.” the scary part is that their path through preference-space has been permanently closed behind them.

in any case, my point - we’re probably not getting perfect stimsim anytime soon, but we are building increasingly powerful systems whose job is to predict what we will click on, show us things that influence what we want, recommend what we should consume next, act as companions, and eventually perhaps intervene much more directly in our emotional and cognitive lives. there is a huge difference between an ai that gets very good at giving you what you want and an ai that gets very good at making you want whatever it can most easily give you. those objectives can look super similar from the inside, hence the need for the distinction. mybe the final scarce resource is the ability to still become someone who would choose differently!

for the philosophical lineage behind this: robert nozick introduced the experience machine in anarchy, state, and utopia (1974); harry frankfurt developed the idea of second-order desires and volitions in “freedom of the will and the concept of a person” (1971); derek parfit's reasons and persons (1984) contains the russian nobleman and his broader account of psychological continuity; john christman's work on historical autonomy asks how the origins of preferences affect whether they count as autonomous; l. a. paul's transformative experience (2014), “who will i become?” (2020), and especially “whose preferences?” (2020) deal directly with choices that transform the chooser's future preferences; ronald dworkin and agnieszka jaworska take opposing approaches to conflicts between earlier autonomy and later values in the literature on advance directives; david chalmers's reality+ (2022) argues that virtual life need not be fake or meaningless; and nick bostrom's deep utopia (2024) explores what value might look like in a genuinely post-instrumental world.