Valence vs Priority Disagreements in Public Debate; Plus, The Gambit of Extreme Negative Utility

1. In public discourse, usually, most people do not have valence differences when they disagree, but rather differences of priority. This is especially true in politics. Concrete example: most people agree that racism is bad. The disagreement tends to be about how important it is to eliminate racism right now, relative to other problems. A valence disagreement occurs in this case when a white supremacist says no, actually, racism is good. Valence disagreements are of course much more intractable. Online discourse over the past few years has raised disagreements about priority to the same level of intractability, by turning priority disagreements into valence disagreements, e.g., "You shouldn't be talking about anything else right now but keeping out illegal immigrants. I don't care if you say you're against illegal immigration, if fighting illegal immigration isn't your number one priority, then you're actually for illegal immigration."

2. One trick to make your argument demand top priority is to claim that there is a massively, or even infinitely negative consequence for not adjusting one's actions in the ways required of a rational actor if the argument is true - or claiming that there is such a consequence even for ignoring the argument. Organized religion makes the most famous such claims, but the outcomes feared by certain political ideologies (if they don't get their way) can approach the same severity. This is the Gambit of Extreme Negative Utility (GENU.) Related to this, a counterargument to Pascal's wager is that you don't know which version of which religion is the right one, but as I saw a very clever Christian argue online, you have to assume infinite religions to choose from for the expected utility to work out in favor of ignoring Pascal's claim (although this still doesn't tell us which one to choose.) So what do we do? Do we set an arbitrary cut-off for utility beyond which we assume someone is lying to get our attention? This seems very dangerous, unless we think there are no rare events with far greater negative utility than we expect beforehand. So if we throw out Hell, Orwell's boot on a human face forever, and white supremacists' fear of a world overwhelmed by non-white barbarism, don't we also have to stop worrying about the Great Filter, the technological singularity, and CRISPR-derived biological weapons made by suicide cults? [Added later: turns out that Pascal's Mugging, a concept in the rationality world, captures the essence of this - any risk/benefit scenario can be overwhelmed by claimed arbitrarily large or infinite negative utility, which Christians like Pascal call "Hell".]

3. It matters which arguments we spend time and attention considering, because although humans seem to have an implicit assumption of infinite time and attention to consider arguments, of course we do not.

Hence, most authorities use some form of agenda-setting and distraction to influence discourse.

Why Do People Remain Loyal to a Losing Team?

Cross-posted to the MDK10Outside and the Late Enlightenment.

tl;dr Sports fan behavior is explained by a combination of constant identity-forming team loyalty which is an end in itself, and status signaling by association which is modulated by team performance. These two factors differ between individuals and are associated with different cognitive styles, with constant loyalty more associated with moral foundations and intransitive preferences.

It's been observed that you can tell who a team's true fans are by noticing who remains loyal to the team even when that team is losing. I think this is meaningful, but it does beg the question: what are those fans getting out of it?[1] Of course any speculation about this must mention the very real example of the Cleveland Browns, who over the past 2 years have a 1-31 record, and this year after going 0-16 they were on the receiving end of a sarcastic "perfect season" parade.

Humans get utility from associating with others with high status. Much of the happiness that a sports fan gets from their emotional connection to their team derives from this, and many observations are consistent with what a status-by-association theory would predict: fans are happier when their teams win because they feel high status and can signal higher status, they engage in extreme dominance displays when their teams win important contests (i.e., people acting like idiots as they come out of a championship game if their team won, yelling, jumping on cars, setting off fireworks) but not if they didn't win, they attend games more when the team is winning and less when the team is losing, and they wear branded gear to identify themselves with the team and otherwise let others know of their association.[2]

But this theory falls short of explaining why, for example, there is any such thing as a team's consistent fanbase. By this model, everyone should just cheer for the best team, game by game (or even play by play!) It especially doesn't explain why the the Cleveland Browns have any fans left at all; supposedly they're a football team but I've seen a number of convincing arguments against that, for instance, every game of the 2017 season. During an 0-16 season you would expect that if fandom is about fully rational people maximizing utility by associating with high status teams, the fans would stop posting on forums, they would put their gear away and deny to others that they were fans, and the stadium would not just have lower attendance, it would be completely empty. Yet this is not what happened.

I think the answer here very likely has to do with the gap we see between two types of beliefs/behaviors that often produce apparent impasses in other domains of life, especially religion and politics, the intensity of which differs between individuals. This gap in rational and more instinctual behavior will seem very familiar to readers of books like Jonathan Haidt's Righteous Mind, or Simler and Hanson's Elephant in the Brain. Humans demonstrate some domains in their cognition which are inflexible and impervious to reason - to use Haidt's categories, harm, fairness, loyalty, authority, and purity. By "inflexible" I mean "not open to discussion, or conversion into money or other goods/services." For example, you likely do not believe that murdering children is morally acceptable. Are you interested in hearing arguments about why it might be morally acceptable? If you would never consider such a thing, and you're uncomfortable that I would even suggest it in a thought experiment, you're showing inflexibility in discussing it. Okay - would you kill an adult for $50,000? I see that also upset you, I'm sorry to have opened with such a low offer! $75,000 then? You're being inflexible (I hope!) in reacting by thinking "It's not about the number!" Okay, what's the conversion rate between adults and children? Forget murder, how about urinating on a picture of your family for money? etc., you get the point. "Inflexible" means it can't even be suggested as open for discussion, which includes not being allowed to convert between moral-foundation-violating acts and money, or between different types immoral acts. (A favorite of action movies and dramas to demonstrate the extreme evil of an antagonist is to have them force someone declare the relative value of immoral acts, e.g. Sophie's Choice.) To connect back to the abstract, the philosophical term for having values that cannot be negotiated, and for which there is no relative value like this, is that they are intransitive.

I took you on this little tour of moral darkness to illustrate that morally normal humans do not adhere to consistent rationality, and the ones that actually do are psychopaths.[3] (You may be interested to know that Haidt found that when he surveyed the business students he was teaching, they scored low on every single moral dimension, taught as they are that everything is negotiable.) So what does all this have to do with the Cleveland Browns? Many of us have noticed that "hardcore" sports fans - the ones who stick around with long faces even when the Browns are losing, and falsify the first model above - tend to have certain personality and cultural characteristics that fit well with some of these inflexible moral foundations: they tend to be more religious, more nationalistic, more conservative and more valuing of loyalty and authority.[4] Sports fans rarely become hardcore about a team after entering adulthood, and very often there is a family lineage of fandom - and these are exactly the times and ways in which characteristics of core identity are formed. Also telling, while there were about 3,000 people who showed up for the Cleveland Browns parade, there were many fans who were quite angry about it - but online objections were mostly that it was "embarrassing". (No mention of the 0-16 record that inspired the parade.)

Before I put into words what might be motivating them and make predictions, here's a summary of the two kinds of of beliefs, producing two kinds of motivation. While these beliefs exist in everyone, there is going to be a distribution in the population, with one category of beliefs dominating the fandom-related cognition of some fans, and the other category dominating that of others.

HARDCORE FAN	CASUAL FAN
motivated by moral foundations	by utility calculations
end in themselves	deliberate, external goal-oriented
higher value on loyalty	lower value on loyalty
adopted in childhood, maybe from family	adopted voluntarily in adulthood
not negotiable	negotiable
central to identity	not central to identity
unwilling or unable to verbalize	position clearly verbalized
more often encountered in person	more often encountered online
sees casual fans as untrustworthy, sleazy	sees hardcore fans as stupid, gullible

Of course it's a spectrum, and every fan is somewhere on this spectrum, but many of us clearly lean toward one or the other end. (If you're reading this, you're more likely in the right column than the left.)

To summarize the hardcore fan: he is motivated by more basic, instinctual moral drives, especially loyalty. Being a good fan is an end in itself, and an offer to burn a team jersey, to cheer for the other team, etc. in exchange for money is likely to not only be immediately refused but to provoke active offense. These fans consider their fandom a crucial part of their identity, to the extent of including team-related themes in their weddings or mentioning it in obituaries ("he lives and dies by the Browns"; "a Browns fan to the core.") He can get uncomfortable when the business aspects of a professional sport are discussed and overshadow the games on the field. Asking him to explain his fandom will be met with puzzlement, anger, or a jumbled set of team cheers and slogans, in the same manner as a person asked to explain why they are patriotic or follow a certain religion - "If I have to explain it to you, you'll never understand." And finally, because tribal loyalty sentiments are more warning-barks or team cheers than any kind of actionable proposition, you're more likely to hear such sentiments when talking to him in person, where the nonverbal (affect-laden and irrational) part of communication dominates. He will be a fan for life. When the bandwagon people disappear during losing seasons the hardcore fan says "Good riddance, good-time Charlie."

To summarize the casual fan: he is motivated by utility calculations about external goals (this team might win this year so I'll cheer for them, maybe I can make friends this way, maybe I'll look successful if I follow a good team.) He doesn't see what's impressive about staying loyal to losers, and really doesn't understand why making fun of your team when they lose is shameful or embarrassing. He probably picked up his fandom after college, maybe when he moved to a new city. He probably don't care either way about the business dealings of the team. If someone offered him money to stay home from a game or burn team logos, he would seriously consider the offer. He doesn't introduce himself to strangers as a fan, and five years from now he might not be following the team, or might not be following the sport at all. He can give clear reasons why he started following the team, and you're more likely to hear from people like him online. He shakes his head at the hardcores who keep shelling out cash for losing teams' jerseys.

Both the hardcores and non-hardcores gain utility in proportion to the team's performance. A team's performance can be negative, causing you to lose utility by associating with them.[5] But there must be another source of utility for the hardcores, who somehow gain utility from the association no matter the team's performance - and that source of utility is a constant ability to demonstrate loyalty, period, to others as well as to themselves to reinforce their own identity. And this signal is most informative when your side is losing.[6] Speaking quantitatively, in the utility equation for this model, there are two terms, loyalty (a constant for everyone, hardcore or not), plus the product of team performance times associative utility. Associative utility is how much your utility changes per team winningness. Both loyalty and associative utility vary by individuals, and team performance of course is determined by the team. The equation looks like this:

Total utility = Loyalty-based utility + (Team performance * associative utility)

Team performance can be positive or negative. For the hardcores, loyalty is such a large term that it doesn't matter how negative team performance is, loyalty will alway be greater and the total utility will always be positive (this could be the definition of "hardcore", "rain or shine", etc.) Further toward the other end of the spectrum, the value of loyalty signaling decreases and the team performance makes more of a difference in whether people keep following the team. It's also worth pointing out that this explains people who don't care about sports at all, because they have zero loyalty and zero associative utility - that is, it doesn't matter how much the team wins, they still won't care.


PREDICTIONS

Many of these predictions seem trivial, but the point is to relate these predictions to specific components of the hardcore fan's motivation structure as noted in the table above, which would be more informative.

• While utility is hard to measure directly, there are good proxies for it, like revenues, attendance, or Nielsen ratings. Given that there will be a distribution of hardcore to non-hardcore fans, there will be a non-zero floor to revenues so even 0-16 teams don't go to zero, as we observed. If we graph all of the teams on performance vs utility proxy, I would expect a mostly linear-looking scatter plot with an increase in the slope at the good end, for those teams with some expectation of a national championship, and possibly a flattening at the bottom. This may depend more on expected utility (if fans are pleasantly surprised by a win vs. they expect their team always to win.) I plan to try to collect some kind of utility-proxy data and see if this is in fact the case.
• In general a sport will be more successful in inspiring loyalty, the more similar it is to tribal warfare (always a reliable revenue stream for every team); maybe this is why football has eclipsed baseball as the national pastime.
• The more hardcore, the more they will pay attention to the outside charity activities of their own team, and the more outraged they will be by disloyalty-demonstrating acts, e.g. kneeling during the national anthem. They will also be more interested in the moral failings of opposing teams, especially rivals.
• The more hardcore, the less they will be interested in statistics, especially of other teams, even ones their teams are playing in important games.
• The more hardcore, the greater the difference in their interest in a player when he is on their team, vs. after he is traded. That is, hardcores think each of their players is a great person on and off the field - when he plays for their team - and any suggestion that they'll stop caring about him the second he is traded is likely to be met with hostility, but in fact this is the behavior they demonstrate. (He will also be annoyed when asked why, or when Seinfeld is cited - "Essentially you're cheering for clothing.")
• The more hardcore, the more they will feel sad or angry after a loss, and the more likely they are to attend or watch the next game despite having been very sad or angry at the last game's outcome.
• The more hardcore, the less tolerant they will be of fans behaving negatively toward the team, even when the team loses (very concrete and contra expectations here: you might expect hardcore fans to support a parade showing anger against the people making their Browns lose, but it seems to be exactly the opposite. Parallels to gay marriage here too: how exactly does the 0-16 parade degrade your fandom, when you didn't attend?)
• The more hardcore, the more they will confuse the team with a government agency or public good (i.e., demanding that the city finance a new stadium.)[7] More recent teams with cities that have highly educated and/or mobile populations (i.e. the coastal Pacific) will therefore find that they can't get what they want from those cities, because the voters don't care (Seattle, San Francisco, San Diego) where other cities filled with less mobile, less educated people would crucify their mayor for allowing a team to leave on their watch.
• It's often been noted that the Midwest with its brutal early winters has far more rabid sports fans than the mild West Coast. One possibility is that the loyalty-demonstration value of attending every game is diminished when all of those games are 70 F and sunny, vs some of them being freezing cold. (Think of the people who still wait in line in the cold and dark on Black Friday morning to buy things for their families. They do know that Amazon exists. So what do you think they're really doing?) Of course there could be a climate-independent cultural difference between east and west coast, but the model's prediction would be that Miami has equally low loyalty.
• The more hardcore, the more they will be upset if a star player leaves for another franchise, or the whole team moves to another city, and they use words like "betrayal."[7]
• The more hardcore, the less tolerant they will be of long-term, off-field strategies, especially ones that alter play and result in on-field losses. (Both the 2008 Detroit Lions and 2017 Cleveland Browns had 4-0 preseasons, then went 0-16. Tanking (here and here) and/or salary cap manipulation? Difficult to explain as mere incompetence. And if it were confirmed that this is what is happening, the hardcore fans would be angry; casual fans might say "Huh, that's kind of clever, although it means you've been putting a bad product on the field." "My team is not a 'product'!" the hardcore fan says.)
• I'm not sure what to predict about the impact of hardcoreness on betting. The hardcores' loyalty may make them become overconfident in their team's performance. On the other hand, moral foundations-related beliefs are often kept carefully separate from anything affecting real-world decision-making. By that I mean: sacred beliefs are often more tribal chant than actionable proposition, and in general, people desperately avoid any bet that touches their moral foundations (next time someone makes a verifiable statement about religion or politics that you disagree with, offer to bet them, and see what happens. Typically they backtrack to a non-verifiable version of what they said, and/or get very offended that you would "cheapen" such an important matter by betting on it - which are all moves to avoid testing their belief.) Then again, the hardcore fans presumably know more about their team than most others, which means they should be more confident in their predictions, and be more willing to bet. Consequently they may be less willing to bet proportional to their claimed confidence, than would a casual fan with equal knowledge of the team would be. In my one test of this during March Madness, I found that self-identified fans did more accurately predict the outcome of a game involving their team than non-fans, but I collected no information on willingness to bet.

Footnotes

[1] This very article is diagnostic. By trying to dissect loyalty, instead of taking it as an obvious good and discussing it in the context of a specific team, I mark myself as someone with a small loyalty term in my equation - whereas people whose sports utility equation is dominated by loyalty would not understand, and/or be actively be offended by, a question like "What do you get out of being a fan of your team?"

[2] One might argue that a purely rational human being would ignore sports altogether - what do a bunch of guys chasing a ball on a field somewhere else in my city have anything to do with me, I've never even met them! - and I'm sympathetic to that argument.

[3] I hope no one read the paragraph about the price of murder and thought, "Hmmm...What is my price to kill someone?" In the case of exemplar psychopath Richard Kuklinski, he got positive utility from harming people so he kept doing it even after he ran out of work.

[4] When people do not have VNM-consistent rationality (that is, they have these inflexible, non-negotiable, non-fungible beliefs - i.e., intransitive preferences) - they can be turned into money pumps, by observant and unscrupulous characters who can carve their motivation structure at the joints, i.e. focusing on the the inconsistencies. While this has been reproduced now in artificial settings, not only salespeople but politicians have been doing it since the dawn of civilization. The NFL and in particular the Cleveland Browns are doing exactly this to the fans by exploiting the intransitive preference of loyalty, and I would be very surprised if their marketing does not already have a model of their fans and spending patterns similar to what I've described here. Another follow-up is to look for literature on whether psychopathy allows one to see these disconnects more easily, or (hopefully) the ability to see them and the willingness to act on them are unrelated and therefore form a mercifully narrower sliver on a Venn diagram of the population.

[5] There's probably a Markovian/hedonic treadmill effect here too, where the utility multiplier from a team's win is not constant but rather influenced by expectations based on the team's record. Next year if the Patriots go 9-3, fans leaving a game after a win won't be as happy as Browns fans if the Browns have the same record.

[6] Remember Karl Rove dragging out the 2012 election night broadcast and refusing to accept the outcome, seeming a little nuts? But simultaneously advertising to ten million Republicans watching that he never ever gives up. Say what you will about Karl Rove, but "bad strategic thinker" was not among the many epithets hurled at him.

[7] When the Baltimore Colts were about to move to Indianapolis in 1984, the city actually tried to pass an eminent domain act (!) to take over the team, but the Colts escaped with the team's property under cover of darkness the night before. Other teams like the Chargers have found a much more lukewarm reaction on threatening to leave, and found themselves without many fans.

[8] While I wrote this post I was wearing a Garfunkel and Oates sportsball T-shirt, so you can guess which end of the spectrum I'm near.

If You Take Parfit Seriously, You Should Commit Yourself To Creating Superintelligence

Cross posted at Speculative Nonfiction and The Late Enlightenment.

Derek Parfit makes the argument that if utilitarianism as it is commonly understood is to be taken to its conclusion - the greatest good for the greatest number - that mathematically we should care not just about making individuals happy, but making more individuals, to be happy. If you can have a world of a billion people all just as happy as a world of a million people, then that that's a no brainer.

The problem is when you get to the math of it. The "repugnant conclusion" that if the total amount of happiness is what matters, then you should favor numbers over quality of life. That is, a world of a hundred billion people with lives just barely worth living is better than a world of a hundred people with great lives - because the great lives are probably not a billion times greater than those of the hundred billion in almost total misery.

The obvious objection is that you're talking about theoretical people when you talk about those hundred billion. The counterargument is that we do care about theoretical people - our descendants - and you might already make environmental decisions to preserve the environment for the happiness of your grandchildren; right now you avoid (hopefully) littering the street to avoid upsetting people you've never met and will probably never meet.

There are other objections of course; for instance, that experienced happiness in an individual is what matters; otherwise slave plantations could be (in fact, probably are) morally acceptable.

But following Parfit's repugnant conclusion to its end, if the total amount of utility is what matters, then increasing the amount of utility possible to be experienced also matters. That is to say, there is no reason to stop at considering theoretical people, but rather we should consider theoretical kinds of experience, and theoretical kinds of experiencers. And there is nothing in Parfit's thesis provincial to or chauvinistic about humans. (If there were, that might solve the problem, because you could say "the closer something is related to me, the more I should be concerned with its happiness" - me and my brother against my cousin, et cetera - which, at very close genetic distances, is in fact what most humans already do.)

Therefore, we should try to make a world of a hundred million bipolar (manic) people who can experience hedonic value far in excess of what most of us ever do (assuming we can keep them manic and not depressed.) Or, even better, created an artificial superintelligence capable of experiencing these states, and not devoting all our resources to creating as many copies of it as possible. But cast aside those constraints - if you believe it is possible for a self-modifying general artificial intelligence with consciousness (and pleasure) to exist, then by Parfit, the only moral act is to give up all your recreation and resources to live in misery and dedicate your life to the single-minded pursuit of getting us one second closer to the creation of this superintelligence. The total suffering and happiness of life on Earth up until the moment of the singularity would quickly shrink to a rounding error, compared to the higher states these replicating conscious superintelligences might experience. Therefore, if you are not already singlemindedly dedicating yourself to bringing such a superintelligence to life, you are forestalling seconds of these agents' pleasurable experiences (which far offset your own suffering and maybe those of all living things) and you are committing the most immoral act possible.

This problem is superficially similar to Roko's basilisk (in the sense of your actions being changed by knowledge of a possible superintelligence) but I think it should still be called Caton's basilisk.

As a result of these objections, I do not think we need to take the repugnant conclusion seriously, and I do not think not dedicating yourself to creating a super-hedonic superintelligence is immoral.

Toward a Physical Measure of Utility

"Electroencephalographic Topography Measurements of Experienced Utility", emphasis on experienced. Pedroni A. et al, The Journal of Neuroscience, 20 July 2011, 31(29): 10474-10480. The response they measured unexpectedly increased disproportionately increasing reward, i.e. it did not demonstrate diminishing returns but rather the opposite.

A measure of the mismatch between decision and reward utility, and understanding its biological basis and how it differs between individuals, would be excellent for psychology as well.

A Possible Solution to Hedonic Recursion in Self-Modifying Agents: Knowledge-Driven Agents

A problem of systems with goals which can self-modify those goals is stability and survival over time. It seems to be a positive that a system could identify goals which are in conflict and modify them to behave consistently in its own benefit, although conflicts can also be solved in favor of the less-survivable goal (example below). The stronger danger is that an agent that can "get under its own hood", so to speak, is able to short circuit the whole process and constantly reward itself for nothing, breaking the feedback loop entirely. This is called hedonic recursion.

An example of conflicting goals: a person wants to be healthy. The same person also really likes eating chocolate. A person with access to his own hardware could resolve the conflict either by modifying himself to make it less fun to eat chocolate, or by to not care about the negatives of being unhealthy. It seems obvious that the first option is the better one for long-term survival, but in the second case, after you modify yourself, you won't care either. And even this second resolution is far less dangerous than outright short-circuiting one's reward center, getting a shot of dopamine for doing nothing. And this short-circuit option would be on the table for a fully self-modifying agent. And, for any self-modifying goal-seeking agent, this will very quickly be realized.

Fortunately or otherwise, this hasn't been a problem for life on Earth yet, because the only way living things here can get rewards is through behavior - because we cannot modify ourselves. The things that cause pleasure and pain are set in stone (or rather, in neurons) and only through behavior (modifying the external environment as opposed to yourself) are rewards obtained. But there are hints in higher vertebrates of small short-circuits - nervous system hacks they have stumbled across which tweak their reward circuits directly. Elephants remember the location of, and seek out fermented fruit (to get happily buzzed). Elephant seals dive rapidly to unnecessary depths to cause narcosis (we think). Primates (including us) masturbate incessantly. And humans specifically have found things like heroin. As we humans learn still more about ourselves and learn how to manipulate the neural substrate, this may be changing. Consequently, if ever humans are able to alter our nervous systems directly and completely, ruin may follow quickly. And indeed, this has been shown with rats: give them the ability to directly stimulate their reward centers with electrical current, and they will do so to the exclusion of all other activities, including those required for survival - hedonic recursion.

In a great discussion at the Machine Intelligence Research Institute website, Luke Muehlhauser talks to Laurent Orseau about how to solve the problem of what kinds of self-modifying agents avoid this problem. The discussion is about how to build an artificial intelligence, but it applies to biological nervous systems that, like us, are increasingly able to self-modify.

One of the theoretical agents Orseau conceived was a knowledge-driven, as opposed to a reinforcement-driven agent, a goal-seeking agent, or a prediction-confirming agent:

...knowledge-seeking has a fundamental distinctive property: On the contrary to rewards, knowledge cannot be faked by manipulating the environment. The agent cannot itself introduce new knowledge in the environment because, well, it already knows what it would introduce, so it's not new knowledge. Rewards, on the contrary, can easily be faked.

I'm not 100% sure, but it seems to me that knowledge seeking may be the only non-trivial utility function that has this non-falsifiability property. In Reinforcement Learning, there is an omnipresent problem called the exploration/exploitation dilemma: The agent must both exploit its knowledge of the environment to gather rewards, and explore its environment to learn if there are better rewards than the ones it already knows about. This implies in general that the agent cannot collect as many rewards as it would like.

But for knowledge seeking, the goal of the agent is to explore, i.e., exploration is exploitation. Therefore the above dilemma collapses to doing only exploration, which is the only meaningful unified solution to this dilemma (the exploitation-only solution leads either to very low rewards or is possible only when the agent already has knowledge of its environment, as in dynamic programming). In more philosophical words, this unifies epistemic rationality and instrumental rationality.

There's a lot more to the argument (you really should read it), but there are several points to be made with respect to this paper.

1) These are not fully self-modifying agents. In this environment their central utility function (reward, knowledge, etc.) remains intact. The solution is to collapse exploitation (reward) into exploration (outward orientation). The knowledge agent can only get buzzed off of novel data, so it has to keep learning. But exploitation and exploration are two conceptually separable entities; so if modification of the central utility function is allowed, eventually the knowledge agents will split exploration and exploitation again, and we're back to reward agents. (At the very least, given arbitrary time, the knowledge agents would create reward agents, to get more data, even if they didn't modify themselves into reward agents.)

2) Orseau's point is taken that if novel data is what's rewarding them, as long as that utility function is intact, they cannot "masturbate" - they have to get stimulation from outside themselves. In another parallel to the real neurology of living things, he states "all agents other than the knowledge agent are not inherently interested in the environment, but only in some inner value." The core of utility is pleasure and pain, which are as much an inner value as it is possible to be. Light is external data, but if you shine a bright light in someone's eyes and it hurts, the pain is not in the light, it's in the experience the light creates through their nervous system. Utility is always an inner value. The trick of the knowledge-based agents is in pinning that inner value to something that cannot arise from inside the system.

3) The knowledge-based agent is maximizing experienced Kolmogorov complexity. That is to say, it wants unexpected information. Interestingly, Orseau says this type of agent is the best candidate for an AI, but such an agent could never evolve by natural selection. He points out that the agents he's using are immortal and don't suffer consequences to their continued operation by any of their experiences. But an agent that can be "damaged" and that is constantly seeking out unexpected environments (ones it doesn't fully understand) would quickly be destroyed. In contrast, Orseau commented that the reinforcement-based agent ends up strongly defending the integrity of its own code. Evolutionarily, any entity that does not defend its own integrity is an entity you won't see very many of (unless the entity is very simple, and/or the substrate is very forgiving of changes. This is why you see a new continuum of viral quasispecies appear after a single year, but why animal species reproductively isolate and you shouldn't hold your breath for, say, hippos to be that much different any time soon.)

4) No doubt real organisms are imperfect amalgamations of all of these agent strategies and more. To that end, Orseau found that the reinforcement (reward)-based agent acts the most like a "survival machine". In his system, I would wager that living things on Earth are reinforcement-based agents with a few goals sprinkled in. (There are many animals, including humans, that startle when they see something snake-like. fMRI studies have even suggested that there are actually specific brain regions in humans corresponding to certain animals - it's really that klugey.) However, of further interest here is that even between humans there are substantial differences in how much utility is to be gained from unexpected novelty, some of them known to be genetically influenced. Some of us are born to be surprise-seeking knowledge agents more than others. The meaning of having multiple genes not at fixation would be useful to investigate. (Only recently valuable in evolutionary time, now that our brains have enough capacity?)


If your goal is to create agents that act to preserve and make more of themselves and remain in contact with the external environment rather than suffering a hedonic recursion implosion, there are a few stop-gaps you might want to put in place.

1. Make self-modification impossible. This is the de facto reality for life on Earth, including us, except for a few hacks like heroin. Life on Earth has at least partly done this, converting early on from RNA to the relatively inert DNA as its code.

2. Build in as strong a future orientation as possible, with the goal being pleasure maximization rather than pain minimization. That way pleasure now (becoming a wirehead) in exchange for no experience of any kind later (pain or pleasure, meaning death) becomes abhorrent. You might complain about the lack of future orientation in humans* but the fact that any organism has any future orientation is testament to its importance.

It could be that we haven't seen alien intelligences because they all become wireheads, and we haven't seen alien singularities expanding toward us because Orseau's E.T. counterparts built their AIs to seek novelty, and the AIs destroy themselves in that way.


Speaking of poor future orientation where reward is concerned: I have seen a man literally dying of heart failure, in part from not complying with his low-sodium diet, eating a cheeseburger and salty, salty fries that he brought with him into the ER.

Utility Calculations Are Not Allowed For Sacred Things

One place where the vast majority of human beings fall short of attaining full Homo economicus status is in the perseverance of sanctity, for certain values or objects; for example, the protection of children, the value of human life, and the evil of inflicting pain for its own sake. And the conclusion of this post is troubling from a rationalist, post-Enlightenment standpoint: that it's exactly our most critical values where reason fails, and indeed, must fail if those values are to be preserved. In The Righteous Mind, Jonathan Haidt points out that sacredness is one of the six moral foundations of human beings, whether or not we think of it in religious or secular terms.

Please note: to make my point I need to provoke an emotional reaction in you, the reader, so it's going to get a little rough when I violate universally sacred values.

A good working definition of sacredness (in a religious sense or otherwise) is for an object or value to meet at least one of these two conditions:

1) Its truth or necessity cannot be questioned. To do so is to cause moral outrage, and a dramatic devaluing of the questioner's perceived moral character. (Hence my disclaimer above, even though I'm clearly speaking in the abstract.)

2) The object or value cannot be involved in transactional discussions, with either non-sacred or other sacred values and to do so is morally outrageous; questions of valuation or exchange are off the table. (More of a sacred value violation is worse than less of the same sacred value violation, but comparisons between two different sacred values are outrageous.) In other words, "you can't put a price on [sacred value]."

These both really boil down to this: if something does not admit of utility calculations, it's sacred. (Questioning why calculations aren't allowed is a meta-calculation that is also forbidden.)

A concrete example of the first qualification: why is it wrong to torture and kill children? If somebody wants to, why shouldn't they? Maybe right now you're reading a blog and your frontal lobe is keeping you from calling me a monster for even asking that, and maybe you defended yourself against the flash of outrage by assuming I'm asking to make a point - but imagine a stranger asking you this earnestly tomorrow, in person, and really pressing you on it. I don't think you'd feel compelled to think of an explanation. For my part (and don't be a jerk and quote this out of context!) in the utility-based rationally self-optimizing way I tend to think (deliberately) about morality and decision-making, I cannot explain why it would not be okay for a sadistic psychopath to do exactly that, if they derived pleasure from it and wouldn't get caught. Obviously I feel that it's about the worst thing you can do. My logical shortcoming does not cause me to reconsider my position on the matter (even for a second), but rather to call my moral theory incomplete. I wouldn't do it or allow it to happen for all the money in the world, and I'm not interested even in being influenced in that direction for all the money in the world. There's no calculation going on around it. To me, it's sacred.

A concrete example of the second qualification: how much would that person have to pay you to torture and kill a child? You would refuse to put a price on it and would likely be offended by the question. You also would likely not want to get involved in a discussion of the relative evil of torturing and killing children, versus deliberately infecting people with AIDS. (10 AIDS to 1 torture-murder? 12 to 1? Come on, there has to be some exchange rate!) So there's just not going to be any talk about relative value. Interestingly, sacred objects do allow comparisons between units of the same sacred-value-violation (obviously it's worse to torture and kill two children than one) but there's no comparison allowed between different types of sacred-value-violations. Of course the world does not always respect our moral categories, and in point of fact when people are put in a situation where they do have to choose between sacred value violations, they suffer badly, but their heads don't smoke and sputter like broken computers; they clearly are capable of making such calculations. In Sophie's Choice, a movie that depresses me just from hearing about it (I haven't seen it and I won't) a concentration camp guard forces a woman to choose which of her two children is taken to the ovens (or if she won't choose, he'll take both). And she finally chooses, much to her bereavement of course.

A way to think about decision-making and morality is to assign utility values to things. We look at utility lost and gained and sometimes we sacrifice utility now as an investment for utility later (as I will in a few minutes when I go back to studying).

But consideration of sacred values is nothing like this. In line with property #1, it's not that the possibility of transgressing a sacred value even crosses our minds in the first place if only to be immediately rejected: "You know, this kid that woke me up in the morning by playing outside my window really made me mad. It would give X amount of utility to get out my frustrations right now and also know that I'll be able to sleep in from now on, if only I torture and kill this kid, and I even know a way to do it without getting caught. But no, I would actually feel so bad about it that I would have infinitely negative utility, so it works out in favor of non torture-murder." That's not what happens. And in keeping with property #2, though it's a dark and sad thing to say, such an act might be one of the worst things imaginable but it's still not really infinitely bad. (If it was infinite, two such violations wouldn't be detectably worse than one.) Of course it's not just child torture-murder that's not within the realm of possible deliberation, but most of our moral values, which were programmed early in life and which give us flashes of disgust or happiness, mostly quite beyond our control to change.

It's worth pointing out that the classic infinite negative utility scenario philosophers discuss is being damned to Hell as in Pascal's wager, a fate literally worse than death if you believe in Hell. But the experience of considering the horrible actions I discussed above is very different than the Pascal's wager consideration. When you imagine suffering in Hell forever, you imagine feeling bad and you become afraid. It doesn't go that far when violating a taboo is suggested to you - you don't picture, as did the moral reasoner in the previous paragraph, the way you would feel if you did such a thing, you just don't even for a second consider it.

Someone who is really making utility calculations about their decisions would not behave in this way. It's as if you're in a "possible actions store", with certain actions on a display shelf, not for sale. If we were really performing "moral reasoning" of some kind, we would at least be entertaining these not-for-sale actions, even if we never did them. The problem is that conscious, frontal-lobe based utility calculations (however they're performed) do tend to be corrosive to traditional values, because utility calculations are effective at creating successful novel actions - and this corrosion in traditional values is frequently effected through markets, once sacred objects or acts have been assigned a commensurable value. Markets are aggregates of utility-based decisions that accumulate massive power to influence actions.

This may explain why people who are otherwise very pro-free-market find market infiltration into certain arenas (especially traditional culture) to be extremely offensive - because now the tradition is subject to utility calculations, and it will surely change, and quickly. The commercialization of Christmas is an excellent example. But the clash of values in healthcare, both in patient perceptions as well as for practitioners, is a much larger and more profound one. (Philosophers like to talk about utility in hypothetical units of "utilons", but in the real world, there actually is a unit of utility that healthcare organizations and policymakers use, the QALY, quality-adjusted life year. You don't want to assign relative values for treating AIDS and cancer? Too bad, because your government is doing it at this very moment, in the real world. Of note, some QALY tables for diseases do recognize fates worse than death, although they still don't assign infinite negative utility. Incidentally I imagine that the committees that build these tables are Sarah Palin's "death panels".)

Once those values are "for sale" - once they enter into the realm of conscious deliberation and value-assignment - they're almost certainly not going back to the display shelf. In a brilliant aside in Predictably Irrational Dan Ariely makes several observations about exactly this problem, in the commoditization of social relationships, particularly by banks.

The following might at first seem a strange observation for a self-described libertarian to be making, but for those of us who think the market (and reason) is usually the best method for increasing utility, it behooves us to understand it, warts and all. Haidt later noted that the business school students he taught (and collected surveys from) were on average low in every one of the six moral foundations he describes, in terms of how their values were influenced. These are individuals who do in fact act out of rational self-interest - they're members of Homo economicus for which everything is for sale, and nothing is sacred. What's more, a study has shown that low empathy predicts utilitarian judgment. (Do note the categories there: if you have low empathy you're more likely to be utilitarian. No word on whether utilitarianism predicts low empathy.)

But my concern is not that eventually all businesspeople (and everyone else) will be lured by utility calculations into becoming child murderers. Some of our sacred values are likely to be very biologically innate, and others taught. Given its universality, protecting children is probably strongly innate. Others, for example treating certain religious or national symbols with respect, are learned and likely subject to erosion over time. The bigger question is what this does over time to our ability to cooperate and sub-optimize over the long-term.

(More interesting relationships between pre-rational neurology and moral behavior here.)

Punishment Strategies for Games With Indefinite Lengths

There are many findings in psychology which seem puzzling from the standpoint of utility maximization. A favorite one is that humans are often willing to punish a wrong-doer at significant cost, even though the damage is already done. And in a "1-round game", this is indeed stupid. But this instinct evolved in a social animal that historically was in contact with the same individuals of its species for most of its life; punishing a wrong-doer after the fact still makes similar wrong-doing in the future less likely, and serves as a warning for those thinking about it.

It is likely that similar puzzling findings in psychology, which are puzzling when viewed in a one-round game, are actually utility-maximizing when considered in the context of longer time spans.

Hedonic Recursion as A Problem of Utility-Seeking

Summary:  We try to align our goals so they are consistent with each other but still often have contradictory ones, often owing to hard-wired aspects of biology.  Therefore, we continually try to adjust our preferences to be consistent (i.e. rational).  The ultimate end of rational goal-directed behavior then would be an ability to absolutely "edit" even our most profound preferences.  This may result in a system not conducive to continued survival; that is, if the source of our utility can be adjusted to itself increase utility, then our decisions lose all input from the world outside our nervous systems.  For this reason self-awareness may universally be an evolutionary dead-end.


A problem of agents with goals is that goals can be contradictory.  Goals are behavioral endpoints that maximize pleasure (utility) and minimize pain.  An evolved system like a human being should be expected to have many conflicting goals, and indeed we do, especially in areas where the goal's context has changed rapidly due to culture (e.g., "always consume high quality forage" vs. "stay slender to maintain health").  Of course this is only one of many examples.

Unfortunately, many goals are fairly hard-wired; no one has ever decided to be drawn to sugar and fat, and it's only with great difficulty and cleverness that a few of us in the developed world find hacks to prevent one of the competing goals from erasing gains from the others.  Furthermore it should be emphasized that we can't decide NOT to be drawn to sugar and fat.  We don't have the ability to modify ourselves to that degree; not yet, anyway.  (A hypothetical technology doing just this is discussed in more detail at Noahpinion.)

Would such an ability be a positive one?  It's anything but clear.  In this situation, we could make sugar and fat less appealing (or activity less painful), and/or make hunger pleasurable.  Less obvious but equally valid, you could program yourself to just stop worrying about heart disease and diabetes and gorge away.  To get at the core issue, what we're trying to do is maximize our utility by editing our goals - but there you see the circularity.  Achieving goals are what increase utility!  In some cases like food consumption, it's a relatively easy fix.  But let's consider mortality.  Assume that at some future time, you have the ability to edit our goals (our utility-providing endpoints) even at the most basic biological level.  However, also assume that there are still terminal diseases and finite lifespans; you have contracted one of these and are approaching the end of your own.  Unpleasant?  Of course not!  Make death something you look forward to!  Make your breakthrough cancer pain like an orgasm!  Why not?  Hurry up and let me suffer and die already why don't you - ahhhh, that's the stuff.  If you object "pleasure should only be a feedback mechanism and has to reflect reality", the completely-editable hedonist will bluntly ask you why, and probably take pity on you for not being able to get into your own source code.

In point of fact, why not make the most minute actions throughout life (or even absence thereof) result in untold pleasure?  And here is the problem.  Once we can manipulate our utility at such a profound level, we may be in dangerous territory.  Most of us can't control ourselves around potato chips.  It is likely that being given the key to our own reward centers this way would result not just in a realignment of utility with respect to our changed environment, but in giving ourselves easy utility-points for anything - and then what happens to motivation?  Why do anything, except give yourself more utility points?  (This is called hedonic recursion., and this is why it may be a good thing that for the most part, what we like and don't like is biologically tied down and non-negotiable.)  Even systems to assess the rationality of preferences (like the von Neumann-Morgenstern axioms) are only about internal consistency.  An individual that "solved" his own preferences in such a way that was consistent with death being desirable is not irrational.  No, he won't leave many offspring, but that wasn't what gave them utility after he got under his own hood and did extensive rework.

The ideal "logical" hedonic solution would seem to be giving ourselves pleasure in such a way that our likelihood to survive over time is increased, to increase total pleasure exposure.  (This satisfies our moral instinct that it's better to make yourself stop liking junk food than to stop worrying about dying.)  At first glance, one might argue that this is what evolution has done anyway, notwithstanding any mismatch effects from recent rapid cultural changes to our physical environment.  But evolution optimizes survival of genes over time, regardless of what conscious individual(s) they happen to be in at the moment, and pain and pleasure are always subservient to that.  Pain and fear just as often (perhaps at least as often) motivate survival behavior.  Here we're explicitly minimizing one in favor of the other.  To illustrate the difference in the two processes (and the recursion inherent in one):  evolution uses pleasure and pain to maximize gene numbers over time.  Pleasure uses pleasure to maximize pleasure.  One of these looks very much like a broken feedback loop.

Very speculatively, assuming that such goal plasticity is possible (perhaps after mind-uploading) it could be that hedonic recursion is not something unique to the way our nervous systems are put together, but rather a fundamental problem of any representational tissue that develops a binary feedback system.  So far, we've  been protected by the highly manufacturing-process-constrained substrate of our minds.  If so, over time such systems will increasingly learn how to please themselves and shut of pain, independent of what's going on in the outside world, to a point where they're so cut off from stimuli that causes them to survive less that they disappear.  Self-awareness always goes off the rails.  This is one possible solution to the Fermi paradox:  that pleasure inevitably becomes an end in itself, divorced from survival, and the experiencing agents always disappear.

Strongest Sapir-Whorf Evidence Yet: Economic Implications of Future Tense

Languages with obligatory future tense marking have higher future discount rates, across a host of behaviors. So far the paper seems solid and this observation is pretty amazing. In terms of obvious implications, this sure beats distance and color classification!

The obvious speculation is that future time marking increases the cognitive distance of future plans. It's immediately more interesting that many languages use future tense or future tense-like constructions for their subjunctive. If we put future discounting into procrastination equation terms, we could say that placing future acts into further away, less concrete categories is raising the delay, and/or lowering the expectancy.