Borrowing from Peter to Pay Paul: Atrocities, Guns, and the Misuse of Expected Utility Theory

“We often find, upon a thorough review, that our expedients, while they have for a time seemed to produce very valuable results, have in fact corrected one evil by creating or enhancing another. We have borrowed from Peter to pay Paul.” -Frederick Law Olmsted, A Journey through the Back Country

It is, I think, for the reason of epistemic modesty that even the most virtue-inclined of us quaver at the prospect of discussing the great atrocities of the past: slavery, genocide, mass murder. We want to believe that our modern values are correct, neither too cruel nor too enabling. But the best counterargument to our new and improved morality is a simple wave at the vast cohort of people throughout history who similarly believed themselves to be right while upholding principles we have since decided are unthinkable.

The conventional defenses of these grave affronts to human decency all ring the same. James Henry Hammond, governor of South Carolina during legalized slavery, claimed, “It is by the existence of slavery, exempting so large a portion of our citizens from labor, that we have leisure for intellectual pursuits.” Historian Tony Judt reports that in war zone surveys of German nationals during World War II, a majority of the population claimed that the killing of Jews and other non-Aryan peoples was “necessary for the security of Germans.”

The idea that citizen nationals turned a blind eye to the evils of slavery and genocide is a myth. Indeed, they recognized the shattering and sundry sufferings of their fellowmen as not only manifest, but instrumental to their own fulfillment. As Olmsted succinctly surmises, “We have borrowed from Peter to pay Paul.” In this case, the Pauls that the participants paid were themselves and their cultures, and the Peters were very unfortunate indeed.

So why, then, when I encountered the Olmsted quote above, did I think first not of atrocities but of the type of moral justification I hear daily? I and the people around me are aware of the great quantity of suffering in the world, and at least profess that we want to bring about better circumstances. In so doing, we claim that the reasons we’ve chosen to pursue a university education rather than join up with an international relief organization are ultimately altruistic. We are bettering ourselves, we declare, so that we can be of better help to others someday. But how much do we need to be “improved” before we are ready to go put such improvement to use towards the final end of reducing human suffering? We say of our meta-values that it is virtuous to pursue the kind of world-class education that will permit us to grow in virtue. But is there really nothing immoral about the kind of “virtue development” that tells us to spend four years enjoying the company of friends, languishing in well-furnished dormitories, gorging ourselves on ready-made food? We claim that the time we spend here gives us the direction we need to figure out in what way we can best help the world in the era of specialization. But how long and costly must a cost-benefit analysis be before just taking the plunge and guessing becomes a more effective mechanism?

In his famous essay “Singer’s Solution to World Poverty,” Peter Singer (with whom I generally don’t agree at all) argues, “The formula is simple: whatever money you’re spending on luxuries, not necessities, should be given away.” I’ll slightly reformulate Singer’s point in a way that makes more sense for a long-term value maximizer of the type I’ve described: “Whatever time and money you’re not spending toward your own necessities should be redirected toward altruistic ends.”

But the alleged long-game virtue-seeker worms his way out of this request using an application of expected-utility theory (EUT). “If I were to right away begin a life of service,” he responds, “I’d be losing career opportunities that would later allow me to cut a fatter check for charity.” He plugs in some arbitrary values for probability and reward and breathes a sigh of relief at the implication that he is, in fact, living virtuously. I’ve seen people go ridiculously far down this line of reasoning: “I’m morally obligated to attend a party with my colleagues tonight instead of volunteering at X shelter, because that way I marginally increase my chance of getting a promotion from the boss, and thus will receive more money for my later donation.” And depending on his Bayesian priors, he can prove this to be true. Sure, “self-centered” doesn’t necessarily mean bad, but these two scenarios lie on an increasing axis of puerility. Somewhere before either of them is the point where self-improvement turns into self-indulgence and altruism becomes a weapon against itself. Either that, or letting Econ majors and politicians use EUT for effective altruism just happens to create results that seem very convenient in light of their first-order desires.

We readily acknowledge the depravity of slavery and genocide, but we don’t often talk about why they are so depraved–because, as it turns out, the best defense of these unconscionable acts is also the best defense of the way many of us currently live. If we ignore for a moment the vast difference in moral charge between allowing the deaths of millions of people worldwide and literally killing them firsthand, then the perpetrators of atrocities meet our reasoning with uncanny accuracy, almost point for point. If they had declared that they wanted to grow in virtue for the particular end of being better toward the people whose well-being they were sacrificing to accomplish these ideas of virtue, then the situations would be isomorphic. But they didn’t. They instead espoused a perverted brand of preferencing, e.g. “Paul’s net worth is more important than Peter’s,” to justify their moral abuses. We, living in the era of equality, have rejected this notion and replaced it with the affirming market-based rhetoric of “If he takes enough of Peter’s money to begin with, then Paul can invest it through Bain and pay Peter back tenfold in two years!” And to me, this line of thought doesn’t seem all that much safer.

I’m not claiming working in finance to “save up” for charity donations is anywhere near as awful an act as genocide. The manner in which the two are comparable is purely at the meta-level. Using EUT-based morality calculations to justify our ceaseless borrowing is actively destructive to the fabric of ethics in a way that selective usage of the concepts of freedom and nationhood wasn’t. This conclusion stands regardless of how much more pain mass atrocities caused, or how much more directly its authors were involved in its causation. How poor must Peter be before we pay him back? It doesn’t matter anymore, amidst all the dollar signs that will pull him from destitution in two years…if he survives that long. Yes, we are not responsible for Peter’s death in the way that the perpetrators of genocide were. But their meta-principles, unlike ours, didn’t validate the ability of anyone, anywhere, to use the tools of morality against morality itself. In fact, the reason we’re aware of their moral fallacy at all is that at the meta-level, their principles were not even coherent. Our successors will have a much harder time disproving our meta-principles than we did theirs.

This is all unnecessary, because one really doesn’t need to consider the two evils as comparable in order to want to avoid both. The old “killing versus letting die” distinction is toothless here. Singer is aware of the salience of ignoring moral qualms in which we are not directly involved. “To be able to consign a child to death when he is standing right in front of you takes a chilling kind of heartlessness; it is much easier to ignore an appeal for money to help children you will never meet,” he admits. But the metric of difficulty should not be confused with the metric of virtue. We do plenty of good without even thinking about it (and if you can’t recall an example, that’s equally likely an indicator that you never perform unconscious virtuous acts as it is that you often do and don’t even consider them consequential enough to remember). Anyways, even the strong version of this argument–that killing someone is, de jure, worse than letting her die–does not mean that we should a priori take to be supererogatory any moral actions toward remedying situations which we did not cause. How virtuous should we be expected to be? My answer: virtuous. There’s no spectrum. I’ve long believed that “good” and “bad” are measured in degrees, whereas “virtuous” and “sinful” are binary. A friend asked me today, “If you’re off the mark, doesn’t it matter by how much?” I answered, “Yes, but first you check whether you hit the target, which is a Yes/No question, and then you measure the distance in terms of inches or feet.” Similarly, consider two pregnant women; one is due in 3 months, the other in 6. Would you regard one as “more pregnant” than the other? No, that’s ridiculous–“pregnant” is a binary variable, and “time until due date” is a continuous, distinct variable. Just so for morality: we should turn the “virtue” switch on, and strive to maximize the continuous variable of “good.”

We could perhaps take this blatant misuse of EUT more charitably and claim that the Paul paid in the exchange is not ourselves, sheepishly holding up an IOU, but future recipients of aid, endowed with all the good we can instantiate with this carefully augmented nest egg. In such a calculus, we’re foregoing present good for the sole purpose of maximizing future good, and we do not enter into the equation as agents. But then isn’t it a failure of epistemic modesty for us to assume we can invest these “Schrodinger’s utils” more reliably to produce a long-term sum than can the organizations and people we’d be helping if we forked it over now? It’s at least paternalistic, which is usually a warning sign. And in the case of those economic geniuses who are truly best suited to monitor the growth of the utils, I find it hard to believe that they’re doing so purely to provide invaluable, unpaid financial management to the United Way. Even if they were, how is it not completely self-defeating to take from the present to give to the future–when those from whom we are withholding real-time aid would perhaps otherwise have become the parents, educators, missionaries, and relief advocates for the agents of the future?

It doesn’t matter if killing is different from letting die, because this kind of reasoning, as it turns out, causes both. A few days ago, my nation was hit by a devastating act of violence, only the most recent in a series of cases that have seemed to demonstrate an upward trend in mortality. On the afternoon of Ash Wednesday 2018, a former student opened fire in Marjory Stoneman Douglas High School, killing 17 students and teachers at the scene and injuring 15 more.

The Wikipedia articles for school shootings in all other world countries are consolidated into a single page. But America has perpetuated and suffered enough violence in this avenue to constitute its own page, as long as the rest combined. Yet lawmakers who refuse to clamp down on gun sales attribute their reticence to precisely the manner of thinking we’ve just explored.

Yes. Proponents of easy market access to automatic weapons use exactly the method of poorly-applied EUT to justify their farcically circular reasoning. They are, quite straightforwardly, robbing innocent lives to grant what they perceive as a public right that will then theoretically be used to prevent further loss of life. After the First Baptist Church shooting, two civilians grabbed their own rifles and pursued the shooter until he crashed his car, a heroic action that launched them into the national spotlight. One of these vigilantes, Texan Johnnie Landendorff, received overwhelming praise from far-right media such as Townhall and Breitbart. Said one Townhall staff member, “In the hands of a ‘good guy,’ a gun is what finally put an end to the massacre.”

Any good NRA hack will tell you that the best defense against a mass shooting is an armed, vigilant citizen. Sure, he argues, it is a drain on the aggregate value of society for some dangerous individuals to have access to guns, but this is a necessary price in order to allow the righteous citizens of America to purchase the guns they’ll use productively–to grow in virtue to the point where they can overcome such criminals. The logic of needing any guns in the first place if their telos is to resolve violence caused by people wielding other guns is blatantly self-defeating, like all instantiations of classic misuse of EUT. But no matter! We must pay Paul! It’s just too bad if we are both Peter and Paul, and we never break even.

Now you see the stark line between atrocity and innocent misuse of expected-value theory is not so stark as we thought. This Peter/Paul reasoning can feed the continuity of tragic and evil acts. It is as dangerous as they.

Given that even statistically-minded me has a penchant for permitting gun ownership, I’ll give any pro-gun readers some epistemic credit: there are better reasons not to want stringent gun control than this paradoxical argument. Using it is an insult to our intelligence and our values. If you are pro-gun, then other factors are entering into your reasoning than absolute aggregate utility–factors like freedom as a first principle, or concerns about effectiveness of the governmental measures themselves, or a non actor-network theoretic understanding of the relationship between weapons and people. For reference, nobody who believes in the legalization of heroin explains their position with “If everyone has access to heroin, we can use our experience with heroin to grow in virtue to better help addicts.” If you strongly believe that gun control legislation is wrong, then you can make a better case by avoiding such pathetically post facto justification.

The point is, there’s clearly something absurdly wrong with using EUT for post hoc justification of selfishness. This problem stems partly from the fact that it doesn’t make any sense to consider every present action we take through the lens of expected-utility theory. I posted earlier about the paralysis this approach causes. But I espoused a principle in that article that needs clarification in order to be consistent here: that we should care just as much, in certain situations, about future actors as we do about present ones. What does this mean, when the starving children represented by the $1K you didn’t pay Peter ten years ago are now dead because of your inaction, and any reparations you can make by paying forward your $10K to Peter’s sons will go to completely different people? What does it mean when the healthy children at Stoneman Douglas are now dead, rendering worthless the glorious bullets that would have been used to save them when the Civilian Hero arrived on the scene?

We should preference existent actors over nonexistent actors in cases where the existent actors are in clear and present danger, especially if helping them may help to avert the need by future actors of help.

To do this rigorously, we could use multivariate EUT in which probability/value pairs are Borel products with different distributions that are time-sensitive to earlier selections. Karni (1989) has already done good work on EUT over multivariate probability distributions, which could be a place to start.

In less mathematical terms, we could think of this as a binary indicator. Is there danger for existent actors that is greater than some small epsilon? Probably. There likely will be for a long time–or at least until we stop using backwards EUT to implode morality. If there is such a danger, then act now to stop it instead of saving up to help people who will be born into suffering in the interval in which your bond was doubling–people who might’ve been born into comfort if you’d helped someone else. (Aside: “those people might not have existed, exactly” isn’t a counterargument worth rebutting here; helping earlier agents may linearly help later agents that don’t yet exist, regardless of the counterfactual details.)

This comment brings me to an important corollary. I don’t believe my metric applies to work that is done to overturn a long-term political principle that is contrary to virtue, or to instantiate one that is in accordance with virtue. In these cases, the future population in danger will likely look the same regardless of the individuals helped, and working to stop the cause of the danger rather than its effect may be more potent. If your current actions are directly and non-hypothetically correlated with a virtuous end goal that you can’t yet enact, perfect! Tell the truth when asked: that instead of joining Teach for America, you’re pursuing your law degree or self-studying machine learning in order to illegalize the death penalty or better implement AI alignment strategies, respectively. The moral threats you have diagnosed as salient–for you just can’t find everything salient–are best helped by what you are actually doing. This argument, I think, only works for systematic risks, in which one can have a good idea of what the climate of his concern of interest will look like moving forward. If you’re this kind of person, you’re probably already donating or tithing to the cause of current moral actors, so continue to do that in tandem with your altruistic project. One such altruistic project might constitute bipartisan endeavors to create a better understanding of what “guns” are and should be, who should have access to them, and what they should be used for.

So, reader, even if you’re just not going to donate to charity or volunteer for a helpful cause no matter what argument I present, stop using EUT to justify that impulse. Regardless of what Singer says, it would be unreasonable to expect you to donate all of your resources to relief, and the idea that this is the “only” moral way to live is a contributing factor in the emergence of post facto bad use of EUT. Perhaps going out for a nice dinner to clear your head will meaningfully improve the good you can do in the coming week, but this conjectural claim isn’t the way to frame your choice if someone angrily asks why you’d do such a thing. Neither is the default response that you are entitled to such days off, which harkens ominously back to the infamous atrocities of history–whose instigators claimed justification through the preferencing of their own abstruse interests over other moral actors’ very basic rights. You are not entitled to a day off from morality any more than you are entitled to a gun.

This shouldn’t be taken as a claim that there is only one model for a virtuous life, a model that always puts others before the self. When Luther wrote “On Temporal Authority,” he argued that God would not begrudge a thoughtful and principled monarch the occasional joust or hunt, but a monarch who strove to be thoughtful and principled would find little, if any, time in which to pursue such pastimes. As humans, we are constantly sinning in small ways by preferencing our immediate interests over more important concerns. The fact that it is to some degree inevitable doesn’t make it any better–but honesty does. No, it’s not moral to value your $200 fuzzy parka over the life of the child you could have saved with the money you used to buy it. In fact, it’s probably immoral. But it’s not anti-moral to admit that uncomfortable truth. It is anti-moral to say you bought the parka to keep yourself dry so that when you one day climb a mountain to save a stranded Himalayan child, you’ll have slightly less slippery skin. And it is deeply anti-moral to claim that the reason you own a gun is to someday be the hero who steps in to protect somebody from a villain wielding a gun just like your own.

If I have convinced you, call (800) 367-5437 now to donate to UNICEF. Go to http://www.gofundme.com/stonemandouglasvictimsfund to support survivors of the Stoneman Douglas tragedy. All you need is a credit card and a sense of doubt.



The Libertarian Theodicy

There is an eminent tautology that emerges when we consider where federal power ends. All powers that are not claimed by the national government are left to and reserved for the subsidiary bodies of government, especially the states. The question of where, exactly, federal power ends has been one of much controversiality in American history, from the Virginia and Kentucky Resolutions to the Civil War to the Seventeenth Amendment to the civil rights movement.

Tremendous swaths of people have fought bitterly for legal decisions to be made closer to home, whether home was Texas or Massachusetts. Indeed, it’s a common misconception that states’ rights were the Southern battle cry during the Civil War. The Northerners used those arguments too–especially in their rally against the Dred Scott decision and Fugitive Slave Act, which they saw as federal infringements upon their states’ illegalization of slavery. The claim “on our land, you follow our rules” has further manifested in recent times, surrounding the Obergefell decision. Even today, the legality of state nullification and secession is an open question.

States’ rights’ advocates will tell you that the argument against these extensions of federal power is orthogonal to any metric of virtue. Small-government enthusiasts claim not to oppose the federal enfranchisement of minorities or gay people out of racism or fear or even the belief that these laws are wrong. They won’t defend their desires to have the matters relegated to state courts for any ideologically-based reason. Rather, they’ll tell you they fight such measures because of their nefarious sweeping scope. Libertarians detest the notion that the national government is settling what they see as intra-state affairs on states’ behalf.

1865 libertarians wanted the issue of slavery left to the states. 1965 libertarians wanted the same for civil rights. Abortion, capital punishment, gay marriage, marijuana? Advocates and detractors alike of the individual issues have argued for the legality of these actions and products to be determined on a smaller scale.

In a world where righteous indignation has always been the motivator of activism, it seems remarkable that so many have crusaded for the right to be wrong–not, in fact, believing themselves to be correct, but rather seeking the opportunity to determine their own stance; to have moral choices not prescribed unto them.

Why fight for states’ right to decide, even if that means some states won’t legislate the way I want them to? At first blush, the problem looks isomorphic to that of non-natural theodicy, the consideration of why men commit acts of evil in the world. The canonical answer is that free will and perfect goodness are mutually exclusive. God had to pick one.

While the natural world is an object-level entity, the realm of the divine ought be considered as the meta-level whose principles manifest in the world that we see. In the eyes of the divine, when is free will better than goodness? Clearly, if there is either something good in free will exercised in and of itself, or something bad in permanent and immutable goodness. Object-level goodness is not the same as meta-level goodness; choosing the option that maximizes the amount of good is the correct solution, and free will both allows individual choice (meta-level good) and results in some people who perform acts of virtue (further instantiation of object-level good). Instilling perfect goodness allows only the latter. It is better for Man and for God’s glory that humans have ownership and possession of the acts they perform.

Now is the part where I show my cards and say that there’s an enormous logical fallacy taking place in the application of human theodicy to government. The two are not isomorphic, because states’ rights constitute a case in which ownership can differ greatly from possession, involvement, and agency, all of which need to be present to justify free will.

Can a Texan claim that he has more ownership of local politics than of federal–that a court decision made locally is more his than one made by SCOTUS? Probably. He has a small chance of sitting on a Texas jury, and none of sitting on a federal one. He has some tiny margin of influence on ballot propositions in his home state, again narrowly beating out his influence vector for federal propositions. Any positive epsilon is greater than zero; given infinite time, our Texan will, sooner or later, find himself the determining vote in an issue important to him. So his state’s decisions belong to him more viscerally. So far, so good.

Is his home state more likely to side with him in terms of deciding legal matters? Definitely. Statistically, the population of Texas is likely to look more like him than the voter base of the nation at large, as there is some self-segregation due to shared state values. His ten neighbors are more likely to agree with him politically than ten people selected at random from the United States voter base, by sheer dint of the fact that they chose to live near him, which they probably wouldn’t have done if his politics were anathema to them. This also makes his state’s decisions more his, because they are more likely to resemble his own decisions.

But does the Texan have more involvement in his state’s decisions than in national ones–that is, more of a say? Not really. It is more likely that the outcome will turn out in a way he’d find agreeable, but his participation had near nothing to do with it. In terms of issues, he has narrowly more of a say in state matters, but not so in elections. In Boolean terms, the influence of individual voters within a state (especially a small one) is just about as negligible as that of voters in a federal election.

Does the Texan have more agency in his state than in the nation? No–that one’s just ridiculous, because agency is concerned more with actions than results. Not that the results of his political activities are really that different in the discrete spheres. He can write an angry letter to his Congressman, to his governor, or to the President. None of those people is significantly more likely to read the letter than the others. (One of his congressman’s staffers will probably read it, but won’t be able to do anything about it. So in each case, he took the same actions and got the same null results.)

So the Texan has more ownership of his state’s politics, but no more involvement or agency in their determination. The free will argument doesn’t hold up, because even if God predetermines our actions, they still belong to us, and the culpability is ours. Calvin, the father of determinist theology, argued in Institutes of the Christian Religion that in the predeterminist worldview, Man retains full ownership of his actions. For Man to have free will, he needs something greater than ownership–involvement and agency, which the Texan gets no more out of Texas than out of the U.S.A.

Thus I conclude that he wants the issue at hand to be left to the states not out of a want of self-sovereignty but because he concludes, not incorrectly, that the population of his state is more likely to vote his way. That seems pretty likely when you realize that none of the approximately three Green Party members in Texas wants the issue of abortion determined closer to home.

Is there a saving grace for the states’ rights argument? Yes! The empirically validated belief that communities themselves make the rules that work best for them. But this is clearly an argument from epistemic modesty, rather than from the lofty rhetoric of freedom as its own good. Groups of people should be self-governing because they are more likely to do a good job than their detached overseers, not because their freedom is viscerally important in the abstract.

This understanding, radically different in its first principles, doesn’t change political results all that much. It does, however, alter the approach with which we view the negative externalities I mentioned above: that some states won’t legislate the way want them to. For issues that are one-size-fits-all, we no longer need gaze upon our fellowmen in different territories with sympathy at the unfortunate political results their freedom has prompted. We might instead assume their model works better for them, and wonder whether it would for us, too.

Error, Noise, and Data: The Allegory of the Lightning Strike

The Allegory

Adam and Eve lived with their children Cain and Abel in an otherwise empty world. Cain and Abel did not go hungry, for their parents farmed the land; they were not bored, for they had each other for amusement. But they were uncertain, for, having been expelled from God’s presence, they had no heavenly voice to tell them what things they did were good or bad.

“Hey, Cain!” Abel cried. “I want to roll this marble, but I fear that if I do so, it will roll under the bed, and Mom will be annoyed.”

Cain responded, “I wonder if God could tell us whether you should roll the marble or not. Maybe he’d send a sign–a lightning strike, or a sudden gust of wind.”

“Good point.” Abel took a deep breath. Then he bellowed, “Heavenly Father! If the marble will roll under the bed, then send a lightning strike!”

No lightning strike emerged from the cerulean blue sky. It was a pleasant spring midday, after all. So Abel went ahead and rolled the marble, and it rolled under the bed.

“Huh,” said Cain. “We know God exists, is benevolent, and is listening. But he did not warn us that the marble would roll under the bed. Therefore, I conclude that the consequences of the marble’s rolling weren’t important enough to get God’s attention.”

It became something of a game between the brothers: whenever a small decision weighed on their minds, they’d ask God to send a lightning strike to elucidate their thinking. No such strikes occurred, and so they went ahead with their actions, and got the expected outcomes most of the time.

As they grew older, Cain and Abel found themselves facing more challenging and important decisions. What should they get Mother and Father for their joint birthday celebration? How much corn should they plant this year? Which of the suddenly emergent womenfolk should they marry? And it was with regards to this last question that a strange event proceeded to occur.

“Heavenly Father, send a lightning strike if you wish me to marry Wilhemina!” cried Cain, and to his immense surprise, lightning came screaming from the blue skies.

“Wow!” Abel yelled. “Heavenly Father, send me a lightning strike if you wish me to marry Philomena!” No lightning. Cain married, and Abel remained a bachelor.

Later that year, they were playing with marbles when Abel wondered if he might roll a marble under the bed. “Heavenly Father,” he cried, “send down lightning if this marble will roll under the bed!”


“Wait, Abel,” cautioned Cain as Abel prepared to put his marble away. “God only responds to important questions, remember? This lightning was just a fluke.”

The Interpretation

And so Abel rolled the marble. The story ends there, and it doesn’t matter whether the marble rolled under the bed. Cain and Abel grew so used to the absence of lightning caterwauling down in their (comparably many) small decisions that they felt that it should only be used as a value metric in their (comparably few) large decisions. Rendering this in probabilistic terms, we can establish the following. First, lightning is very unlikely to occur from clear skies; secondly, if it does occur, it’s more likely that God sent it than that it was random.

  1. P(no lightning strike) >> P(lightning strike)
  2. P(God sent it|lightning strike) >> P(It was random|lightning strike)
  3. P(God sends lightning strike|important question) > P(God sends lightning strike|unimportant question)

Cain and Abel assumed this third, non-verifiable relationship at the get-go. Ultimately, out of the myriad “unimportant” questions they asked God, only one was ever answered by a lightning strike (Abel’s last marble roll). Let’s say Cain and Abel asked 1,000 unimportant questions. Out of the maybe ten “important” questions they asked God, one was also answered by a lightning strike. It’s obvious that 10% > 0.1%, so the brothers’ initial assumption that God was considerably more likely to send a lightning strike in response to an important question seems empirically verified. Nonetheless, in concluding that the final lightning strike was accidental, they make a logical error.

Cain and Abel’s assumption that the lack of lightning strikes about unimportant issues means that the ultimate lightning strike was a fluke is unfounded. It’s a reductio argument: if God didn’t discriminate between important and unimportant questions, and if one lightning strike occurred in each, then the respective presence or absence of lightning strikes in each category up to that point is incredibly unlikely.

For instance, assuming that God does not discriminate in response between important and unimportant queries, then if we use the unimportant question set to gain an approximation of the likelihood of question response, the odds that one of the first ten important questions would be answered by a lightning strike (given that the approximate odds of response are roughly one in a thousand) are approximately 1%. This does not, however, mean that the lightning strike was a fluke; it merely means that we are faced with two unlikely-looking explanations, one of which has to be true:

  1. God responded to a question within the first ten, which is very unusual.
  2. The lightning strike was random.

How likely is a random lightning strike? If P(It was random|lightning strike) < 1%, then Option 1 is favorable.

The margin at which we decide it is no longer more likely that God engineered the lightning strike varies based on which set we use as the probability determiner. If we instead use the important-question set (which makes sense, because a lightning strike answered a question out of that set first), then we can assume that the odds are about 1/10, so assuming God is equally willing to answer unimportant questions, the probability of his not doing so for 1000 questions is a whopping 2E-44%. But the operative question, still, is not How unlikely is that? but Is that less likely than the alternative (a random lightning strike occurrence)?

Importantly, since both of the possible implications of a lightning strike (randomness or divine intervention’s occurrence seeming very non-Bayesian) are fairly unlikely in this framework, the situation itself–lightning strike occurring once in the large data set and once in the small–is highly improbable. Given that it has obtained, though, the more likely explanation is probably the correct one.


Something about these calculations might feel off to you, and if you think that, you’re right. The problem with the above reasoning is that when considering the overall probabilities of God-sent lightning strikes via the totality of data, I didn’t weigh in the possibility that one of the two lightning strikes Cain and Abel experienced might have been a random fluke.

That probability is small, but given that we have no way of knowing after the fact whether any individual lightning strike was God-sent or random (only that both have a greater-than-zero probability of occurring), it must be considered.

Random lightning strikes can here be considered a stand-in for noise in data sets in cases where that noise could significantly confound our prior probabilities. We could liken this to an arbitrarily long Boolean string in which a value of 0 in the kth place in the original string is highly unlikely, and the string then undergoes some fuzziness after which the k-value is, with some degree of likelihood, altered. The question that this poses to statisticians, which is more intuitively apparent in the lightning strike example, is the following:

If we attain our prior probabilities through an empirical analysis of the data that we have, knowing that noise can be a factor, and then find that our distributions are odd, in what cases does this make it likely that random noise altered some of the strings in the original set?

If noise is known to be less or more probable than the result we are looking for, we would conclude accordingly; the problem emerges in cases where we cannot see the original data set, and thus have no way of knowing what our priors should be that noise occurred. In the lightning strike allegory, there is a solution to this problem, that perhaps could be generalized.

The Solution

Cain and Abel should repeat their experiment arbitrarily many times–let’s say, until they have achieved sufficiently many lightning strikes that the number of strikes is suitable for data analysis. Perhaps after one hundred years of asking questions, Cain and Abel receive one hundred lightning strikes. In this case, there is an obvious mechanism through which the brothers can discern whether lightning strikes are God-sent: whether the lightning strikes make correct truth claims. (E.g. if Abel asked for a lightning strike if the marble would roll under the bed, a lightning strike occurred, and the marble did not roll under the bed, then that lightning strike was not divine in origin.) The following claims seem to be reasonable:

  1. P(lightning strike is accurate|lightning strike was God-sent) = 1
  2. P(lightning strike is accurate|lightning strike was random) = appx. .5, barring cases in which one of the outcomes was very unlikely, in which case the brothers wouldn’t have needed divine advice anyway).

Now we have a set of probabilities through which we can come to ascertain the approximate proportion of random lightning strikes to divine lightning strikes.

The same can be said of noise in data sets. For cryptography, there is an obvious, immediate parallel to the case I describe: does the resultant string make sense? If not, then it is much more likely to have been noise-altered. By ascribing sufficiently accurate prior probabilities to these cases, we can discern the relative likelihood of noise to that of the genuine obtainment of unlikely outcomes, and then can proceed with the analysis described in “Interpretation.”




Why Fermi’s paradox is neither

I will attempt here to demonstrate why the so-called Fermi paradox is not a paradox. The blog post title is a little flippant (the idea isn’t really Fermi’s; Tsiolkovsky was the first known asker, but that’s neither here nor there).

The idea of the Fermi paradox is as follows:

  1. There is a high probability that there are many (perhaps infinitely many) extraterrestrial civilizations.
  2. Given 1, there is a high probability that many extraterrestrial civilizations have achieved space colonization, by or before 10^8 years ago.
  3. A generous time estimate for the process of traversing nearby galaxies is 10^8 years.
  4. If extraterrestrial civilizations had colonized space, we would know it.
  5. So where are they? Why is there no evidence that any extraterrestrial civilizations have done so, in the form of their observable presence?

There are a lot of realms for fallacy here. 4 comes out of nowhere, and isn’t necessarily true–discarding the conclusion doesn’t necessarily require one to disagree with Premise 1.

But we will here focus on the derivation of 2 from 1. To explain this, I’ll posit a thought experiment. Imagine the following:

A function will be held that you have vague reason to believe is very important or interesting–suffice it to say, you’re relatively sure it’s a good idea for you to attend, although it isn’t mandatory. You don’t know much about what the event will be. Entirely through your own logic, you’ve reasoned that a famous person is likely to stop by, or a new technology will be unveiled there, or you’ll get to witness and participate in an important political interaction unfold firsthand, an anecdote that will later be written about in many newspapers. Or maybe you just think it’ll be a great deal of fun.

It is not easy for you to go to this event. It’s pricey and located in a different country. Also, you know the day on which the function will occur, but not the time–it’s possible you could arrive very early or very late, although the event is likely to continue late into the night. You’ve put the necessary steps in place to cover the costs of attendance and transportation. The function will occur on a high floor of a skyscraper, a sought-after event venue that will be entirely booked for the occasion. This is problematic for you, because you have gripping claustrophobia, and do not use elevators. Thus, you’ll have to climb up many flights of stairs. You also have weak knees, so this process will take considerable time and effort.

You can’t see the top floor of the skyscraper from the ground, but you have good reason to believe it is well-lit and windowed. When you get there, you will be able to see everyone who is in the room with you, and some of the people who are on the higher floors of other nearby hotels.

After landing in the city, you rent a car and drive toward the event. As you pass through a tollbooth, you tell the operator that you are attending the event, and he argues with you about why you shouldn’t, saying that you’re going far out of your way for a development that will inevitably cause bad consequences for your life. The interaction takes fifteen minutes. Finally, you get to the venue, and while standing in the parking garage, you realize something odd: no other cars or people are in the garage.

You have not communicated with other attendees in advance of this event and have no idea who’s coming and if they’re driving themselves. Maybe others are using taxis, or bicycles, or trains, or subways. The only evidence you have that if there are people, then there should be cars is that you drove yourself.

You start climbing up the many stairs, and you see no one. As you approach this event–to which you have not yet arrived–and see not a single fellow person, is it rational to wonder, “Where is everybody?” Furthermore, is it rational to conclude that because you haven’t seen any people or cars, no one else is going to show up?

Of course not. Anyone who didn’t drive himself to the event wouldn’t have needed to stop by the garage, and certainly few, if any, people are taking the stairs. You know it’s possible to likely that you’re very early or late, which would explain not seeing people as you move from the ground floor to the top floor. As you noted, the occupants of the top floor can only be seen from the top floor itself.

This experiment is an allegory for the human quest to expand indefinitely. We have vague reasons to justify this desire, just as you have some idea of why it will be a good use of your time to attend this event, but our justifications are more innate than rational. It’s greatly difficult to accomplish our task: technological progress itself is difficult and costly, and we put up with, at every turn, our own or others’ desire to just let things be. Similarly, you had to deal with paying, arranging your travel, the rhetoric of the annoying tollbooth operator, and your claustrophobia and weak knees. Furthermore, there’s an insufficient knowledge problem: we have no idea what this end looks like if properly achieved. Just so, you can’t see the top floor, and you don’t know when the event starts. The idea that you can’t see the occupants of the top floor means that perhaps the most sustainable form of expansion is not observable, e.g. living digitally.

There are of course meaningful differences between the two situations, and I’ll handle them in turn. First, you have fair reason to believe that your being early or late or in the stairwell is a good reason for not seeing anyone even if there are plenty of other people, which probably makes it too easy for you to self-edit your own observer-selection bias. It’s reasonable to expect the other attendees not to have claustrophobia. So let’s change the situation so that you don’t, and you can take the elevator. Let’s say, too, that you have high prior probabilities that the other attendees–if they exist–don’t know when the event starts either. You arrive at the venue as before, and you get in the elevator, which is empty. As you pass each floor, you listen for the sound of conversation, or the sound of the button as people just arriving call for the elevator. You hear nothing, nothing, nothing. You’re now passing floor 4, and you haven’t seen or heard anyone.  Is it now rational to think no one else is coming?

No, although it might be more so than it was in the earlier example. The fact that the elevator was empty when you got there is of note, but that fact also is the reason that you see nobody now, because you’re still in an elevator that will remain empty until you get to your destination unless some of the attendees can walk through walls. Not hearing chatter isn’t necessarily a good reason to believe the venue is empty; for all you know, the event might be a silent movie screening. While certain occurrences (the silence, the emptiness) may be evidence that no one else is coming, they might just as well be indicators of the nature of the event itself.

If you get to the top floor and find it empty, then of course nobody was there, and the alternate explanations of not seeing or hearing anyone until then were useful rather than misleading. But let’s say you get to the top floor and find it to be a rather uninteresting silent movie screening. Then the real question that falls out of my allegory, and the question most important to the deconstruction of the Fermi paradox, is this one: given that you didn’t know pretty much anything about this event, why did you want to go in the first place–and why would anyone else?

I’ll modify my thought experiment to address this question. You have survived a global catastrophe, and to the best of your knowledge, you are the only survivor. You’re right by the mountain K2, to the top of which you can climb to better survey the surrounding area. For the sake of the metaphor, let’s say the physical attributes of K2 and its air pressure are such that if you yell from its peak, you can be heard for hundreds of miles. You climb the mountain so that you can be heard, although it takes you considerable time, and along the way you see nobody, and when you get there you see nobody. The whole time, your ears are primed for the noise of someone yelling from the mountain, and you hear nothing. Is everyone, then, necessarily dead?

No! First off, you can’t see from that high up if anyone is scavenging through the remnants of the destroyed buildings, and you can’t see anyone who’s indoors (which, I suspect, would be the premier place of refuge for intelligent apocalypse survivors, although that may not be what the movies would have us believe). So the only evidence you’ve gathered is that nobody who’s loudly careening across the plains, making their presence obvious to the AI and predators, and their skin primed for infection by the zombies and supervirus, has survived. Congratulations! You’ve rederived evolution.

“But wait!” you cry. “If other survivors exist, then they want to see if others are alive, because they want to find food, protection, reproductive partners, and company! So they too should’ve climbed to the top of this mountain to survey the area, and I should hear them yelling, or if they’re still climbing, I should see them on the mountain face now!”

It is this facet of Fermi-paradox reasoning against which I take up arms. From first principles like “food,” “protection,” “company,” “reproduction,” and “consolidation and expansion into a greater civilization,” we do not get to the obvious outcome of “climb to the top of K2.” It’s possible other survivors don’t know this unique sound capacity of K2. Or it’s possible they just don’t want to waste their existent resources on this wild goose chase. When you plugged your desires into your inner calculator, you decided the rational way to maximize utility was climbing K2, but I think that was dumb. That should be concerning to collectivists, because you and I are both people; we share almost all our DNA sequences.

Okay, sure, maybe nobody in that situation would actually climb the mountain, so I’ll provide an alternate anecdotal claim: If two friends lose each other in a crowd, they often don’t start looking for each other in the same place, even though their ends (in this case, “find my friend”) are exactly the same. And while greater discourse and broadening the horizons of communication would seem to have restricted the space of obvious choices, we find that the opposite is true: intra-American disparity in political beliefs is higher now than it’s ever been in a single population. It seems to be the case that with a more universal notion of what our ends are comes a less universal notion of how to achieve them. If you and I can’t agree on a first-order action item that best instantiates our values, how can we expect biological entities that look nothing like us to agree with the outcomes we societally derive–if they even have the same values in the first place?

With that said, this is the problem with the Fermi paradox: It is reasonable to expect extraterrestrial biological agents to have an interest in survival, expansion, and self-improvement. It is unreasonable to expect them to manifest those interests in the way we do.

We have good reason to believe that extraterrestrial life would have also developed from evolutionary processes; therefore, extraterrestrial life forms are probably invested in pursuing, avoiding, or maintaining things like heat, pressure, certain molecular compounds, food, shelter, and so forth. From here, we don’t right away get to “expand”–this doesn’t fall straight out of “reproduce.” Yes, small population size makes a species more prone to extinction, but there is an optimal population size, and it’s less than infinity. Unbounded growth tends to destroy environments necessary to the very survival the pursuance of which would motivate a species to expand. Yes, one solution to this is “consume resources further away in the universe,” but it isn’t the obvious solution, just like climbing K2 wasn’t the obvious solution above. At any rate, extraterrestrial intelligent beings may not have arisen out of the kind of carbon-hydrogen-oxygen-and-nitrogen-based cellular “life” with which we are familiar. But even if we assumed that in the cosmos some (lesser) number of biological populations have this generalized “expand” principle, that gives us zero grounds to claim that even one other group would seek to instantiate it in the same way we are now considering.

Objection: “But Tessa, if there are infinitely many extraterrestrial civilizations, then there are infinitely many that are exactly like us, and therefore would colonize space! And some of those must have occurred by however many years ago it would’ve taken for them to now be visible!”

Response: Okay, okay. My diatribe above is only true in a finite universe. Framing the paradox in an infinite universe means that there necessarily will be civilizations interested in space colonization, but it also means that they are overwhelmingly likely to be significantly temporally or spatially displaced from us. If they lie outside our observable light cone, we’ll never interact with them. Period. And the amount of “stuff” in our observable light cone shrinks tremendously every nanosecond.

In general, I fear people have a difficult time rendering distinct the concepts of “very large” and “infinite.” That’s dangerous, because very large things and infinite things work quite differently, as is apparent to anyone who’s ever studied Knuth up-arrow notation or ordinal numbers. If you put an infinite number of monkeys in a room with infinite typewriters for infinite time, then one will eventually type War and Peace. This is true of even one monkey and one typewriter, or even of a typewriter by itself, given slight variations in atmospheric pressure. The point is that the operative factor in the saying is infinite time. The saying isn’t true of a billion monkeys with a billion typewriters for a billion years, especially if the prior probability of typing War and Peace is lower than one in a trillion per year. The concept of “everything in the probability space happens given infinite time” is useless in most situations, because the odds of a monkey typing War and Peace, or a black hole emitting a complete and functioning human brain (cf. Bostrom, who discusses this in Anthropic Bias in a different context), are so low that the probability of them happening within some time or space bound within which we are operating is functionally zero.

Objection: “But even if we don’t have access to infinitely many alien species, there must still be a large enough number of them accessible to us that at least one would have achieved observable space colonization! The priors can’t be that low!”

Response: Actually, they can. In a finite universe, it is very possible that the prior probability that “some other species X will have the desire to colonize space and the means to instantiate it” is low enough that this won’t happen in our lifetime even if there are many other intelligent beings in our observable light cone.

Objection: “But Tessa, intelligent species have a tendency to become outward-looking!”

Response: Says who? We have one reference point. And even if you’re willing to extrapolate wildly from this one anecdotal case, our particular anthropomorphic brand of intelligence can at best maybe be called “outward-looking.” Do we think about space colonization? Sure. Do we wish the good of the other for the sake of the other? Eh, perhaps. But we have spent eons trying to characterize the “human condition,” trying to catalogue the mechanisms of the human body, roiling at the mercy of nationalist and isolationist ideologies. Anyone who thinks humans are uncontroversially outward-looking should crack a book, because it’s all there writ large across the pages of history: large groups of humans are the most myopic structure known to the Earth.

And even if we arbitrarily draw a line through this single and controversial reference point, other civilizations could cognize “space exploration” like “climbing to the top of K2 in a post-apocalyptic wasteland”–technically in accordance with their ends, but not something on which they’d waste breath.

Thus there is a solution to the Fermi paradox that isn’t either the depressing option that we are alone in the universe, or the depressing option that space colonization is impossible for some Great-Filter-related reason (e.g. intelligent life self-destructs, periodic natural extinctions). And if I’m all wrong, and some other civilization out there is musing over their version of the Fermi paradox, then let’s take to the stars so we can find it! I’d posit that it constituents would be happy to meet us: any civilization with a notion of loneliness likely has a notion of hospitality. That’s why I’m excited about the implications of my argument. It’s likely that other civilizations don’t look enough like us to have “space colonization” as a proposed project, but if they do, perhaps they share other ends with us and aren’t just searching for resources. If we ever find ourselves at the mercy of a Type IV civilization that arrives from the skies, I think we have good reason to believe it will be friendly–because it will likely resemble us, in the important ways at least.


(P.S. Regarding my last sentence: yes, given human militaristic history, perhaps “friendly” and “resembles us” are not codependent concepts. If there’s interest, I’ll write about this topic as it relates to space colonization later.)

Bostrom’s Paperclip Problem: Certainty and Artificial Intelligence

If the AI is a sensible Bayesian agent, it would never assign exactly zero probability to the hypothesis that it has not yet achieved its goal.” -Bostrom, Superintelligence: Paths, Dangers, Strategies

Decision theorist and philosopher Nick Bostrom has posited the following caveat in our machine design: a superintelligent AI given the final goal of maximizing paperclip production would seek to turn everything in the observable universe into paperclips. It would create unlimited amounts of computronium to help it determine how to achieve this goal, and cover any surface with paperclip-producing technology. All humans would inadvertently be either converted into paperclips or unable to survive due to the AI’s commandeering of their biological necessities for the purpose of making paperclips.

The paperclip experiment is an enlightening toy problem. Bostrom has framed a non-obvious conjecture about the behavior of nonhuman entities in an intuitive, familiar setting. The harder implicit claim here is that when an agent includes free parameters in its decision calculus, these unbounded variables will be set to extreme values if that makes it easier to maximize the other variables. From the perspective of humans, this is not obvious, because we fail to take into account the way that we subconsciously bound even the parameters we consider free in our calculus. A superintelligent human paperclip-maker with human values (maybe a self-improving, high-speed, faithful whole brain emulation) probably wouldn’t turn the entire universe into paperclips–not only because he knows that then the paperclips would never be used. The AI too is aware of the uselessness of the paperclips once humans are all dead. But this is of no import to the AI, because it is single-minded in pursuance of its final goals.

The better reason for the superintelligent human not to make paperclips out of all materials at his disposal is that he does not consider the amount of resources he is permitted to consume in doing so to be unlimited. Thus, his limit for “percentage of the observable universe that should be converted into paperclips” is set far lower than the default bound. Humans often say “I’m going to do X, no matter what,” but they never mean it. There are no free parameters in human value calculus. Part of this is related to the liquidity of human capital: consider the case of an individual who wants to maximize the number of stores from which she buys a carton of milk. The longer her road trip, the more she pays for gas, and this money then can’t be used for buying milk. The kinds of goals humans will have also weigh into this phenomenon: why on earth would anyone pursue this geographically-disperse-milk end, except for some sort of competition, the victory of which would constitute a finite reward? This thinking implies its own parametric settings, e.g. the total costs of gas plus milk plus whatever else cannot together exceed the competition prize, or must be lower by some lambda-value. But the more interesting aspects of human value calculus have to do with the preferencing of the self and others as ends, which de facto prevents setting parameters arbitrarily high in ways that would have negative externalities for mankind. In the sense of a priori parameters, this reasoning seems circular, but humans don’t have any prior parameters in the true sense. We learn simultaneously in rationality and virtue.

So the superintelligent AI makes as many paperclips as it can with no heed paid to the destruction of the universe. Critics of this first iteration of the paperclip problem respond, “Easy! Let’s tell it to make exactly one million paperclips–no more, no less–and then turn off.”

This is where Bostrom’s true genius manifests, in the form of the quote excerpted above. The new-and-improved AI in the second iteration of the paperclip problem will make exactly one million paperclips, and then might direct a ridiculous quantity of computronium to the purpose of doing calculations that will heighten its credence that it really exists, that the paperclips really exist, and that it has therefore achieved its goal. Perhaps it will build factories for the microscopic inspection of each paperclip, or superprocessors to read and evaluate Hume and Berkeley and Avicenna and Descartes. But one thing it likely won’t do is shut down–because, as I mentioned before, it is single-minded in pursuit of its goal. Here, single-mindedness looks like achieving certainty that it has in fact completed the goal, with probability 1.

This kind of reasoning is alien to us. Because all human value calculus works exclusively with bounded variables, certainty is not a human first principle. We do the best we can, given the circumstances, because our resources are not unlimited in any direction, and therefore we cannot conduct evidence-gathering whose results have diminishing returns as probability approaches 1. A superintelligent human paperclip-maker might have some low Bayesian prior that she is dreaming, or hallucinating, or in a simulation, but this fact does not weigh into her paperclip-constructing calculus substantially, because the amount of time and energy she deems necessary to come to a definitive conclusion on whether 1 million paperclips have certainly been produced–which is probably asymptotically infinite–exceed the amounts she is willing to contribute to this end.

Bostrom is acutely aware of the fact that the certainty dilemma might not adversely affect AI alignment at all. Superintelligent machines will probably throw out the Bayesian framework and find a better one. Perhaps there are lines of reasoning far beyond our human capacity that would allow an advanced agent to come to certainty of his and the paperclips’ physical existence in finite time. For aught we know, it’s perfectly possible to prove a system works from within the system, even though Godel and Tarski have shown this is impossible within some of our systems. Maybe there are meta-principles other than the deductive, inductive, or abductive ones–systems such that conclusions stand outside the system. On the other hand, maybe the opposite is true: that higher intelligences will only be more torn by this problem than our finest logicians have been. They’ll likely conceive of scenarios we couldn’t dream up–ways in which the paperclips could be real when the AI isn’t, for example–that will then weigh into their value calculus for one side or the other.

Nonetheless, it would be reasonable to expect AI to behave roughly as Bostrom describes, and this behavior could have disastrous consequences, so we should investigate how to avoid intelligent machines’ characterization of the universe at large as a bunch of free variables. (See? I, a human thinker, don’t use certainty at all–just Bayesian priors and expected-utility theory, both of which have been specifically built to avoid certainty-paralysis.) Here, the difficult thing for human thinkers to parse through is the relationship between the ends “completion of goal” and “attainment of certainty that goal is complete.” Let’s say some AI was instructed to make a million paperclips and then [immediately] turn off, and it spent 30 minutes making the paperclips and 12 years proving definitively that it had in fact made a million paperclips. In retrospect, it can conclude that it completed its goal 12 years ago–so did it obviate the goal of turning off right afterwards? How should we view the goal of “immediately” turning off in light of a) making a million paperclips and b) achieving certainty that it had done so?

If we better understood what our values are, we could attempt to instantiate some metric that would give intelligent machines goals in terms of finite quantities of resources. For instance, consider an AI told to “make exactly 1 million paperclips, but spend no more than 1 thousand utils doing so,” for some arbitrary util metric involving space and time and matter and human happiness flourishing. But if the AI had good reason for high Bayesian priors that it wasn’t real, then subjecting it to this metric could lead it to spend the thousand utils researching existential philosophy instead of making paperclips at all. Or perhaps the juxtaposition of this NP-hard certainty question with finite resources would lead the AI to first engage at the meta-level, in a cost-benefit analysis, the completion of which would use up some of its resources and still might lead it to conclude it should research its existence rather than make paperclips. In these cases, the AI would be useless to us, which is better than disastrous, but defeats the purpose of creating intelligent machines in the first place. Alternatively, we could program our seed AI to uphold as a final goal “don’t weigh [Bayesian] certainty in your calculations of whether you’ve completed your other final goals,” or “accept with probability 1 that you and the things you produce are real.” Or we could give it our unproven meta-principles: “Take as proven the inductive method on sets larger than [arbitrary] n.” But if the superintelligent machine could prove the verity of existence, then we are wasting our time using it for such a meaningless task. We need an approach that will diversify such that the AI will perform a menial task when that’s what we would want it to do if we were smarter, and research existential philosophy when we’d want it to do that instead. And anyway, the inductive method is not always right–what will happen when the AI inevitably discovers an exception, and realizes it is pursuing its other final goals using unsound logic?

I think the proper approach is to program the AI to preference human-motivating probability metrics over absolute certainty in its goal completion criteria unless it is epsilon-sure–defined in terms of a human metric–that it can do better in polynomial time, for some high epsilon. Of course, we’d have to formulate this such that the AI doesn’t then devote non-polynomial time to deciding whether it’s epsilon-sure, and such that the programming doesn’t preclude the AI from developing better frameworks than Bayesianism if it can do so quickly. This approach will require us to energetically revisit our own uses of rapid cost-benefit analysis for CS problems. Graph-theoretic problems are generally easy to categorize at a glance into P, NP, and NP-hard, so this might be a good place to look.

Earlier, I sidestepped the issue of how a superintelligent human would handle the issue of certainty, assuming they’d operate similarly to how we do. We can see very clearly why an intelligent machine would pursue certainty as a corollary to its end goal, but faithful WBEs are trickier. Consider a superintelligent WBE instructed to make exactly 1 million paperclips (and who values this as an end goal), but has no free parameters in decision calculus and values inherent human values, whatever they are. In other words, in what way will an increase in intelligence affect human certainty preferences? I’d be inclined to say it won’t change much, but, as I’m human, I must admit I’m not sure.


The Virtue Transduction: Reflections on Ash Wednesday

“The majority of signal transduction pathways involve the binding of signaling molecules, known as ligands, to receptors that trigger events inside the cell. The binding of a signaling molecule with a receptor causes a change in the conformation of the receptor, known as receptor activation.” -The Wikipedia article on signal transduction

Before I was a Catholic convert, I was an atheist missionary.

Well, an ineffective one. I didn’t stand on street corners handing out bottles of water and copies of On the Origin of the Species to clubgoers. I didn’t go door to door asking Cleveland residents if they’d heard of our lord and savior, David Hume. If you’d asked me why not, I would’ve said, “I don’t go door-to-door telling people not to believe in Santa Claus either. The burden of proof is on the religious, as it should be.”

I would’ve been lying. For all my feigned reticence from activities antireligious, I eagerly climbed atop the pulpit when safely nestled within my steadfast circuit of militantly atheist friends. Together we’d scorn the blind credence of the faithful, not bothering to hide our laughter when someone said grace at the neighborhood Easter party.

So, I repeat: I was an atheist missionary, albeit an ineffective one. I chose to push effortlessly at open doors, rather than fruitlessly at locked ones. For the same reason, I had my most ardent political debates within the confines of my school’s Green Party club, opening discussions by repeating the words we all already knew we espoused.

There’s a cultural notion that each individual is a missionary, peddling his worldview. In some ways, this idea is correct. The soapbox orator seeks to attract passersby with flashing lights, bright colors, and raised voices. His beliefs are his signal, presented to the outside world, eagerly awaiting reception. He particularly enjoys preaching to the choir, because this process is more likely to result in a pleasant cacophony of go-ahead lights. Because he doesn’t care if the receptor’s admission of his signal was ready or hard-earned, there is no point in wasting his time converting the heathens.

Therefore, these orators more resemble cell ports than peddlers in their signaling. They await a predictable response light–a signal of “Well done, keep going”–rather than attempting to persuade a reluctant buyer. With this in mind, I posit the following theorem: the missionary-to-convert ratio will always be high. Why? Because we are all more concerned about our capable transmission of the signals we project than we are about their reception in unlikely places. Fittingly, then, we are least concerned with our own reception of others’ signals.

Has value transfer itself become a process of virtue signaling? Are we doomed to forever be ineffective missionaries, more concerned with demonstrating that we are missionaries–a procedure forwarded by the green-light sign from the community of believers–than with actually converting anyone? Echo chambers are safe, warm, and comforting, but they’re no place for politics. If this phenomenon exists as I’ve diagnosed it, we must stop it.

My computer hood is unfashionably bare. It doesn’t contain the requisite effective altruist stickers or space colonization decals typical of laptops belonging to those of like minds. But how far must I take this ideological purity? Most of the time that I wear a rosary or walk around with a conspicuous copy of Borges or Losing Ground or Global Catastrophic Risks, it’s thoroughly accidental and not meant to signify anything. Is intent the level on which the wrongdoing is occurring–and if so, is this the level on which it ought to be corrected? (Those two statements are not equivalent.) Must we engage in exhaustive good-faith meta-analysis of all our signaling mechanisms, whether unconscious, directed, or just preening? There are multi-level operators at work: perhaps I bought my brand-name shirt to ally myself with a cause, but wear it for convenience, or vice versa.

This morning, I will go to an early mass and participate in an age-old Catholic ritual: a cross of ash, blaring loud on my forehead. Why am I doing this? Because it’s in accordance with my faith. Why is it in accordance with my faith? Various reasons, some theological, others practical. I won’t get into the theological here–it’s important and beautiful, but not related to my point. Practically speaking, the Church is reasonable to demand this small act of devotion from its constituents. Technically I’m allowed to wash off the ash, but many Church officials encourage Catholics not to do so, as a public declaration of faith. Indeed, it is good to have the kind of faith that makes an individual proud to have a cross of ash on Ash Wednesday. Unfortunately, this virtue of proud faith has the inadvertent corollary of being a signaling mechanism, easily trivialized by a poorly understood notion of why one has ash on one’s forehead on Ash Wednesday–“because I’m supposed to.”

Imagined conversation between Catholic 1 with no ash on her forehead and Catholic 2 with ash on his:

C1: “Why are you wearing ash on your forehead?”

C2: “Because I am a Catholic and I want everyone to know it, and it is traditional to present that devotion in this way.”

C1: “Yes, but can you defend the particular act of wearing ash on your forehead, as a distinct and special indicator of your faith? You don’t wear a Catholic T-shirt every day, or very conspicuously bless your food before eating. Why is this a good method of engaging in an easily corruptible signaling procedure? Given that people might be wont to misinterpret the theological and practical implications of your wearing ash in a way that they wouldn’t misinterpret a large-font sign hung around your neck containing Catholic doctrine, why are you doing this particular thing? And given that the gesture of having ash on your forehead is immediately interpretable to other ash-wearing Catholics, with whom you exchange a response signal, but not to non-Catholics who perhaps have never seen it before, this can’t be an effective missionary mechanism. Are there other virtuous purposes of loudly stating your beliefs to the world that aren’t just about converting people?”

C2: “I could answer all your questions with some version of ‘I do what I’m told because of the mystery of faith,’ but I don’t think that’s the correct answer here. No, the ashes are not a missionary mechanism. Placed on my forehead, they encourage me and perhaps subconsciously others to engage in meaningful reflection; that makes them a better manner of engagement than, say, a T-shirt. Whenever I look in a mirror, I will remember that to dust I will return, and celebrate the good news of the Lord. But ultimately, I wear them because outwardly declaring my love for Christ is an end in itself. Saint Peter’s mistake in denying Christ thrice was to eschew that end.”

Thank you, Catholic 2! I love it when the attacked person in my imagined conversations is able to simply and elegantly defend himself. Yes–there are several brands of signaling here that we ought to separate.

  1. Creating a signal so that others will join your cause (e.g. handing out fliers for veganism).
  2. Creating a signal so that others will notice that you self-affiliate with a cause; social signaling, virtue signaling (e.g. linking an article without providing commentary).
  3. Creating a signal because manifesting your devotion to the cause is helpful to the cause. (This one’s trickier; e.g. wearing a work uniform so you can be identified by clients who need your help, which will cause the business to be more successful.)
  4. Doing something signaling your devotion to a cause, as an end in itself (e.g. wearing ash on Ash Wednesday).

1, 3, and 4 seem to me to be good-faith manners of engagement, and constitute the ways in which we should think about our signaling.

“And when you pray, do not be like the hypocrites, for they love to pray standing in the synagogues and on the street corners to be seen by others. Truly I tell you, they have received their reward in full. But when you pray, go into your room, close the door and pray to your Father, who is unseen. Then your Father, who sees what is done in secret, will reward you.” ” -Matthew 6:5-6, NIV

The hypocrites love to be seen by others–but this fact in itself is not the source of their problem. They pray only to be seen by others, manifesting virtue without needing to enact any. In proclaiming devotion to a cause as an end in itself, we can avoid this hypocrisy and signal a virtue that is thoughtful, deliberative, and, above all, felt.


Space Colonization Manifesto

“Of the arguments for Vegetarianism, none is so weak as the argument from humanity. The pig has a stronger interest than anyone in the demand for bacon. If all the world were Jewish, there would be no pigs at all.” – Leslie Stephen, “Social Rights and Duties”

A shadow overhangs the frameworks of utilitarianism and effective altruism, a shadow blatant and nefarious in its scope. This shadow is the collective preferencing of the group “things that exist” over the group “things that do not [yet] exist.”

The former group here is of special notice. We know with a high degree of certainty its subgroup “things that exist and are observable by us”—namely, the planet Earth and the life that exists on it. The (perhaps) larger group it contains is vaguer. Our anthropic biases inform us that “humanity” is important; that things that have some degree of proximity to humans have moral worth; that therefore “life,” or some subset of “life,” is valuable. There may be other elements within “things that exist” that do not at all resemble “life” as we know the term, but should still have moral value to us: God, if he exists; intelligent information processors that do not constitute cellular “life,” if they exist. So the frameworks are haunted too by the equally pervasive shadow of the anthropic bias.

But the second group, “things that do not [yet] exist,” is completely unknowable by all modern means, as is clear from the bracketed word. Some things that do not now exist will later come to exist; others will not. How should we discern between these two categories? Furthermore, future entities that are temporally closer to us are relatively certain to come into existence, but are ill-defined: a woman who wants to have children values these unknown children as “things that will soon exist,” although she cannot imagine their conscious minds or attributes (and indeed, the child she will have if she decides to have it now will be a different child than the one she would have had in ten years).

Yet these entities seem very knowable in contrast to far-future entities, which are altogether unpredictable. The woman is fairly sure of the effect it would have on the child with which she is pregnant if she smokes a cigarette. Far less certain are the effects it would have on her child’s children, or her child’s children’s children, and so on.

Furthermore, even the concrete results of many of our actions are unknowable. A value-utilitarian who imposes no meta-principles finds himself torn between ways to spend his time, because of the unknowable consequences of relatively small actions. If he drives to the store to buy milk, a car crash may result. A friend of mine feared even disrupting the placement of a leaf when she was a child, because that leaf might be a signal left by a spy for his handler, the destruction of which would have disastrous fallout.

A life of meaning means a life of moral worth, which means a life of service to some good principle. But, as I’ve expressed above, these principles are loaded. So we apply meta-cognition; we decide to build the following preferences into our value calculi:

  1. Actions that have likely or obvious consequences should weigh more heavily than those whose consequences are indiscernible; and
  2. Actors who exist should weigh more heavily than actors who do not exist.

There are two situations that are very easily handled by these principles;

  1. Actions that have likely consequences for actors who exist.
  2. Actions that have indiscernible consequences for actors who do not exist.

One situation gives us pause:

  1. Actions that have indiscernible consequences for actors who exist.

But this one can be written off. We should cast 3 aside, because it leads us to paralysis. If we aren’t sure what the immediate consequences are of going to the store to buy food for the homeless, this indecision will prevent us from doing the good that is an expected result. We turn our attention to the much more challenging fourth permutation of the two meta-principles I describe.

  1. Actions that have likely consequences for actors who do not exist.

Why should this avenue be important? Expected value theory does not differentiate between “10% chance of winning $100” and “100% chance of winning $10.” We need a principle to distinguish between these cases, a principle that too applies to separating 3 and 4. My system views salience as critical. The salient attribute of the lottery example above is certainty–that is, we should prefer the 100% chance of winning $10, because there is something enticing about absolute certainty that is not reflected in our standard calculus of Pr(X)Value(X). Similarly, almost nobody would enter a lottery with a 99% chance of winning $1M and a 1% chance of death–because very negative outcomes are more salient than very positive ones.

Existential risk researchers get an unfair reputation of being frantic doomsayers. And sure, Pascal’s wager is silly, but what’s sillier is to dismiss the undeniable salience of high-impact outcomes. Millions of people play lottery. They may all be wrong and irrational, but they’re also all human–and therefore give a decent indication of what human values are or should be.

Thus, Situation 4 should be a topic of focus; a virtuous life necessitates a lasting commitment to the principle of human happiness flourishing, a prioritization of existential threats to humanity, and a certainty that the significance of global catastrophe is the salient attribute of situation 4, rather than the insignificance of nonexistent actors.

The Carter-Leslie Doomsday argument posits that the end of human existence is nearing, and, as a corollary, that human experience is approaching its peak. The pinnacle of virtue, of epistemic modesty, and of service lies in embracing the desire not to live at the peak of human experience—that is, to want greatly for human existence to continue, and to continue advancement, long after its current constituents no longer exist. What altruistic desire could be more selfless than to want dearly for millions or trillions of humans to live satisfying lives when we cannot be around to revel in that self-indulgent part of virtue–the part that says “I made this happen”? Certainly it is a more virtuous pursuit to act well solely for the sake of the other.

Yet the Earth will not be able to sustain trillions of humans, considering its size, the window during which it will be habitable, and the disturbing consequences of approaches like Downsizing. Therefore, I posit that we expand.

First we must rein in our destructive capacities, clarify our human values and mission, and settle on what the human condition is and should be. We must figure out how are values are changing and develop further meta-principles to direct the ways in which they should change. These preliminaries are not safety caveats to be checked off before zooming to the skies–their resolution will likely constitute all of the groundwork for interstellar colonization that we will ever see, and will probably still be incomplete when all of us are gone. But by solving these vital questions, we will place our descendants in a position to take up the mission of human happiness flourishing–thoughtfully, deliberately, intelligently, and humanely.

“Be fruitful and multiply; fill the earth and subdue it,” God is said to have instructed the first humans. And to Abraham: “I will make your descendants as numerous as the stars.”

To fulfill that mandate, let’s place our descendants among the stars.