Inbox - Cold Takes (new posts on Tuesdays) - GiveWell blog - AK impression: HK likes excited altruism; doesn't really buy aggregation; kind of optional, predicted on him being excited about it - ... --- https://forum.effectivealtruism.org/posts/nA3R6Hm8x8CyzRHS2/future-matters-march-2022-1 How can we avoid making terrible moral mistakes, like those of past generations? Holden Karnofsky suggests three features of ethical systems capable of meeting this challenge: (1) Systematization: rather than relying on bespoke, intuition-based judgements, we should look for a small set of principles we are very confident in, and derive everything else from these; (2) Thin utilitarianism: our ethical system should be based on the needs and wants of others rather than our personal preferences, and therefore requires a system of consistent ethical weights for comparing any two harms and benefits; and (3) Sentientism: the key ingredient in determining how much to weigh someone’s interests should be the extent to which they have the capacity for pleasure and suffering. Combining these elements leads to the sort of ethical stance that has a good [track record](https://www.utilitarianism.net/introduction-to-utilitarianism#track-record) of being ‘ahead of the curve.’ ## Other centred ethics https://forum.effectivealtruism.org/posts/iupkbiubpzDDGRpka/other-centered-ethics-and-harsanyi-s-aggregation-theorem#The_scope_of_utilitarianism 1. **Utilitarianism allows "utility monsters" and "utility legions."** A large enough benefit to a single person (utility monster), or a benefit of any size to a sufficiently large set of persons (utility legion), can outweigh all other ethical considerations. Utility monsters seem (as far as I can tell) to be a mostly theoretical source of difficulty, but I think the idea of a "utility legion" - a large set of persons such that the opportunity to benefit them outweighs all other moral considerations - is the root cause of most of what's controversial and interesting about utilitarianism today, at least in the context of effective altruism. ## Defending One-Dimensional Ethics https://www.cold-takes.com/defending-one-dimensional-ethics/ the “win-win” principle. Say that you’re choosing between two worlds, World A and World B. Every single person affected either is better off in World B, or is equally well-off in both worlds (and at least one person is better off in World B). In this case I think you should always choose World B. If you don’t, you can cite whatever rules of ethics you want, but you’re clearly **making a choice that’s about you and your preferences**, not about trying to help others. Do you accept that principle? --- I think that “what most people recognize as morality” is a mix of things, many of which have little or nothing to do with making the world better for others. Conventional morality shifts with the winds, and it [has often included](https://www.cold-takes.com/future-proof-ethics/) things like “homosexuality is immoral” and “slavery is fine” and God knows what all else. [...] “You’re scratching your own moral-seeming itches. You’re making yourself feel good. You’re paying down imagined debts that you think you owe, you’re being partial toward people around you. Ultimately, that is, your philanthropy is about you and how you feel and what you owe and what you symbolize. My philanthropy is about giving other people more of the lives they’d choose. “My giving is unintuitive, and it's not always 'feel-good,' but it's truly other-centered. Ultimately, I'll take that trade.” [...] Risk aversion is a fundamentally selfish impulse - it makes sense in the context of personal spending, but in the context of donating, it’s just another way of making this about the donor. [...] if your giving doesn’t conform to what the beneficiaries would want under the veil of ignorance, then it has to be in some sense about you rather than about them. You have an impulse to feel that you’re “doing your part” on multiple causes - but that impulse is about your feelings of guilt, debt, etc., not about how to help others. ## Future-proof ethics https://www.cold-takes.com/future-proof-ethics/ **Systemization can be weird.** It’s important to understand from the get-go that **seeking an ethics based on “deep truth” rather than conventions of the time means we might end up with some very strange, initially uncomfortable-feeling ethical views.** The rest of this series will present such uncomfortable-feeling views, and I think it’s important to process them with a spirit of **“This sounds wild, but if I don’t want to be stuck with my raw intuitions and the standards of my time, I should seriously consider that this is where a more deeply true ethical system will end up taking me.”** ## Are ideas getting harder to find? {{Prefers "innovation as mining" to "innovation as conjuring ideas from thin air".}} When an innovator publicizes an idea, they're _speeding up_ how fast the world benefits from the idea, but if it weren't for them, _someone else_ would have had a chance to come up with something similar. [...] I think that today's discourse around "ideas get harder to find" is overly obsessed with improving culture and institutions, as opposed to increasing the sheer number of people with a shot at being innovators - which I see as a likely larger and more sustained route to increased innovation. ## Most important century: in a nutshell The last few millions of years - with the start of our species - have been more eventful than the previous several billion. The last few hundred years have been more eventful than the previous several million. If we see another accelerator (as I think AI could be), the next few decades could be the most eventful of all. A sense that the world and the rules we're all used to can't be relied on. That we need to lift our gaze above the daily torrent of tangible, relatable news - and try to wrap our heads around weirder, wilder matters that are more likely to be seen as the headlines about this era billions of years from now. There's a lot I don't know. But if this is the most important century, I do feel confident that we as a civilization aren't yet up to the challenges it presents. If that's going to change, it needs to start with more people seeing the situation for what it is, taking it seriously, taking action when they can - and when not, staying vigilant. ## Most important century: Klein summary 1. Longrun future could be just radically unfamiliar. It could be a radical utopia, dystopia, anything in between. 2. The long-run future could become the near-term future if the right kind of A.I. is developed to accelerate science and technology dramatically. 3. That kind of A.I. looks more likely than not this century. (4) A natural reaction is that this implies we’re in a very special time. And it sounds too wild to be true. But if you step out and look at our place in history, it looks like we’re in a very weird and wild time for many reasons that have nothing to do with A.I. And so we should be ready for anything. ## Key open questions some key open questions: Open question: how hard is the alignment problem? 몭e path to the future that seems worst is Misaligned AI, in which AI systems end up with non-human-compatible objectives of their own and seek to 몭ll the galaxy according to those objectives. How seriously should we take this risk - how hard will it be to avoid this outcome? How hard will it be to solve the "alignment problem," which essentially means having the technical ability to build systems that won't do this?9 Some people believe that the alignment problem will be formidable; that our only hope of solving it comes in a world where we have enormous amounts of time and aren't in a race to deploy advanced AI; and that avoiding the "Misaligned AI" outcome should be by far the dominant consideration for the most important century. 몭ese people tend to heavily favor the "caution" interventions described above: they believe that rushing toward AI development raises our already-substantial risk of the worst possible outcome. Some people believe it will be easy, and/or that the whole idea of "misaligned AI" is misguided, silly, or even incoherent - planning for an overly speci몭c future event. 몭ese people o몭en are more interested in the "competition" interventions described above: they believe that advanced AI will probably be used e몭ectively by whatever country (or in some cases smaller coalition or company) develops it 몭rst, and so the question is who will develop it 몭rst. And many people are somewhere in between. even among the few people who have spent the most time thinking about these matters, there is practically no consensus or convergence on how hard the alignment problem will be. Should we be expecting transformative AI within the next 10-20 years, or much later? Will the leading AI systems go from very limited to very capable quickly ("hard takeo몭") or gradually ("slow takeo몭")? 12 Should we hope that government projects play a major role in AI development, or that transformative AI primarily emerges from the private sector? Are some governments more likely than others to work toward transformative AI being used carefully, inclusively and humanely? What should we hope a government (or company) literally does if it gains the ability to dramatically accelerate scienti몭c and technological advancement via AI? ### Robustly helpful actions - Technical research on the alignment problem - Pursuit of strategic clarity: doing research that could address other crucial questions (such as those listed above), to help clarify what sorts of immediate actions seem most useful - Helping governments and societies become, well, nicer - Spreading ideas and building communities. Today, it seems to me that the world is extremely short on people who share certain basic expectations and concerns, such as: Believing that AI research could lead to rapid, radical changes of the extreme kind laid out here (well beyond things like e.g. increasing unemployment). Believing that the alignment problem (discussed above) is at least plausibly a real concern, and taking the "caution" frame seriously. Looking at the whole situation through a lens of "Let's get the best outcome possible for the whole world over the long future," as opposed to more common lenses such as "Let's try to make money" or "Let's try to ensure that my home country leads the world in AI research." I think it's very valuable for there to be more people with this basic lens, particularly working for AI labs and governments. If and when we have more strategic clarity about what actions could maximize the odds of the "most important century" going well, I expect such people to be relatively well-positioned to be helpful. ## All possible views are wild I discuss alternatives to my view: a "conservative" view that thinks the technologies I'm describing are possible, but will take much longer than I think, and a "skeptical" view that thinks galaxy-scale expansion will never happen. Each of these views seems "wild" in its own way. I am not advocating excitement or glee at the prospect of expanding throughout the galaxy. I am advocating seriousness about the enormous potential stakes. there's a good chance of a productivity explosion by 2100, which could quickly lead to what one might call a "technologically mature" civilization 몭e di몭erence between your timeline and mine isn't even a pixel, so it doesn't show up on the chart. In the scheme of things, this "conservative" view and my view are the same. It's true that the "conservative" view doesn't have the same urgency for our generation in particular. But it still places us among a tiny proportion of people in an incredibly signi몭cant time period. And it still raises questions of whether the things we do to make the world better - even if they only have a tiny 몭ow-through to the world 100,000 years from now - could be ampli몭ed to a galactic-historical-outlier degree. Maybe humanity is destined to destroy itself before it reaches this stage. But note that if the way we destroy ourselves is via misaligned AI, 7it would be possible for AI to build its own technology and spread throughout the galaxy, which still seems in line with the spirit of the above sections. In fact, it highlights that how we handle AI this century could have rami몭cations for many billions of years. So humanity would have to go extinct in some way that leaves no other intelligent life (or intelligent machines) behind. [...] Ultimately, it's very hard for me to see a case *against* thinking something like this is at least *reasonably* likely: "We will eventually create robust, stable settlements throughout our galaxy and beyond." It seems like saying "no way" to that statement would itself require "wild" con몭dence in something about the limits of technology, and/or long-run choices people will make, and/or the inevitability of human extinction, and/or something about aliens or simulations. [...] But space expansion seems feasible, and our galaxy is empty. 몭ese two things seem in tension. [...] My current sense is that the best analysis of the Fermi Paradox available today favors the explanation that intelligent life is extremely rare: something about the appearance of life in the 몭rst place, or the evolution of brains, is so unlikely that it hasn't happened in many (or any) other parts of the galaxy.9 몭at would imply that the hardest, most unlikely steps on the road to galaxy-scale expansion are the steps our species has already taken. And that, in turn, implies that we live in a strange time: extremely early in the history of an extremely unusual star. If we started 몭nding signs of intelligent life elsewhere in the galaxy, I'd consider that a big update away from my current "wild" view. It would imply that whatever has stopped other species from galaxy-wide expansion will also stop us. It looks for all the world as though our "tiny dot" has a real shot at being the origin of a galaxy-scale civilization. It seems absurd, even delusional to believe in this possibility. But given our observations, it seems equally strange to dismiss it. And if that's right, the choices made in the next 100,000 years - or even this century could determine whether that galaxy-scale civilization comes to exist, and what values it has, across billions of stars and billions of years to come. So when I look up at the vast expanse of space, I don't think to myself, "Ah, in the end none of this matters." I think: "Well, some of what we do probably doesn't matter. But some of what we do might matter more than anything ever will again. ...It would be really good if we could keep our eye on the ball. ...[gulp]" ### 80K Cold Takes interview RW: lots of people think the longtermist worldview is kinda weird or somehow violates common sense, but RW thinks that "if you just zoom out a bit and look at the broad situation that humanity is actually in, then the topics and projects longtermists focus on make a lot more sense both common and otherwise than the alternative approaches to doing good that I often hear proposed. I don't really feel like I'm on the back foot defending this work, because the prima facie common sense case for longtermism is in my view as strong as what anyone else has." Other factors… I mentioned the thing about just coming to believe that we’re in a wild time and really struggling with the burden of proof argument, which is something that I really focus on a lot. Another really big factor for me was a shift in the nature of the argument. So I think traditionally there’s a very similar argument to what I’ve been making that motivates a lot of e·ective altruists, but it is di·erent in an important way. Holden Karnofsky: And the argument goes… It’s much more of a philosophy argument and much less of an empirical argument. The argument goes, the best thing we could do is reduce the risk of an existential catastrophe, or the best thing we could do is increase the probability that humanity builds a really nice civilization that spans the galaxy. And then now let’s look for something that could a·ect that. And the thing that could is AI. So now if there’s any reasonable probability, that’s what we’re going to work on. And that’s an argument where a lot of the work is getting done by this philosophical point about valuing… Having this astronomical valuation on future generations, because you’re not making an argument that it’s going to happen or that it could happen with reasonable probability, you’re just saying, “Well, if it did.” And then of course you can get to there, you can say— Rob Wiblin: You’re not starting there. You’re starting the argument with looking for something else, rather than saying, “It seems like if we look out into the world that this thing could possibly happen quite soon, just on its own face.” Holden Karnofsky: Exactly. You’re not starting there. And then you’re also not ending there either. [...] Holden Karnofsky: I’ve had internal discussions at Open Philanthropy about why we work on what we work on, and I call them the ‘track one’ and ‘track two’ arguments. Track one is the empirical, this is a huge event that is coming and I don’t know exactly how to value it, but even if you value it pretty conservatively, even if you just say, like, I don’t know, preventing an existential catastrophe from AI is 100 times as good as saving all the lives of people on Earth today, then it looks like something we should work on. And then the other direction is saying, no, this is just so astronomically consequential that even a very tiny probability makes it something we should work on. And they’re di·erent arguments, but I’ve never been a person who’s incredibly comfortable making a big contrarian bet that is based on that reasoning. The kind of philosophical, “This is how much value there is.” Holden Karnofsky: And I do feel way more conviction and way more buy-in from just saying I don’t know if I’ve got the values right. I don’t know if I’ve got the population ethics right. But there is something really, really, really big that could be happening, and we have to zoom out, look at our place in history, see how weird it is, and ask what’s going to be coming next. That’s the most responsible thing for us to do as people trying to help the world. And then we can run the numbers and make sure it pencils, but that’s a better starting point for me. Holden Karnofsky: I think I agree with everything you said. I do want to caution a little bit that I think a lot of people who talk about these topics, they’re transhumanists, and they’re very enthusiastic, and it’s all with an air of enthusiasm, and it’s all under this assumption that technology is great, and more of it is great, and everything’s going to be great, and let’s talk about all the ways things would be great. And I just, I actually just waÀe a lot on this, but I think it’s important to recognize that rather than take a position on whether it’ll be great or whether it’ll be terrible — because I think a lot of non-transhumanists just hear this stu· and they think it’s the most horrifying thing they’ve ever heard — rather than take a position either way on this, I just want to be like, “It could be either.” Holden Karnofsky: And I think that’s one of the things I’ve had trouble with with this series. Everyone wants to read everything as “This is going to happen. It’s going to be awesome.” Or, “This is going to happen. It’s going to be terrible.” And I’m really, really, really… I’ve had trouble convincing people of this, and I have to say this over and over again in this series, I’m really not saying either one. It really could be either. When I think it could be the most important century, I’m not thinking, “Woo-hoo, it could be the most important century. We’ll never have disease.” And I’m also not thinking, “Oh no, it’ll be the most important century. Everything’s going to be bad. It’s going to be worse than it was.” Holden Karnofsky: I’m just thinking, “Oh my god, oh my god, I don’t know what to do. Oh geez. Anything could happen. It could be the best. It could be the worst. Geez. We’re not ready for this. And we really need to think about this. And we really need to think about how this could go. It could go really well. It could go really poorly. We need to think about how to start to get a grip on this.” So, I think that’s important as the way I’m trying to approach this stu·. I think it’s hard for a lot of people to not be thinking, “Is this going to be good or bad?” Well, I think it is good to think it could be good or bad. It’s good to recognize it could be either. [...] Holden Karnofsky: But I will say something important, which is I don’t think anyone has done the projections the right way. So I think when people are saying, “Well, I think this is all too crazy.” They’re not MENUlike, “I did the projections, they’re not happening.” That’s not what’s happening there. So I think someone needs to ¹gure this out. [...] On Simulation argument: Holden Karnofsky: So I think that’s fundamentally reasonable, but we concluded that it doesn’t really change much. It doesn’t change nothing. It changes some of the numbers, the numbers you see in the essay Astronomical Waste, those numbers change, but it doesn’t really change the bottom line. It doesn’t change, what should we be doing? What matters? In a way that we were easily able to see. And I don’t really think this is a topic that is worth a lot of deep investigation of, or needs a big deep dive, but it was something I wanted to look at. I don’t think it should just be tossed out. I don’t think it ends up mattering very much, but it was important to me to think that piece of it through. [...] Holden Karnofsky: One thing you could mean is, how could it turn out that this isn’t the most important century, and none of this stu· happens? And for that one, I don’t know, there’s a zillion ways, and all I’m even saying is that it’s reasonably likely. There’s another thing you might mean, which is, are there investigations you could do today that when you are done, might change your mind and make you think that this is not a good bet? That this is not likely enough? Then another thing you could mean is, well, without doing further investigation, what are the best things people say that are just objections, that might be right? And then, a ¹nal thing — which I actually ¹nd maybe the most interesting — is just like, how could it be that this whole idea, and this whole vibe, is just the wrong thing to be thinking about? Even if the predictions are true, maybe it’s just the wrong thing to be thinking about, or just, “Holden is leading people o· the wrong cli· here.” Or o· some cli·, probably most cli·s that you’d go o· are the wrong cli·. How could that be? Those are di·erent things, so which one do you want to tackle? ## Ezra Klein interview EK: It can I think be performatively cold or logical in a way that's actually quite narrow about human flourishing. Huge philantrhopic success stories: borlaug green Revolution; the birth control pill. EZRA KLEIN: I think something striking about that list is the sheer diversity of things you all fund. Not only in terms of causes but categories of causes. And this gets to what I think of as one of the most interesting things Open Philanthropy does, which is the way you intentionally divide up your giving portfolio into buckets based on really different ethical, arguably even metaphysical, assumptions. So tell me about worldview diversification. HOLDEN KARNOFSKY: I need to start with the broader debate that worldview diversification is a part of. At Open Philanthropy, we like to consider very hard-core theoretical arguments, try to pull the insight from them, and then do our compromising after that. And so, there is a case to be made that if you’re trying to do something to help people and you’re choosing between different things you might spend money on to help people, you need to be able to give a consistent conversion ratio between any two things. So let’s say you might spend money distributing bed nets to fight malaria. You might spend money getting children treated for intestinal parasites. And you might think that the bed nets are twice as valuable as the dewormings. Or you might think they’re five times as valuable or half as valuable or ⅕ or 100 times as valuable or 1/100. But there has to be some consistent number for valuing the two. And there is an argument that if you’re not doing it that way, it’s kind of a tell that you’re being a feel-good donor, that you’re making yourself feel good by doing a little bit of everything, instead of focusing your giving on others, on being other-centered, focusing on the impact of your actions on others, which you can get from there to an argument that you should have these consistent ratios. So with that backdrop in mind, we’re sitting here trying to spend money to do as much good as possible. And someone will come to us with an argument that says, hey, there are so many animals being horribly mistreated on factory farms and you can help them so cheaply that even if you value animals at 1 percent as valuable as humans to help, that implies you should put all your money into helping animals. On the other hand, if you value them less than that, let’s say you value them a millionth as much, you should put none of your money into helping animals and just completely ignore what’s going on factory farms, even though a small amount of your budget could be transformative. So that’s a weird state to be in. And then, there’s an argument that goes, but even more than that — and this idea is called long-termism — if you can do things that can help all of the future generations, for example, by reducing the odds that humanity goes extinct. Then you’re hoping even more people. And that could be some ridiculous comic number that a trillion, trillion, trillion, trillion, trillion lives or something like that. And it leaves you in this really weird conundrum, where you’re kind of choosing between being all in on one thing and all in on another thing. And Open Philanthropy just doesn’t want to be the kind of organization that does that, that lands there. And so we divide our giving into different buckets. And each bucket will kind of take a different worldview or will act on a different ethical framework. So there is bucket of money that is kind of deliberately acting as though it takes the farm animal point really seriously, as though it believes what a lot of animal advocates believe, which is that we’ll look back someday and say, this was a huge moral error. We should have cared much more about animals than we do. Suffering is suffering. And this whole way we treat this enormous amount of animals on factory farms is an enormously bigger deal than anyone today is acting like it is. And then there’ll be another bucket of money that says, animals? That’s not what we’re doing. We’re trying to help humans. And so you have these two buckets of money that have different philosophies and are following it down different paths. And that just stops us from being the kind of organization that has stuck with one framework, stuck with one kind of activity. EZRA KLEIN: Before we move on, I want to unpack this a little bit more. So let’s focus in on animals for a minute. You alluded to the fact that even if you assign a very low moral worth to animals or to their suffering, 1 percent or 0.1 percent of that of a human, that it ends up adding up to quite a lot. Can you run through that math for me and its implications? HOLDEN KARNOFSKY: Well, the math would be that — I mentioned before that if you’re distributing insecticide treated bed nets, you might avert the death of someone from malaria for a few thousand dollars, which is pretty amazing. And it’s going to be very hard to find better than that when you’re funding charities that help humans. However, with the farm animal work, for example, the cage free pledges, we kind of estimated that you’re getting several chickens out of a cage for their entire lives for every $1 that you spend. And so this is not an exact equivalence, but if you start to try to put numbers side by side, you do get to this point where you say, yeah, if you value a chicken 1 percent as much as a human, you really are doing a lot more good by funding these corporate campaigns than even by funding the bed nets. And that’s better than most things you can do to help humans. Well, then, the question is, OK, but do I value chickens 1 percent as much as humans? 0.1 percent? 0.01 percent? How do you know that? And one answer is we don’t. We have absolutely no idea. The entire question of what is it that we’re going to think 100,000 years from now about how we should have been treating chickens in this time, that’s just a hard thing to know. I sometimes call this the problem of applied ethics, where I’m sitting here, trying to decide how to spend money or how to spend scarce resources. And if I follow the moral norms of my time, based on history, it looks like a really good chance that future people will look back on me as a moral monster. But one way of thinking, just to come back to the chickens question, one way of thinking about it is just to say, well, if we have no idea, maybe there’s a decent chance that we’ll actually decide we had this all wrong, and we should care about chickens just as much as humans. Or maybe we should care about them more because humans have more psychological defense mechanisms for dealing with pain. We may have slower internal clocks. A minute to us might feel like several minutes to a chicken. So if you have no idea where things are going, then you may want to account for that uncertainty, and you may want to hedge your bets and say, if we have a chance to help absurd numbers of chickens, maybe we will look back and say, actually, that was an incredibly important thing to be doing. EZRA KLEIN: I want to note something here because I think it’s both an important point substantively but also in what you do. So I’m vegan. Except for some lab-grown chicken meat, I’ve not eaten chicken in 10, 15 years now — quite a long time. And yet, even I sit here, when you’re saying, should we value a chicken 1 percent as much as a human, I’m like, ooh, I don’t like that. To your point about what our ethical frameworks of the time do and that possibly an open-field comparative advantage is being willing to consider things that we are taught even to feel a little bit repulsive considering, how do you think about those moments? How do you think about the backlash that can come? How do you think about when maybe the mores of a time have something to tell you within them, that maybe you shouldn’t be worrying about chicken when there are this many people starving across the world? How do you think about that set of questions? HOLDEN KARNOFSKY: I think it’s a tough balancing act because on one hand, I believe there are approaches to ethics that do have a decent chance of getting you a more principled answer that’s more likely to hold up a long time from now. But at the same time, I agree with you that even though following the norms of your time is certainly not a safe thing to do and has led to a lot of horrible things in the past, I’m definitely nervous to do things that are too out of line with what the rest of the world is doing and thinking. And so we compromise. And that comes back to the idea of worldview diversification. So I think if Open Philanthropy were to declare, here’s the value on chickens versus humans, and therefore, all the money is going to farm animal welfare, I would not like that. That would make me uncomfortable. And we haven’t done that. And on the other hand, let’s say you can spend 10 percent of your budget and be the largest funder of farm animal welfare in the world and be completely transformative. And in that world where we look back, that potential hypothetical future world where we look back and said, gosh, we had this all wrong — we should have really cared about chickens — you were the biggest funder, are you going to leave that opportunity on the table? And that’s where worldview diversification comes in, where it says, we should take opportunities to do enormous amounts of good, according to a plausible ethical framework. And that’s not the same thing as being a fanatic and saying, I figured it all out. I’ve done the math. I know what’s up. Because that’s not something I think. EZRA KLEIN: I’m struck by that. I really like worldview diversification as a way of thinking about things. And I think it’s also relevant as an individual practice. Something I see in my travels around the world, the internet, is people are very intent. Even if they would not say they are 100 percent confident in their worldview, their political ideology, their whatever, they are really interested in making it dominant against all comers. So, just tell me a bit about organizationally, intellectually, the discipline of maintaining a certain level of agnosticism between worldviews whose differences you can’t really answer. HOLDEN KARNOFSKY: So one of my obsessions is applied epistemology, which is like just having good systems for figuring out what your beliefs are in kind of an overwhelming flow of information that is today’s world. And I think one of the tools that some people use for it that I find really powerful and I’m going to write about is what I call the Bayesian mind-set, which is this idea that when you’re uncertain about something, you can always portray your uncertainty as a number. And you can portray it as a probability. There’s thought experiments. There’s tools for doing this. You can say, instead of something is true or false, that it’s 30 percent. And you can look back later and you can see if things that you said were 30 percent likely come through 30 percent of the time. I think this is a very powerful framework. And using it can often get you out of the head space of believing that things are true or false and just having degrees of belief in everything and often taking something very seriously, even when you think it probably won’t happen, just because it’s important enough and it has a high enough probability that it deserves your attention. And on the other hand, I think this framework sometimes can take people back into a state of fanaticism, where you might say, hey, here’s something that would be a really huge deal. And it’s at least 1 percent likely. So that means it should be the only thing I think about. It should be my obsession. It’s like the examples I was giving before. And that, I think, just lands you in a similarly dogmatic place. And so, Open Philanthropy is kind of operating two levels of uncertainty. It’s often using this Bayesian mind-set. But when the Bayesian mindset brings you to this implication that you’ll have to be all in on one thing or another, we’ll say no to that, too. And then we’ll just go to another level of diversification. And we’ll have different buckets with different philosophies on the world. EZRA KLEIN: I want to pick up on the fanaticism component. And I’m not accusing anybody here of fanaticism. But one of my critiques of the effective altruist world is that it can get very obsessed by that conversion number you were talking about a minute ago. And in particular, I think, it’s a culture as it has matured a bit more. There’s now an aesthetic, sometimes, of being willing to take the most hardhearted logic experiment seriously and show that you’re the real effective altruist because even though it sounds like a kind of terrible thing to do, you ran the math, and it’s not. And the way I’ll put this is, Will MacAskill, who’s a philosopher and was a founder of the effective altruist movement, used to have this thought experiment where there’s a building on fire. And there’s a family in one room who could die. And then there’s another room — or I think it was, actually, an attached garage or something — that has a bunch of very expensive art in it. What do you save? And the point of the experiment originally was you should, of course, save the family. And he was making the meta point that many people are donating to museums, instead of to malarial bed nets. I think now, a lot of effective altruists would answer it the other way, because the point is, well, if that art is worth $500,000 and you can turn that $500,000 into x number of malarial bed nets, that saves more than five lives. And so, of course, you need to do that. And I think that gets you into pretty dangerous territory. But I’m curious how you think about those questions. HOLDEN KARNOFSKY: I do agree that there can be this vibe coming out of when you read stuff in the effective altruist circles that kind of feels like it’s doing this. It kind of feels like it’s trying to be as weird as possible. It’s being completely hard-core, uncompromising, wanting to use one consistent ethical framework wherever the heck it takes you. That’s not really something I believe in. It’s not something that Open Philanthropy or most of the people that I interact with as effective altruists tend to believe in. And so, what I believe in doing and what I like to do is to really deeply understand theoretical frameworks that can offer insight, that can open my mind, that I think give me the best shot I’m ever going to have at being ahead of the curve on ethics, at being someone whose decisions look good in hindsight instead of just following the norms of my time, which might look horrible and monstrous in hindsight. But I have limits to everything. Most of the people I know have limits to everything, and I do think that is how effective altruists usually behave in practice and certainly how I think they should. EZRA KLEIN: What do you think the limit of that actual thought experiment is, of the just convert lives into money? You can save x number of lives for x number of money. And so if you get more money by getting the money as opposed to saving the lives, you should do it. HOLDEN KARNOFSKY: I think there’s a lot of problems with that argument. And I could sort of go into them. So there’s things about setting norms. There’s things about following rules so that you don’t want to be the kind of person who is constantly behaving in strange, unexpected ways and screwing over people around you because you’ve got this strange mathematical framework that’s going on. So I think there’s a bunch of things that are wrong with running in and saving the painting. But I think I also just want to endorse the meta principle of just saying, it’s OK to have a limit. It’s OK to stop. It’s a reflective equilibrium game. So what I try to do is I try to entertain these rigorous philosophical frameworks. And sometimes it leads to me really changing my mind about something by really reflecting on, hey, if I did have to have a number on caring about animals versus caring about humans, what would it be? And just thinking about that, I’ve just kind of come around to thinking, I don’t know what the number is, but I know that the way animals are treated on factory farms is just inexcusable. And it’s just brought my attention to that. So I land on a lot of things that I end up being glad I thought about. And I think it helps widen my thinking, open my mind, make me more able to have unconventional thoughts. But it’s also OK to just draw a line. I think it’s OK to look at this art thing and say, that’s too much. I’m not convinced. I’m not going there. And that’s something I do every day. EZRA KLEIN: So let’s say you have somebody listening, and you buy into this, or at least, you buy into it as a real possibility. And you want to try to help make the best of it. What is tractable for an individual here? What does this imply for just somebody’s life, if they want to try to live on this time scale? HOLDEN KARNOFSKY: The first answer I want to give is that I wish I had a better answer. I wish I could just tell you, hey, I’ve not only figured out what’s going to happen, which I haven’t — I figured out what needs more attention. But I figured out exactly what to do about it. But I haven’t done that either. And one of the things I say in my blog post series is, if you’re thinking, hey, this could be a billion-dollar company, maybe the right reaction is, yeah, awesome, let’s go for it. And if this could be the most important century, I think my reaction tends to be just like, ooh. I don’t know what to do with this. I need to sit down. I need to think about it. I think there’s a number of reasons that it’s actually very hard to know what to do. And I think we need more attention and more thought and more reflection because most things that I could name as an attempt to make the most important century better could be really good or really bad, depending on your model of what the most important considerations are. So I can get to that in a second, but I don’t want to be totally gloomy about it. I think there are things that look robustly helpful. There are things that look good. One of them is A.I. alignment research. So just, if we could get to a point where we’re capable of being confident that we can build advanced A.I. systems that aren’t just going to run the world according to whatever objectives they have, and that is a field that exists. That’s a field that people are working on. We fund work in it. Another robustly helpful thing is just trying to find and empower more people who are seeing the situation as it is, taking it seriously, being as thoughtful about it as they can be, and approaching it from the perspective of what’s best for humanity, instead of what’s best for them narrowly. So I do think there are some activities. And there’s some discussion of it in the blog post series. # 2021-08-27 Notes on Cold Takes ## Preparing the ground Warming up the audience to take weird futurist arguments seriously. Main ways of doing that: 1. We may find it normal, but our current situation is very weird and unusual. Point to: 1. History of economic growth 2. History of science and technology, impressive achievements 2. There are no non-weird but still plausible "business As usual" future scenarios Saying: future will be turbulent and weird. But also saying: there are important things we can know about the future. Don't write off all attempts to think about the future. ## Two headspaces - Business as usual - This can't go on In Business As Usual: > the world is constantly changing, and the change is noticeable, but it's not overwhelming or impossible to keep up with. There is a constant stream of new opportunities and new challenges, but if you want to take a few extra years to adapt to them while you mostly do things the way you were doing them before, you can usually (personally) get away with that. In terms of day-to-day life, 2019 was pretty similar to 2018, noticeably but not hugely different from 2010, and hugely but not crazily different from 1980.3 If this sounds right to you, and you're used to it, and you picture the future being like this as well, then you live in the Business As Usual headspace. When you think about the past and the future, you're probably thinking about something kind of like this: This can't go on: you zoom out from past 200 years and see a more turbulent past and a more uncertain future. > I think there should be some people in the world who inhabit the Business As Usual headspace, thinking about how to make the world better if we basically assume a stable, regular background rate of economic growth for the foreseeable future. And some people should inhabit the This Can’t Go On headspace, thinking about the ramifications of stagnation, explosion or collapse - and whether our actions could change which of those happens. But today, it seems like things are far out of balance, with almost all news and analysis living in the Business As Usual headspace. > One metaphor for my headspace is that it feels as though the world is a set of people on a plane blasting down the runway. [...] We're going much faster than normal, and there isn't enough runway to do this much longer ... and we're accelerating. And every time I read commentary on what's going on in the world, people are discussing how to arrange your seatbelt as comfortably as possible given that wearing one is part of life, or saying how the best moments in life are sitting with your family and watching the white lines whooshing by, or arguing about whose fault it is that there's a background roar making it hard to hear each other. ## PASTA: Process for Automating Scientific and Technological Advancement What is my credence that PASTA is gonna happen: (a) ever (b) next 15 years (HK: >10%; PH: ) (c) by 2060 (HK: ~50%; PH: ) (D) by 2100 (HK: ~2/3; PH) How can we get a read on PASTA timelines? What can we say about what happens if/when we get PASTA? - Explosive growth - Lots of unpredictable effects from new technologies - Lock-ins - Trouble for biological humans Why not slower growth then plateau / stagnation? HK thinks unlikely but hasn't said why yet. ## Other > A fun book I recommend is Asimov's Chronology of Science and Discovery. It goes through the most important inventions and discoveries in human history, in chronological order. The first few entries include "stone tools," "fire," "religion" and "art"; the final pages include "Halley's comet" and "warm superconductivity." An interesting fact about this book is that 553 out of its 654 pages take place after the year 1500 - even though it starts in the year 4 million BC. I predict other books of this type will show a similar pattern,14 and I believe there were, in fact, more scientific and technological advances in the last ~500 years than the previous several million.15 --- # All Highlights ## Are we "trending toward" transformative AI? (How would we know?) https://www.cold-takes.com/are-we-trending-toward-transformative-ai-how-would-we-know/ *Highlight [page 1]:* In this post and the next, I will talk about the forecasting methods underlying my current view: I believe there's more than a 10% chance we'll see something PASTA-like enough to qualify as "transformative AI" within 15 years (by 2036); a ~50% chance we'll see it within 40 years (by 2060); and a ~2/3 chance we'll see it this century (by 2100). *Highlight [page 1]:* have a background view that something like PASTA is in a sense "inevitable," assuming continued advances in society and computing. The basic intuition here - which I could expand on if there's interest - is that human brains are numerous and don't seem to need particular rare materials to produce, so it should be possible at some point to synthetically replicate the key parts of their functionality. *Highlight [page 2]:* I'm not confident that PASTA will feel qualitatively as though it's "on the way" well before it arrives. (More on this below.) So I'm inclined to look for ways to estimate when we can expect this development, despite the challenges, and despite the fact that it doesn't feel today as though it's around the corner. I think there are plenty of example cases where a qualitatively unfamiliar future could be seen in advance by plotting the trend in some underlying, related facts about the world. A few that come to mind: When COVID-19 first emerged, a lot of people had trouble taking it seriously because it didn't feel as though we were "trending toward" or "headed for" a world full of overflowing hospitals, office and school closures, etc. At the time (say, January 2020), there were a relatively small number of cases, an even smaller number of deaths, and no qualitative sense of a global emergency. The only thing alarming about COVID-19, at first, was that case counts were growing at a fast exponential rate (though the overall number of cases was still small). But it was possible to extrapolate from the fast growth in case counts to a risk of a global emergency, and some people did. (And some didn't.) Climatologists forecast a global rise in temperatures that's significantly more than what we've seen over the past few decades, and could have major consequences far beyond what we're seeing today. They do this by forecasting trends in greenhouse gas emissions and extrapolating from there to temperature and consequences. If you simply tried to ask "How fast is the temperature rising?" or "Are hurricanes getting worse?", and based all your forecasts of the future on those, you probably wouldn't be forecasting the same kinds of extreme events around 2100. To give a more long-run example, we can project a date by which the sun will burn out, and conclude that the world will look very different by that date than it does now, even though there's no trend of things getting colder or darker today. *Highlight [page 2]:* An analogy for this sort of forecasting would be something like: "This water isn't bubbling, and there are no signs of bubbling, but the temperature has gone from 70° Fahrenheit6 to 150°, and if it hits 212°, the water will bubble *Highlight [page 2]:* If we're looking for some underlying factors in the world that predict when transformative AI is coming, perhaps the first thing we should look for is trends in how "impressive" or "capable" AI systems are. The easiest version of this would be if the world happened to shake out such that: One day, for the first time, an AI system managed to get a passing grade on a 4th-grade science exam. Then we saw the first AI passing (and then acing) a 5th grade exam, then 6th grade exam, etc. Then we saw the first AI earning a PhD, then the first AI writing a published paper, etc. all the way up to the first AI that could do Nobel-Prize-worthy science work. This all was spread out regularly over the decades, so we could clearly see the state of the art advancing from 4th grade to 5th grade to 6th grade, all the way up to "postdoc" and beyond. And all of this happened slowly and regularly enough that we could start putting a date on "full-blown scientist AI" several decades in advance. It would be very convenient - I almost want to say "polite" - of AI systems to advance in this manner. It would also be "polite" if AI advanced in the way that some people seem to casually imagine it will: first taking over jobs like "truck driver" and "assembly line worker," then jobs like "teacher" and "IT support," and then jobs like "doctor" and "lawyer," before progressing to "scientist." Either of these would give us plenty of lead time and a solid basis to project when science-automating AI is coming. Unfortunately, I don't think we can count on such a thing. AI seems to progress very differently from humans. For example, there were superhuman AI chess players7 long before there was AI that could reliably tell apart pictures of dogs and cats.8 One possibility is that AI systems will be capable of the hardest intellectual tasks insects can do, then of the hardest tasks mice and other small mammals can do, then monkeys, then humans - effectively matching the abilities of larger and larger brains. If this happened, we wouldn't necessarily see many signs of AI being able to e.g. do science until we were very close. Matching a 4th-grader might not happen until the very end. Another possibility is that AI systems will be able to do anything that a human can do within 1 second, then anything that a human can do within 10 seconds, etc. This could also be quite a confusing progression that makes it non-obvious how to forecast progress. *Highlight [page 2]:* Actually, if we didn't already know how humans tend to mature, we might find a child's progress to be pretty confusing and hard to extrapolate. Watching someone progress from birth to age 8 wouldn't necessarily give you any idea that they were, say, 1/3 of the way to being able to start a business, make an important original scientific discovery, etc. (Even knowing the usual course of human development, it's hard to tell from observing an 8-year-old what professional-level capabilities they could/will end up with in adulthood.) *Highlight [page 3]:* I think the best version of this exercise is Grace et al 2017, a survey of 352 AI researchers that included a question about “when unaided machines can accomplish every task better and more cheaply than human workers" (which would presumably include tasks that advance scientific and technological development, and hence would qualify as PASTA). The two big takeaways from this survey, according to Bio Anchors and me, are: A ~20% probability of this sort of AI by 2036; a ~50% probability by 2060; a ~70% probability by 2100. These match the figures I give in the introduction. 11 Much later estimates for slightly differently phrased questions (posed to a smaller subset of respondents), implying (to me) that the researchers simply weren't thinking very hard about the questions. ## Forecasting Transformative AI, Part 1: What Kind of AI? https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/ *Highlight [page 1]:* It's the single number I'd probably most value having a good estimate for: the year by which transformative AI will be developed.1 *Highlight [page 1]:* By "transformative AI," I mean "AI powerful enough to bring us into a new, qualitatively different future." The Industrial Revolution is the most recent example of a transformative event; others would include the Agricultural Revolution and the emergence of humans.2 *Highlight [page 1]:* particular kind of AI I believe could be transformative: AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement. I will call this sort of technology Process for Automating Scientific and Technological Advancement, or PASTA.3 (I mean PASTA to refer to either a single system or a collection of systems that can collectively do this sort of automation.) *Highlight [page 1]:* PASTA could resolve the same sort of bottleneck discussed in The Duplicator and This Can't Go On - the scarcity of human minds (or something that plays the same role in innovation). PASTA could therefore lead to explosive science, culminating in technologies as impactful as digital people. And depending on the details, PASTA systems could have objectives of their own, which could be dangerous for humanity and could matter a great deal for what sort of civilization ends up expanding through the galaxy. *Highlight [page 1]:* By talking about PASTA, I'm partly trying to get rid of some unnecessary baggage in the debate over "artificial general intelligence." I don't think we need artificial general intelligence in order for this century to be the most important in history. Something narrower - as PASTA might be - would be plenty for that. *Highlight [page 2]:* As with a human brain, we can mostly only guess at what the different parts of the "digital brain" are doing 9 (although there are some early attempts to do what one might call "digital neuroscience.") *Highlight [page 3]:* Like humans, PASTA systems might be good at getting the results they are under pressure to get. But like humans, they might learn along the way to think and do all sorts of other things, and it won't necessarily be obvious to the designers whether this is happening. Perhaps, due to being optimized for pushing forward scientific and technological advancement, PASTA systems will be in the habit of taking every opportunity to do so. This could mean that they would - given the opportunity - seek to fill the galaxy with long-lasting space settlements devoted to science. Perhaps PASTA will emerge as some byproduct of another objective. For example, perhaps humans will be trying to train systems to make money or amass power and resources, and setting them up to do scientific and technological advancement will just be part of that. In which case, perhaps PASTA systems will just end up as power-and-resources seekers, and will seek to bring the whole galaxy under their control. Or perhaps PASTA systems will end up with very weird, "random" objectives. Perhaps some PASTA system will observe that it "succeeds" (gets a positive training signal) whenever it does something that causes it to have direct control over an increased amount of electric power (since this is often a result of advancing technology and/or making money), and it will start directly aiming to increase its supply of electric power as much as possible - with the difference between these two objectives not being noticed until it becomes quite powerful. (Analogy: humans have been under selection pressure to pass their genes on, but many have ended up caring more about power, status, enjoyment, etc. than about genes.) *Highlight [page 3]:* you're interested in more discussion of whether an AI could or would have its own goals, I'd suggest checking out Superintelligence (book), The case for taking AI seriously as a threat to humanity (Vox article), Draft report on existential risk from power-seeking AI (Open Philanthropy analysis) or one of the many other pieces on this topic.10 *Highlight [page 3]:* It's hard to predict what a world with PASTA might look like, but two salient possibilities would be: PASTA could - by causing an explosion in the rate of scientific and technological advancement - lead quickly to something like digital people, and hence to the sorts of changes to the world described in Digital People Would Be An Even Bigger Deal. The next 3 posts will argue that PASTA is more likely than not to be developed this century. PASTA could lead to technology capable of wiping humans out of existence, such as devastating bioweapons or robot armies. This technology could be wielded by humans for their own purposes, or humans could be manipulated into using it to help PASTA pursue its own ends. ## Forecasting transformative AI: what's the burden of proof? https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/ *Highlight [page 1]:* in this piece, I'm going to address the question: how good do these forecasting methods need to be in order for us to take them seriously? In other words, what is the "burden of proof" for forecasting transformative AI timelines? *Highlight [page 1]:* When someone forecasts transformative AI in the 21st century - especially when they are clear about the full consequences it would bring - a common intuitive response is something like: "It's really out-there and wild to claim that transformative AI is coming this century. So your arguments had better be really good." I think this is a very reasonable first reaction to forecasts about transformative AI (and it matches my own initial reaction). But I've tried to examine what's driving the reaction and how it might be justified, and having done so, I ultimately don't agree with the reaction. I think there are a number of reasons to think that transformative AI - or something equally momentous - is somewhat likely this century, even before we examine details of AI research, AI progress, etc. I also think that on the kinds of multi-decade timelines I'm talking about, we should generally be quite open to very wacky, disruptive, even revolutionary changes. With this backdrop, I think that specific well-researched estimates of when transformative AI is coming can be credible, even if they involve a lot of guesswork and aren't rock-solid. *Highlight [page 2]:* Here are some things I believe about transformative AI, which I'll be trying to defend: I think there's more than a 10% chance we'll see something PASTA-like enough to qualify as "transformative AI" within 15 years (by 2036); a ~50% chance we'll see it within 40 years (by 2060); and a ~2/3 chance we'll see it this century (by 2100). Conditional on the above, I think there's at least a 50% chance that we'll soon afterward see a world run by digital people or misaligned AI or something else that would make it fair to say we have "transitioned to a state in which humans as we know them are no longer the main force in world events." (This corresponds to point #1 in my "most important century" definition in the roadmap.) And conditional on the above, I think there's at least a 50% chance that whatever is the main force in world events will be able to create a stable galaxy-wide civilization for billions of years to come. (This corresponds to point #2 in my "most important century" definition in the roadmap.) *Highlight [page 2]:* Often, someone states a view that I can't immediately find a concrete flaw in, but that I instinctively think is "just too wild" to be likely. For example, "My startup is going to be the next Google" or "College is going to be obsolete in 10 years" or "As President, I would bring both sides together rather than just being partisan." I hypothesize that the "This is too wild" reaction to statements like these can usually be formalized along the following lines: "Whatever your arguments for X being likely, there is some salient way of looking at things (often oversimplified, but relevant) that makes X look very unlikely." For the examples I just gave: "My startup is going to be the next Google." There are large numbers of startups (millions?), and the vast majority of them don't end up anything like Google. (Even when their founders think they will!) "College is going to be obsolete in 10 years." College has been very non-obsolete for hundreds of years. "As President, I would bring both sides together rather than just being partisan." This is a common thing for would-be US Presidents to say, but partisanship seems to have been getting worse for at least a couple of decades nonetheless. Each of these cases establishes a sort of starting point (or "prior" probability) and "burden of proof," and we can then consider further evidence that might overcome the burden. That is, we can ask things like: what makes this startup different from the many other startups that think they can be the next Google? What makes the coming decade different from all the previous decades that saw college stay important? What's different about this Presidential candidate from the last few? There are a number of different ways to think about the burden of proof for my claims above: a number of ways of getting a prior ("starting point") probability, that can then be updated by further evidence. Many of these capture different aspects of the "That's too wild" intuition, by generating prior probabilities that (at least initially) make the probabilities I've given look too high. *Highlight [page 2]:* Holden claims a 15-30% chance that this is the "most important century" in one sense or another.3 But there are a lot of centuries, and by definition most of them can't be the most important. Specifically: Humans have been around for 50,000 to ~5 million years, depending on how you define "humans."4 That's 500 to 50,000 centuries. If we assume that our future is about as long as our past, then there are 1,000 to 100,000 total centuries. So the prior (starting-point) probability for the "most important century" is 1/100,000 to 1/1,000. It's actually worse than that: Holden has talked about civilization lasting for billions of years. That's tens of millions of centuries, so the prior probability of "most important century" is less than 1/10,000,000. *Highlight [page 2]:* This argument feels like it is pretty close to capturing my biggest source of past hesitation about the "most important century" hypothesis. However, I think there are plenty of markers that this is not an average century, even before we consider specific arguments about AI. One key point is emphasized in my earlier post, All possible views about humanity's future are wild. If you think humans (or our descendants) have billions of years ahead of us, you should think that we are among the very earliest humans, which makes it much more plausible that our time is among the most important. (This point is also emphasized in Thoughts on whether we're living at the most influential time in history as well as the comments on an earlier version of "Are We Living at the Hinge of History?".) *Highlight [page 2]:* There are further reasons to think this particular century is unusual. For example, see This Can't Go On: The total size of the world economy has grown more in the last 2 centuries than in all of the rest of history combined. The current economic growth rate can't be sustained for more than another 80 centuries or so. (And as discussed below, if its past accelerating trend resumed, it would imply explosive growth and hitting the limits of what's possible this century.) It's plausible that science has advanced more in the last 5 centuries than in the rest of history combined. *Highlight [page 3]:* Report on Semi-informative Priors (abbreviated in this piece as "Semi-informative Priors") is an extensive attempt to forecast transformative AI timelines while using as little information about the specifics of AI as possible. So it is one way of providing an angle on the "burden of proof" - that is, establishing a prior (starting-point) set of probabilities for when transformative AI will be developed, before we look at the detailed evidence. The central information it uses is about how much effort has gone into developing AI so far. The basic idea: If we had been trying and failing at developing transformative AI for thousands of years, the odds of succeeding in the coming decades would be low. But if we've only been trying to develop AI systems for a few decades so far, this means the coming decades could contain a large fraction of all the effort that has ever been put in. The odds of developing it in that time are not all that low. One way of thinking about this is that before we look at the details of AI progress, we should be somewhat agnostic about whether developing transformative AI is relatively "easy" (can be done in a few decades) or "hard" (takes thousands of years). Since things are still early, the possibility that it's "easy" is still open. *Highlight [page 4]:* Conclusions. I'm not going to go heavily into the details of how the analysis works (see the blog post summarizing the report for more detail), but the report's conclusions include the following: It puts the probability of artificial general intelligence (AGI, which would include PASTA) by 2036 between 1-18%, with a best guess of 8%. It puts the probability of AGI by 2060 at around 3-25% (best guess ~13%), and the probability of AGI by 2100 at around 5-35%, best guess 20%. These are lower than the probabilities I give above, but not much lower. This implies that there isn't an enormous burden of proof when bringing in additional evidence about the specifics of AI investment and progress. *Highlight [page 4]:* Occasionally I'll hear someone say something along the lines of "We've been trying to build transformative AI for decades, and we haven't yet - why do you think the future will be different?" At a minimum, this report reinforces what I see as the common-sense position that a few decades of "no transformative AI yet, despite efforts to build it" doesn't do much to argue against the possibility that transformative AI will arrive in the next decade or few. In fact, in the scheme of things, we live extraordinarily close in time to the beginnings of attempts at AI development - another way in which our century is "special," such that we shouldn't be too surprised if it turns out to be the key one for AI development. *Highlight [page 4]:* Other angles on the burden of proof Here are some other possible ways of capturing the "That's too wild" reaction: "My cause is very important" claims. Many people - throughout the world today, and throughout history - claim or have claimed that whatever issue they're working on is hugely important, often that it could have global or even galaxy-wide stakes. Most of them have to be wrong. Here I think the key question is whether this claim is supported by better arguments, and/or more trustworthy people, than other "My cause is very important" claims. If you're this deep into reading about the "most important century" hypothesis, I think you're putting yourself in a good position to answer this question for yourself. Expert opinion will be covered extensively in future posts. For now, my main position is that the claims I'm making neither contradict a particular expert consensus, nor are supported by one. They are, rather, claims about topics that simply have no "field" of experts devoted to studying them. Some people might choose to ignore any claims that aren't actively supported by a robust expert consensus; but given the stakes, I don't think that is what we should be doing in this case. (That said, the best available survey of AI researchers has conclusions that seem broadly consistent with mine, as I'll discuss in the next post.) Uncaptured "That's too wild" reactions. I'm sure this piece hasn't captured every possible angle that could be underlying a "That's too wild" reaction. (Though not for lack of trying!) Some people will simply have irreducible intuitions that the claims in this series are too wild to take seriously. A general take on these angles. Something that bugs me about most of the angles in this section is that they seem too general. If you simply refuse (absent overwhelming evidence) to believe any claim that fits a "my cause is very important" pattern, or isn't already backed by a robust expert consensus, or simply sounds wild, that seems like a dangerous reasoning pattern. Presumably some people, sometimes, will live in the most important century; we should be suspicious of any reasoning patterns that would reliably11 make these people conclude that they don't. *Underline [page 4]:* Something that bugs me about most of the angles in this section is that they seem too general. If you simply refuse (absent overwhelming evidence) to believe any claim that fits a "my cause is very important" pattern, or isn't already backed by a robust expert consensus, or simply sounds wild, that seems like a dangerous reasoning pattern. Presumably some people, sometimes, will live in the most important century; we should be suspicious of any reasoning patterns that would reliably11 make these people conclude that they don't. ## Some additional detail on what I mean by "most important century" https://www.cold-takes.com/some-additional-detail-on-what-i-mean-by-most-important-century/ There are two different senses in which I think this could be the "most important century," one higher-stakes and less likely than the other: Meaning #1: Most important century of all time for humanity, due to the transition to a state in which humans as we know them are no longer the main force in world events. Meaning #2: Most important century of all time for all intelligent life in our galaxy. ## Give sports a chance https://www.cold-takes.com/give-sports-a-chance/ For someone who doesn't care about who wins, what do sports have to offer? High on my list is getting to closely observe people being incredibly (like world-outlier-level) intense about something. I am generally somewhat obsessed with obsession (I think it is a key ingredient in almost every case of someone accomplishing something remarkable). And with sports, you can easily identify which players are in the top-5 in the world at the incredibly competitive things they do; you can safely assume that their level of obsession and competitiveness is beyond what you'll ever be able to wrap your head around; and you can see them in action. ## Forecasting Transformative AI, Part 1: What Kind of AI? https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/ PASTA: Process for Automating Scientific and Technological Advancement. ## Why talk about 10,000 years from now? https://www.cold-takes.com/why-talk-about-10-000-years-from-now/ Sometimes, when I walk down the street, I just look around and think: "This is all SO WEIRD. Whooshing by me are a bunch of people calmly operating steel cars at 40 mph, and over there I see a bunch of people calmly operating a massive crane building a skyscraper, and up in the sky is a plane flying by ... and out of billions of years of life on Earth, it's only us - the humans of the last hundred-or-so years - who have ever been able to do any of this kind of stuff. Practically everything I look at is some crazy futurist technology we just came up with and haven't really had time to adapt to, and we won't have adapted before the next crazy thing comes along. "And everyone is being very humdrum about their cars and skyscrapers and planes, but this is not normal, this is not 'how it usually is,' this is not part of a plan or a well-established pattern, this is crazy and weird and short-lived, and it's anyone's guess where it's going next." I think many of us are instinctively, intuitively dismissive of wild claims about the future. I think we naturally imagine that there's more stability, solidness and hidden wisdom in "how things have been for generations" than there is. […] I sort of try to imagine myself as a billions-of-years-old observer, looking at charts like this and thinking things like "The current economic growth level just got started!" even though it got started several lifetimes ago. […] My main obsession is with effective altruism, or doing as much good as possible. I generally try to pay more attention to things when they "matter more," and I think things "matter more" when they affect larger numbers of persons.1 I think there will be a LOT more persons2 over the coming billions of years than over the coming generation or few. So I think the long-run future, in some sense, "matters more" than whatever happens over the next generation or few. Maybe it doesn't matter more for me and my loved ones, but it matters more from an "all persons matter equally" perspective.3 […] By trying to imagine the perspective of someone who's been alive for the whole story - billions of years, not tens - maybe we can be more open to strange future possibilities. And then, maybe we can be better at noticing the ones that actually might happen, and that our actions today might affect. So that's why I often try on the lens of saying things like "X has been going on for 200 years and could maybe last another few thousand - bah, that's the blink of an eye!" ## This can't go on https://www.cold-takes.com/this-cant-go-on/ We're used to the world economy growing a few percent per year. This has been the case for many generations. However, this is a very unusual situation. Zooming out to all of history, we see that growth has been accelerating; that it's near its historical high point; and that it's faster than it can be for all that much longer (there aren't enough atoms in the galaxy to sustain this rate of growth for even another 10,000 years). The world can't just keep growing at this rate indefinitely. We should be ready for other possibilities: stagnation (growth slows or ends), explosion (growth accelerates even more, before hitting its limits), and collapse (some disaster levels the economy)… […] The times we live in are unusual and unstable. We shouldn't be surprised if something wacky happens, like an explosion in economic and scientific progress, leading to technological maturity. In fact, such an explosion would arguably be right on trend. […] In Business As Usual, the world is constantly changing, and the change is noticeable, but it's not overwhelming or impossible to keep up with. There is a constant stream of new opportunities and new challenges, but if you want to take a few extra years to adapt to them while you mostly do things the way you were doing them before, you can usually (personally) get away with that. In terms of day-to-day life, 2019 was pretty similar to 2018, noticeably but not hugely different from 2010, and hugely but not crazily different from 1980.3 If this sounds right to you, and you're used to it, and you picture the future being like this as well, then you live in the Business As Usual headspace. When you think about the past and the future, you're probably thinking about something kind of like this: […] I live in a different headspace, one with a more turbulent past and a more uncertain future. I'll call it the This Can't Go On headspace. […] This growth has gone on for longer than any of us can remember, but that isn't very long in the scheme of things - just a couple hundred years, out of thousands of years of human civilization. It's a huge acceleration, and it can't go on all that much longer. (I'll flesh out "it can't go on all that much longer" below.) […] One possible future is stagnation: we'll reach the economy's "maximum size" and growth will essentially stop. We'll all be concerned with how to divide up the resources we have, and the days of a growing pie and a dynamic economy will be over forever. […] Another is explosion: growth will accelerate further, to the point where the world economy is doubling every year, or week, or hour. A Duplicator-like technology (such as digital people or, as I’ll discuss in future pieces, advanced AI) could drive growth like this. If this happens, everything will be changing far faster than humans can process it. Another is collapse: a global catastrophe will bring civilization to its knees, or wipe out humanity entirely, and we'll never reach today's level of growth again. […] It seems much more likely that we will "run out" of new scientific insights, technological innovations, and resources, and the regime of "getting richer by a few percent a year" will come to an end. After all, this regime is only a couple hundred years old. […] So one possible future is stagnation: growth gradually slows over time, and we eventually end up in a no-growth economy. But I don't think that's the most likely future. […] In addition to stagnation or explosive growth, there's a third possibility: collapse. A global catastrophe could cut civilization down to a state where it never regains today's level of growth. Human extinction would be an extreme version of such a collapse. This future isn't suggested by the charts, but we know it's possible. As Toby Ord’s The Precipice argues, asteroids and other "natural" risks don't seem likely to bring this about, but there are a few risks that seem serious and very hard to quantify: climate change, nuclear war (particularly nuclear winter), pandemics (particularly if advances in biology lead to nasty bioweapons), and risks from advanced AI. […] We live in one of the (two) fastest-growth centuries in all of history so far. (The 20th and 21st.) It seems likely that this will at least be one of the ~80 fastest-growing centuries of all time.13 If the right technology comes along and drives explosive growth, it could be the #1 fastest-growing century of all time - by a lot. If things go badly enough, it could be our last century. […] This is all based on pretty basic observations, not detailed reasoning about AI (which I will get to in future pieces). […] A fun book I recommend is Asimov's Chronology of Science and Discovery. It goes through the most important inventions and discoveries in human history, in chronological order. The first few entries include "stone tools," "fire," "religion" and "art"; the final pages include "Halley's comet" and "warm superconductivity." An interesting fact about this book is that 553 out of its 654 pages take place after the year 1500 - even though it starts in the year 4 million BC. I predict other books of this type will show a similar pattern,14 and I believe there were, in fact, more scientific and technological advances in the last ~500 years than the previous several million.15 […] I think there should be some people in the world who inhabit the Business As Usual headspace, thinking about how to make the world better if we basically assume a stable, regular background rate of economic growth for the foreseeable future. And some people should inhabit the This Can’t Go On headspace, thinking about the ramifications of stagnation, explosion or collapse - and whether our actions could change which of those happens. But today, it seems like things are far out of balance, with almost all news and analysis living in the Business As Usual headspace. […] One metaphor for my headspace is that it feels as though the world is a set of people on a plane blasting down the runway […] We're going much faster than normal, and there isn't enough runway to do this much longer ... and we're accelerating. And every time I read commentary on what's going on in the world, people are discussing how to arrange your seatbelt as comfortably as possible given that wearing one is part of life, or saying how the best moments in life are sitting with your family and watching the white lines whooshing by, or arguing about whose fault it is that there's a background roar making it hard to hear each other. ## Does X cause Y? An in-depth review. https://www.cold-takes.com/does-x-cause-y-an-in-depth-evidence-review/ ### Notes - If there are hundreds of useless papers, then this suggests that a lot of science is just fake-science. Not actually a truth-seeking endeavour, or at least a remarkably ineffective one. - Most of social science academia as hopeless LARP? Political contest? - Is he saying that nudge is all bullshit? That doesn't ring true to me, on priors. Maybe just that evidence is weak. - But engineers are great at x causes y, we have planes and cars and manufacturing. And physics and chemistry too. It seems like social science is just much harder than more physical sciences. Mainly because it's hard to do very large scale randomised-controlled experiements on people, social groups, societies? ### Highlights I think maybe X was enriched preschool, or just school itself, or eating fish while pregnant, or the Paleo diet, or lead exposure, or a clever "nudge" policy trying to get people to save more, or some self-help technique, or some micronutrient or public health intervention, or democracy, or free trade, or some approach to intellectual property law. And Y was ... lifetime earnings, or risk of ADHD diagnosis, or IQ in adulthood, or weight loss, or violent crime, or peaceful foreign policy, or GDP per capita, or innovation. Sorry about that! Hope you enjoy the post anyway! Fortunately, I think what I'm about to write is correct for pretty much any (X,Y) from those sorts of lists. […] - There are hundreds of studies on whether X causes Y, but most of them are simple observational studies that are just essentially saying "People/countries with more X also have more Y." For reasons discussed below, we can't really learn much from these studies. - There are 1-5 more interesting studies on whether X causes Y. Each study looks really clever, informative and rigorous at first glance. However, the more closely you look at them, the more confusing the picture gets. - We ultimately need to choose between (a) believing some overly complicated theory of the relationship between X and Y, which reconciles all of the wildly conflicting and often implausible things we're seeing in the studies; (b) more-or-less reverting to what we would've guessed about the relationship between X and Y in the absence of any research. […] Now, a lot of these studies try to "control for" the problem I just stated - they say things like "We examined the effect of X and Y, while controlling for Z [e.g., how wealthy or educated the people/countries/whatever are]." How do they do this? The short answer is, well, hm, jeez. Well you see, to simplify matters a bit, just try to imagine ... uh ... shit. Uh. #tweet […] I guess my bottom line is that X does cause Y, because it intuitively seems like it would. I'm glad I did all this research, though. It's good to know that social science research can go haywire in all kinds of strange ways. And it's good to know that despite the confident proclamations of pro- and anti-X people, it's legitimately just super unclear whether X causes Y. I mean, how else could I have learned that? […] There are cases where things seem a bit less ambiguous and the bottom line seems clearer. Speaking broadly, I think the main things that contribute to this are:  - Actual randomization. For years I've nodded along when people say "You shouldn't be dogmatic about randomization, there are many ways for a study to be informative," but each year I've become a bit more dogmatic. Even the most sophisticated-, appealing-seeming alternatives to randomization in studies seem to have a way of falling apart. Randomized studies almost always have problems and drawbacks too. But I’d rather have a randomized study with drawbacks than a non-randomized study with drawbacks. - Extreme thoroughness, such as Roodman's attempt to reconstruct the data and code for key studies in Reasonable Doubt. This sometimes leads to outright dismissing a number of studies, leaving a smaller, more consistent set remaining. ## Honesty about reading https://www.cold-takes.com/honesty-about-reading/ And complementarily, authors should try to make life easy for readers who do not want to carefully read every word of their piece (at least, assuming it is more than a couple thousand words or so). They should have easy-to-find sections of their piece that summarize and/or outline their arguments, with clear directions for which parts of the piece will give more detail on each point. They shouldn’t force or expect readers to wade through all their prose to find a TL;DR on what they are arguing, what their main evidence is, why it matters, and what their responses to key objections are. When someone says “You have to read this piece, it really shows that ___ ,” and I find myself unable to see where and how the piece shows ___ without embarking on a 10,000-word journey, I close the tab and forget about the argument, and this seems like the right thing to do ## First Post https://www.cold-takes.com/first-post/ A general theme of this blog is what I sometimes call avant-garde effective altruism. Effective altruism (EA) is the idea of doing as much good as possible. If EA were jazz, giving to effective charities working on global health would be Louis Armstrong - acclaimed and respected by all, and where most people start. But people who are really obsessed with jazz also tend to like stuff that (to other people) barely even sounds like music, and lifelong obsessive EAs are into causes and topics that are not the first association you'd have with "doing good." This blog will often be about the latter.