Correlation coefficients as angles

Harry Altman has an interesting piece over at Less Wrong suggesting that it makes more sense to think of correlation coefficients as angles. To many people a correlation coefficient is nothing more than a number between -1 and 1 which shows in some easy to quantify but hard to intuit way of how related two variables are. Harry's suggestion gives a useful way of thinking about them and is worth a read. 

Conservatism, racism, and stupidity

A recent study  argues that there is  a correlation between racism, political conservativism, and stupidity. More intriguingly, the study claims that this is not just a correlation but an actual causal link, with the third causing the other two.  The study is not at all persuasive.
Not surprisingly, this sort of thing produces a fair bit of discord. From the abstract:

 We proposed and tested mediation models in which lower cognitive ability predicts greater prejudice, an effect mediated through the endorsement of right-wing ideologies (social conservatism, right-wing authoritarianism) and low levels of contact with out-groups. In an analysis of two large-scale, nationally representative United Kingdom data sets (N = 15,874), we found that lower general intelligence (g) in childhood predicts greater racism in adulthood, and this effect was largely mediated via conservative ideology. A secondary analysis of a U.S. data set confirmed a predictive effect of poor abstract-reasoning skills on antihomosexual prejudice, a relation partially mediated by both authoritarianism and low levels of intergroup contact. All analyses controlled for education and socioeconomic status. Our results suggest that cognitive abilities play a critical, albeit underappreciated, role in prejudice. Consequently, we recommend a heightened focus on cognitive ability in research on prejudice and a better integration of cognitive ability into prejudice models.

While the claimed result is fairly moderate, bloggers on the left end of the political spectrum quickly jumped on the study, with one stating:

Wait a minute.  You’re saying the people who almost all believe a Canaanite Jew rose from the dead 2,000 years ago, who deny global warming, who deny evolution, the people who think losing in court is a better use of money than education, who think letting people love who they wish is destroying America, who think not educating children about sex prevents pregnancy or disease…the study concludes that those people tend to be dumb?

Many simiar comments  ignored the fact that the main part of the study took place in the United Kingdom while the traits mentioned above tend to be more highly associated with American conservatives. This isn't too surprising since few people actually linked to the actual study but apparently got their information from secondary news sources.

Commentators elsewhere on the political spectrum have not been much better, dismissing the study as more liberal propaganda rather than actually addresssing the questions that the study raises.
One of the most substantive responses came from William Briggs. Judging from Briggs's remarks he has over advantage over many of the commentators: he has apparently read the study in question. While most of Briggs's criticism is superficial, minor or unfair, he does have some valid points.

Briggs's first criticism is that only parts of the intelligence test data was used. He is correct to note this. The apparent implication is that the omitted data would if included undermine the claim in question.  While it might be interesting to look at that omitted data, there is a prosaic explanation for the exclusion of this data: The removed tests are for aspects of reading and math ability which are more function of education and background, and are not as useful indicators of actual cognitive ability.

Briggs also criticizes as badly phrased the questions used to measure social conservativism:

When the kids became 33 and 30 year olds, they were asked whether they agreed with 13 or 16 questions like, “Schools should teach children to obey authority”, “Family life suffers if mum is working full-time.”

Another was, “People who break the law should be rehabilitated.” Just kidding! It’s actually, “People who break the law should be given stiffer sentences.” The bias in the question wording is ignored.

There's such as a thing as the framing effect, where the  phrasing a question can influence how people will answer the question. However, in this particular case, neither question seems obviously more biased to me. Moreover, even if there is some form of  framing effect here, that should alter the overall percentage who answer one way or another, there's no reason it should impact whether conservatism and intelligence will be correlated. If any readers can construct a hypothesis about how framing would have a disparate impact, I'd be very interested in hearing it.

Briggs also complains that the intelligence measure used a latent variable, g, for which he felt a need to put in scare quotes.  While Briggs might be unhappy with such a variable, the existence of a general fluid intelligence is well documented, going back to Charles Spearman's work around a hundred years ago. Psychometricians have a good understanding of how g and similar variables behave. While the g here is not precisely identical to Spearman's g, the existence of such an underlying factor is essentially uncontroversial.

By far the best criticism that Briggs makes is to question whether the results are genuinely statistically significant. There are some subtleties with whether their p values are correct, and Briggs is correct to note potential issues there. However, the real problem is as Briggs observes, that the actual impact of intelligence on the issues in question is tiny.

What Briggs does not discuss in detail  is that the second part of the study, which analyzes  attitudes towards gays and intelligence levels in the US sample, showed a much stronger and more clear correlation between intelligence and attitude. The data for the attitudes towards gays isn't  original to this paper but comes from a previous paper by a different author which showed a correlated between intelligence and greater acceptance of gays. While this paper does try to construct a  causal model, the model is complicated and not convincing. However, the weak nature of their model doesn't reduce the fact that the prior paper's data is quite strong.

Despite all this, I think that the conclusion that, within the US self-identified liberals are on average smarter than self-identified conservatives is correct. That conclusion has nothing to do with these two papers but is based on other sources. The General Social Survey shows a strong correlation between larger vocabulary (which is highly correlated wit a variety of intelligence metrics) and liberalism. The actual pattern in the GSS data is actually a bit more subtle:  smarter people are in general more likely to have extreme political views than dumber people, and smarter people more likely to have liberal views. Essentially, as  intelligence increases, the distribution of political opininon becomes more bimodal and moves to the left.  This is part of a general pattern in the US that I've discussed  before - moderates are stupid and ignorant. The GSS isn't the only place one sees these twin patterns.

However, neither of these two trends is strong. Both these trends occur in large samples. They are true only on average. At an individual level, the differences are simply not that large. Moreover, the marginal popularity of a set of viewpoints among more intelligent individuals is not by itself a strong argument for their correctness. Some memes may be more appealing to intelligent people regardless of their validity, and memetic founder effects could easily cause ideas to be associated with certain populations.

A quick note on log odds

Probabilties are usually numbers that range from 0 to 1. However, the standard way of representing probabilities is not always optimal. However, there is another mapping of probability that goes from negative infinity to infinity. This system called "log odds" has a number of advantages. In the standard log odds approach, one maps the probability of an event x to the quantity log(x/(1-x)). Brian Lee and Jacob Sanders wrote a good summary (pdf) of this system which discusses its advantage and disadvantages. As they observe, use of log odds allows one to immediately see how something like the change in probability from 51% to 52% isn't that big whereas the change from 98% to 99% is a much larger change in the sense that the chance of the event not happening has now halved. Log odds helps makes this sort of intuition immediatelty obvious from the numbers. Brian and Jacob discuss the advantages and disadvantages of log odds in detail, and show how it is particularly useful for doing Bayesian updates. I strongly recommend reading their piece.

Occupy Wall Street, Boundaries, and Gratuitous Promotion of Family Members

My sister has a piece up at the Huffington Post discussing exactly how Occupy Wall Street lost her sympathy. She correctly points out actual problems with the Wall Street protesters, but I don't agree with what to her was the final point. She objects to the protesters deciding to protest the homes of the major executives, saying that they have a right to keep their private and public lives separate. I find this argument to be deeply unconvincing. When you are a major enough individual to be running a major corporation you have less of a right to privacy than a random individual. There might be an argument if these protests were directed at the homes of mid-level or upper level management. That argument doesn't apply to the CEOs of billion dollar corporations.

This is not a defense of the Occupy protesters in general. They aren't very coherent and those who have tried to state specific goals have given goals include goals that are unconsitutional, the immoral, the unethical, hopelessly naive, or just bad ideas. In that regard, they are essentially the left-wing equivalent of the Tea Party. It is possible that they will turn into something which does deal with the serious problems this country has, especially in regard to the massive income inequality which has become worse in the last few years but right now I'm not optimistic. The main thing that I would think needs to be done right now is getting the generic lower-middle class voter to understand that people like Herman Cain really have conflicting economic interests. There seems to be a certain class of economically badly off voters who somehow identify with the economic interests of people with incomes that are often an order of magnitude or more higher than their own.

As long as I'm pontificating about Occupy Wall Street, there are a few other things to note. First, whether or not one agrees with the protesters, the treatment of the protests by the police in some examples has been unacceptable. The mass arrest of protesters in Boston is a good example of this. Moreover, mistreating protesters is an easy way for people to build sympathy with a movement and come to agree with it whether or not the movement has any coherence or validity to their points. Second, using protesters behavior as evidence about economic policies is bad epistemology. This has lead to inane pieces like this one where various economic policies (some good, some bad) are justified simply by the existence of protesters. Protesters in this context are evidence of people unhappy with their current economic situation. Assuming that these people have any idea what to do about economic policy or that their existence can be easily traced to specific policies is unjustified.

In any event, my sister's piece is worth reading. She's not in the one percent, but she's not in the low percentages either. If OWS is going to succeed at anything they are going to need the people with average or moderately high incomes like my sister. Right now, they aren't doing that.

A brief note on non-transitive dice

I've talked before about non-transitive dice. We say that given a pair of dice X and Y, X beats Y if more than half the time when the pair is rolled X has a larger number face up than Y. It turns out one can construct dice A, B and C such that A beats B, B beats C, but C in fact beats A. This is a neat and weird property.

During a recent discussion I used non-transitive dice as an example of a counter-intuitive aspect of mathematics, I was pointed to an even weirder variant. Consider the following set of dice: A has sides (5,5,5,2,2,2), B has sides (4,4,4,4,4,1) and C has sides (6,3,3,3,3,3).

Here A beats B, B beats C and C beats A. But here's the really cool part: Let's say I roll two copies of A, two copies of B or two copies of C. Now things actually reverse! That is, a pair of Bs beats a pair of As and a pair of As beats a pair of Cs and a pair of Cs beats a pair of Bs.

This is a much more sensitive property than just non-transitive dice. Most sets of non-transitive dice will not have this property. We can also describe this sensitivity in a more rigorous fashion. Suppose we have a strictly increasing function f(x). That is, a function such that f(x) is greater than f(y) whenever x is greater than y. Now suppose we take a set of non-transitive dice and relable each value x with f(x). Then they will still be non-transitive. But, given a set of non-transitive, reversable dice, reversibility is not necessarily preserved by the f mapping. This reflects the much more sensitive nature of the reversible dice.

Here's a question I have so far been unable to answer: Is it possible to make a set of die which do an additional reversal? That is, is there a set of dices such rolling three copies the dice results in another reversal direction?

Voluntary Taxation and Gratuitous Promotion of Family Members

It looks like the last of my siblings has no entered the blogosphere. My sister Jacoba has a piece up at the Huffington Post discussing the reaction to Warren Buffet's statements that the rich are not being taxed enough. She points out that if people think that they aren't being taxed enough then they can always just right a check to the US Treasury.

Her point is an interesting one but I think it is misguided: Very few rich people think like Buffet does. Almost everyone who is in favor of higher taxes thinks that the group who should be paying higher taxes are the people with income slightly above their own income. Moreover, even if a large number of rich people agreed with Buffet, if those people gave much more of their money to the federal government while others in their income bracket did not those volunteers would suffer a relative loss of income. There's a fair bit of evidence that people's sense of status and wealth is a function of the people around them. So if most of the rich aren't paying as much it is quite understandable that the other rich would not want to. It is thus reasonable for some high income people to call for higher taxes even as they don't make voluntarily payments. In any event, the idea is an interesting one and the piece is worth reading.

The Fermi Paradox, The Great Filter, and Existential Risk

The Fermi Paradox is a classic puzzle proposed by Enrico Fermi. Fermi observed that if one made back of the envelope calculations of the sort for which Fermi was famous, then one would expect to see much intelligent life out in space. Moreover, it doesn't take a society much more advanced than our own before one is likely to see direct evidence of its existence. So where is everyone?

One proposal to explain this apparent paradox is Robin Hanson's explanation that there is some "Great Filter" which culls species before they can reach the degree of civilization necessary to spread out to the stars on a large scale. . Various roadblocks and events can act as filters. For example, severe asteroid impacts every few million years set life back. However, that seems to be a rare and weak filtration effect. One obvious roadblock is the arrival of life itself. Life arising may be much more difficult than we expect, and thus life may be comparatively rare. But, life arose fairly early in this planet's history, rendering this claim unlikely.

The most disturbing possibility, and the one on which both Robin Hanson and Nick Bostrom have focused, is the possibility that, for us, most of the filter lies not in our past, but in our future. This is scary. Events which result in the complete destruction of humanity are described as existential risk. If such events lie in our future, they are not likely due to natural causes such as asteroid impact and gamma ray bursts, since such events are rare. Existential risk to us is more likely the result of dangerous technologies. In a similar vein, during the Cold War, Carl Sagan worried that the apparent absence of life in the universe might be due to every advanced society having nuked itself. In a post Cold War world, that particular worry seems to be less severe. However, Hanson and others have focused on other technologies, especially those arising from nanotechnology and rogue AI.

I am not that worried by the Great Filter. I suspect that the vast majority of Great Filter is behind us. One of the most obvious filtration points are the steps from a species being smart to that species having civilization capable of making sustained technological progress. On Earth, there are many extremely smart species that are almost as smart as humans. Lots of people know that other primates are smart and will name dolphins and elephants as other very species. But there are many others as well, especially birds. Keas, African Grey Parrots, and ravens are only three of the many examples. Almost every species of corvid is extremely bright, and is capable of puzzle solving that rivals that of human children. However, the steps from there to sustained civilization are clearly large. Only a single species developed language, and even after that point, we stagnated for hundreds of thousands of years before developing writing, which is when things really started to take off. So, it seems to me that we can plausibly point to a large filtration step just before the development of civilization.

There are other points which have been proposed as filtration points in the development of life as well. One common argument is the Rare Earth Hypothesis which posits that the existence and success of life on Earth required a large variety of different conditions. For example, Earth has a large moon which helps protect the planet from asteroid strikes. For most of the features frequently cited as part of Earth's rare nature we don't seem to have enough data at this point to reasonably judge how common such features are or how necessary they are for complex life. However, even neglecting the Rare Earth filtration effects, the pre-civilization filtration still seems large.

Moreover, many of exotic anthropogenic events can be safely ruled out as major aspects of the Great Filter. The most plausible anthropogenic events are rogue AIs, false vacuum collapse, bad nanotech, and severe environmental damage with accompanying loss of natural resources.
Rogue AIs are an unlikely scenario because it is unlikely that any AI would be bad enough to wipe out the creating species and then not quickly take large scale control over much of their surrounding space.[1] Thus, if societies are being destroyed by rogue AIs we should be able to see this. Moreover, we should exect our own solar system to have long since come under sway of such AI. Thus, we can safely rule out rogue AI as a major part of the filter.

Similarly, some physicists have proposed that space as we know it is a "false vacuum". While the technical details are complicated, the essential worry is that a sufficiently advanced particle accelerator or similar device could cause space as we know it to be replaced by space that behaves fundamentally differently than what we are used to. The new space would expand at the speed of light.

We don't need to worry about civilizations probing the nature of space to cause a collapse of the false vacuum. If there were a lot of civilizations doing this, we wouldn't be here to notice. It is remotely plausible that the new vacuum would expand slower than the speed of light. If for example, the new type of vacuum expanded at a millionth of the speed of light, that would be enough to quickly destroy any single-planet civilization that triggered such an event, but would be slow enough to take a very long time to spread before it became noticed by another civilizations. However, our current understanding of the laws of physics make it hard to see how a vacuum collapse could occur at less than the speed of light. So we can rule this out as a major part of the Great Filter.

Nanotechnology is one of the most plausible options for a section of the Great Filter in front of us for the simple reason that severe nanotech events don't create results that will destroy or alter nearby stars or the like. While there are a variety of nanotech disaster scenarios, they essentially revolve around some form of out of control replicator consuming resources that humans need to survive or disrupting the ecosystem so much that we cannot survive. If a nearby solar system had a severe nanotech disaster, we wouldn't be able to tell. This situation is similar to Sagan's nuclear war scenario in that it allows civilizations to frequently wipe themselves out in a way that we can't easily observe.

Environmental damage and overconsumption of resources is another possible problem. It is possible that species exhaust their basic resources before they become technologically advanced. If, for example, humanity ran out of all fossil fuels without adequate replacements, this could prevent further expansion. However, this seems to be an unlikely explanation for Fermi's paradox. Even extreme resource consumption and environmental damage is unlikely to result in the complete destruction of an intelligent species. This possibility is the modern equivalent of the Sagan concern about nuclear war, a possibility which gets undue attention due to the current political climate.

So, it seems likely that most of the Great Filter is behind us. However, this is not a cause for complacency. First, the argument that the Great Filter is behind us is a weak one. As long as our sample of civilizations remains a single civilization, we cannot do more than make very rough estimates. Moreover, even if most of the Great Filter is behind us, that doesn't imply that we are necessarily paying enough attention to existential risk. Even back of the envelope calculations suggest that we aren't putting enough resources into dealing with existential risk threats, whether natural or caused by humans.

What needs to be done? First, we need to get a better idea where filtration steps actually exist. The most obvious way to do that is to look for life on other planets. If we don't find any life on other bodies in the solar system, then that increases the chance that a large part of the filtration is overcome by life arising and so we can breathe more easily. If however, we find life elsewhere, especially complex life, this gives us increased reason to think that the filter is ahead of us.
Second, we need to put more resources into dealing with existential risks. One excellent recent step was NASA's WISE mission which looked for asteroids likely to impact the Earth. We're now tracking a lot more of the near Earth asteroids and are probably tracking all of the asteroids that are both large and likely to intersect Earth orbit. At present, we're paying very little attention to human-caused catastrophic risk events. Catastrophic AI seems unlikely, but it is clear that little attention is being paid to the issue. Similar observations apply to nanotech and other concerns. More resources should be devoted to examining these dangers before the technologies become fully developed by which time it may be too late.

Unfortunately, there's a tendency to dismiss risks that appear in popular science fiction precisely because they appear in such works. This is just as bad as using fictional works as a reason to eschew a technology. Moreover, humans have a lot of trouble thinking about large scale problems, and the scale of a problem doesn't get much larger than the complete destruction of humanity.

So overall, the Great Filter doesn't worry me too much. But, even without the threat of the Great Filter, we still aren't doing enough to deal with the big risks to our existence. If most of the Great Filter is behind us, it would be all the more tragic if humanity were to be destroyed now, when we are but a few generations of spreading beyond our planet.



[1] I thought that this point might be original to me, but while writing this blog entry I found that it has been made before. See, e.g. Katja Grace's remarks here.