Archive for the ‘Statistics’ Category.

Judging the Tail

It’s easy to judge who is the fastest runner or swimmer. Judges do not need to be runners and swimmers themselves. They simply need a stopwatch and a camera.

Other competitions are more difficult to judge. Take for example the Fields medal. The judges need to be mathematicians. Since they can’t be experts in all the different areas of mathematics, they have to rely on recommendation letters. The mathematicians who write recommendation letters are biased, because they are interested in promoting their own field. The committee’s job is not simple, not the least because it involves a lot of politics. It is easy to award the medal to Grigory Perelman. He solved a high-profile long-standing conjecture. But other cases are not that straightforward.

Imagine a genius mathematician with a new vision. He or she might be so far ahead of everyone else, that the Fields committee would fail to appreciate the new concept. I wish the math community would create a list of mathematicians who deserved the Fields medal, but were passed over. As time goes by, perhaps a new Einstein will emerge on this list.

The reason the Fields committee more or less works is that the judges do not need to be as talented mathematicians as the awardees. They do not need to create mathematics, they need to understand it. And the latter is easier than the former.

A completely different story happens with IQ tests. Someone has to write those tests. There is no reason to think that writers of the IQ tests are anywhere close to the end tail of the IQ distribution. Hence, the IQ tests are not qualified to find the IQ geniuses.

IQ test

Now might be a good time to complain about the IQ test I took myself. Many years ago I tried an IQ test online through I was so disappointed with my non-perfect score that I never looked at my answers. Recently, while cleaning my apartment, I discovered the printout of the test. I made one mistake in the following question.

Which one of the designs is least like the other four?

The checkmark is the expected answer. They think that the circle is the odd one out because all the other shapes are polygons. The arrow points to my answer. I chose the right triangle because it is the only shape without symmetries. Who says that polygonality is more important than symmetry?


A Probabilistic Paradox

Tanya Khovanova and Alexey Radul

We all heard this paradoxical statement:

This statement is false.

Or a variation:

True or False: The correct answer to this question is ‘False’.

Recently we received a link to the following puzzle, which is similar to the statement above, but has a cute probabilistic twist:

If you choose an answer to this question at random, what is the chance you will be correct?

  1. 25%
  2. 50%
  3. 60%
  4. 25%

There are four answers, so you can choose a given answer with probability 25%. But oops, this answer appears twice. Is the correct answer 50%? No, it is not, because there is only one answer 50%. You can see that none of the answers are correct, hence, the answer to the question—the chance to be correct—is 0. Now is the time to introduce our new puzzle:

If you choose an answer to this question at random, what is the chance you will be correct?

  1. 25%
  2. 50%
  3. 0%
  4. 25%

Too Good at Spider Solitaire

Have you ever been punished for being too good at spider solitaire? I mean, have you ever been stuck because you collected too many suits? Many versions of the game don’t allow you to deal from the deck if you have empty columns, nor do they allow you to get back a completed suit. If the number of cards left on the table in the middle of the game is less than ten — the number of columns — you are stuck. I always wondered what the probability is of being stuck. This probability is difficult to calculate because it depends on your strategy. So I invented a boring version of spider solitaire for the sake of creating a math problem. Here it goes:

You start with two full decks of 104 cards. Initially you take 54 cards. At each turn you take all full suits out of your hand. If you have less than ten cards left in your hand, you are stuck. If not, take ten more cards from the leftover deck and continue. What is the probability that you can be stuck during this game?

Let us simplify the game even more by playing the easy level of the boring spider solitaire in which you have only spades. So you have a total of eight full suits of spades. I leave it to my readers to calculate the total probability of being stuck. Here I would like to estimate the easiest case: the probability of being stuck before the last deal.

There are ten cards left in the deck. For you to be stuck, they all should have a different value. The total number of ways to choose ten cards is 104 choose 10. To calculate the number of ways in which these ten cards have different values we need to choose these ten values in 13 choose 10 ways, then multiply by the number of ways each card of a given value can be taken from the deck: 810. The probability is about 0.0117655.

I will leave it to my readers to calculate the probability of being stuck before the last deal at the medium level: when you play two suits, hearts and spades.

No, I will not tell you how many times I played spider solitaire.


Averaging Averages

Jorge Tierno sent me a link to the following puzzle:

There is a certain country where everybody wants to have a son. Therefore, each couple keeps having children until they have a boy, then they stop. What fraction of the children are female?

If we assume that a boy is born with probability 1/2 and children do not die, then every birth will produce a boy with the same probability as a girl, so girls will comprise half of all children.

Now, I wonder why everyone would want a boy? Y-chromosomes are much shorter than X-chromosomes. If a man wants to pass his genes to the next generation, a daughter should be preferable as she keeps more genes from the father. I am a mother of two boys, so my granddaughters will have my X-chromosome while my grandsons will have my ex-husband’s Y-chromosome, so to keep my genes in the pool I should be more interested in granddaughters.

But I digress. I started writing this essay because in the original puzzle link the answer was different from mine. Here is how the other argument goes:

Half of all families have zero girls, a quarter have 1/2 girls, 1/8 have 2/3 girls, and so on. If we sum this up the expected ratio of girls to boys is (1/2)0 + (1/4)(1/2) + (1/8)(2/3) + (1/16)(3/4) + … which adds to 1 − ln 2, which is about 30%.

What’s wrong with this solution?


A Son Named Luigi

Suppose that we choose all families with two children, such that one of them is a son named Luigi. Given that the probability of a boy to be named Luigi is p, what is the probability that the other child is a son?

Here is a potential “solution.” Luigi is a younger brother’s name in one of the most popular video games: Super Mario Bros. Probably the parents loved the game and decided to name their first son Mario and the second Luigi. Hence, if one of the children is named Luigi, then he must be a younger son. The second child is certainly an older son named Mario. So, the answer is 1.

The solution above is not mathematical, but it reflects the fact that children’s names are highly correlated with each other.

Let’s try some mathematical models that describe how the parents might name their children and see what happens. It is common to assume that the names of siblings are chosen independently. In this case the first son (as well as the second son) will be named Luigi with probability p. Therefore, the answer to the puzzle above is (2-p)/(4-p).

The problem with this model is that there is a noticeable probability that the family has two sons, both named Luigi.

As parents usually want to give different names to their children, many researchers suggest the following naming model to avoid naming two children in the same family with the same name. A potential family picks a child’s name at random from a distribution list. Children are named independently of each other. Families in which two children are named the same are crossed out from the list of families.

There is a problem with this approach. When we cross out families we may disturb the balance in the family gender distributions. If we assume that boys’ and girls’ names are different then we will only cross out families with children of the same gender. Thus, the ratio of different-gender families to same-gender families will stop being 1/1. Moreover, it could happen that the number of boy-boy families will differ from the number of girl-girl families.

There are several ways to adjust the model. Suppose there is a probability distribution of names that is used for the first son. If another son is born, the name of the first son is crossed out from the distribution and following that we proportionately adjust the probabilities of all other names for this family. In this model the probability of naming the first son by some name and the second son by the same name changes. For example, the most popular name’s probability decreases with consecutive sons, while the least popular name’s probability increases.

I like this model, because I think that it reflects real life.

Here is another model, suggested by my son Alexey. Parents give names to their children independently of each other from a given distribution list. If they give the same name to both children the family is crossed-out and replaced with another family with children of the same genders. The advantage of this model is that the first child and the second child are named independently from each other with the same probability distribution. The disadvantage is that the probability distribution of names in the resulting set of families will be different from the probability distribution of names in the original preference list.

I would like my readers to comment on the models and how they change the answer to the original problem.


Mr. Jones

The following two problems appeared together in Martin Gardner’s Scientific American column in 1959.

Mr. Smith has two children. At least one of them is a boy. What is the probability that both children are boys?

Mr. Jones has two children. The older child is a girl. What is the probability that both children are girls?

Many people, including me and Martin Gardner, wrote a lot about Mr. Smith. In his original column Martin Gardner argued that the answer to the first problem is 1/3. Later he wrote a column titled “Probability and Ambiguity,” where he corrected himself about Mr. Smith.

… the answer depends on the procedure by which the information “at least one is a boy” is obtained.

This time I would like to ignore Mr. Smith, as I wrote a whole paper about him that is now under consideration for publication at the College Mathematics Journal. I would rather get back to Mr. Jones.

Mr. Jones failed to stir a controversy from the start and was forgotten. Olivier Leguay asked me about Mr. Jones in a private email, reminding me that the answer to the problem about his children also depends on the procedure.

One of the reasons Mr. Jones was forgotten is that for many natural procedures the answer is 1/2. For example, the following procedures will produce an answer of 1/2:

  • We ask Mr. Jones whether his older child is a daughter and he says “yes.”
  • Mr. Jones flips a coin deciding which child to talk about. After that he has to tell us the gender and whether the child is the oldest.
  • Mr. Jones is asked to say nothing if he doesn’t have a daughter, to select the daughter if has just one, or to pick one at random if he has two daughters. After that he has to tell us whether the daughter he has selected is the oldest.

There are many other procedures that lead to the answer 1/2. However, there are many procedures that lead to other answers.

Suppose I know Mr. Jones, and also know that he has two children. I meet Mr. Jones at a mall, and he tells me that he is buying a gift for his older daughter. Most probably I would assume that the other child is a daughter, too. In my experience, people who have a son and a daughter would say that they are buying a gift for “my daughter.” Only people with two daughters would bother to specify that they are buying a gift for “my older daughter.”

In some sense I didn’t forget about Mr. Jones. I wrote about him implicitly in my essay Two Coins Puzzle. His name was Carl and he had two coins instead of two children.


The Random Sequence

Fifteen years ago I attended Silvio Micali‘s cryptography course. During one of the lectures, he asked me to close my eyes. When I did, he wrote a random sequence of coin flips of length six on the board and invited me to guess it.

I am a teacher at heart, so I imagined a random sequence I would write for my students. Suppose I start with 0. I will not continue with zero, because 00 looks like a constant sequence, which is not random enough. So my next step would be sequence 01. For the next character I wouldn’t say zero, because 010 seems to promise a repetitive pattern 010101. So my next step would be 011. After that I do not want to say one, because I will have too many ones. So I would follow up with 0110. I need only two more characters. I do not want to end this with 11, because the result would be periodic, I do not want to end this with 00, because I would have too many zeroes. I do not want to end this with 01, because the sequence 011001 has a symmetry: reversing and negating this sequence produces the same sequence.

During the lecture all these considerations happened in the blink of an eye in my mind. I just said: 011010. I opened my eyes and saw that Micali had written HTTHTH on the board. He was not amused and may even have thought that I was cheating.

Many teachers, when writing a random sequence, do not flip a coin. They choose a sequence that looks “random”: it doesn’t have too many repetitions and the number of ones and zeroes is balanced (that is, approximately the same). When they write it character by character on the board, they often choose a sequence so that any prefix looks “random” too.

As a result, the sequence they choose stops being random. Actually, they’re making a choice from a small set of sequences that look “random”. So the fact that I guessed Micali’s sequence is not surprising at all.

If you have gone to many math classes, you’ve seen a lot of professors choosing very similar-looking “random” sequences. This discriminates against sequences that do not look “random”. To restore fairness to those under-represented sequences, I have decided that the next time I need a random sequence, I will choose 000000.


Women, Science and The Right Tail of a Bell Curve

by Rebecca Frankel

The article Daring to Discuss Women in Science by John Tierney in the New York Times on June 7, 2010 purports to present a dispassionate scientific defense of Larry Summers’s claims, in particular by reviewing and expanding his argument that observed differences in the length of the extreme right tail of the bell curves of men’s and women’s test scores indicate real differences in their innate ability. But in fact any argument like this has to acknowledge a serious difficulty: it is problematic to assume without comment that the abilities of a group can be inferred from the tail of a bell curve. We are so used to invoking bell curves to talk about group abilities, we don’t notice that such arguments usually use only the mean of the curve. Using the tail is a totally different story.

Think about it: it is reasonable to question whether a single data point — the test score of an individual person — is a true indication of his/her ability. It might not be. Maybe a single test score represents a dunce with hyper-overachieving parents who push him to study all the time. So does that single false reading destroy the validity of the curve? No of course not: because some other kid might have been a super-genius who was drunk last night and can barely keep his eyes open during the test. One is testing above his “true ability” and the other is testing below his “true ability,” and the effect cancels out. Thus the means of curves are a good way to measure the ability of large groups, because all the random false readings average out.

But tails are not. On the tail this “canceling out” effect doesn’t work. Look at the extreme right tail. The relatively slow but hyper-motivated kids are not canceled out by the hoard of far-above-the-mean super geniuses who had drunken revels the night before. There just aren’t that many super-geniuses and they just don’t party that much.

Or let’s look at it another way: imagine that you had a large group which you divided in half totally at random. At this point their bell curve of test scores looks exactly the same. Lets call one of the group “boys” and the other group “girls”. But they are two utterly randomly selected groups. Now lets inject the “boys” with a chemical that gives the ones who are very good already a burning desire to dominate any contest they enter into. And let us inject the “girls” with a chemical that makes the ones who are already good nonetheless unwilling to make anyone feel bad by making themselves look too good. What will happen to the two bell curves? Of course the upper tail of the “boys'” curve will stretch out, while the “girls'” tail will shrink in. It will look like the “boys” whipped the “girls” on the right tail of ability hands down, no contest. But the tail has nothing to do with ability. Remember they started out with the same distribution of abilities, before they got their injections. It is only the effect of the chemicals on motivation that makes it look like the “boys” beat the “girls” at the tail.

So, when you see different tails, you can’t automatically conclude that this is caused by difference in underlying innate ability. It is possible that other factors are at play — especially since if we were looking to identify these hypothetical chemicals we might find obvious candidates like “testosterone” and “estrogen”.

The possibility of alternative explanations for these findings calls into question Tierney and Summers’ claims to superior dispassionate scientific objectivity. Moving from the mean to the tail of a bell curve makes systematic effects on averages irrelevant, true, but it is instead susceptible to systematic effects on deviations, which are irrelevant at the mean. An argument that uses this trick to dodge gender differences in averages cannot claim the mantle of scientific responsibility without accounting for gender differences in deviations. I am deeply disappointed that Tierney and Summers did not accompany their assertions with a suitable reminder of this fact.


How to Live Longer

I just received a mass email on how to live longer and it made these points:

Tip 1. Delay your retirement. Studies show that people who retire at 65 live longer than people who retire at 60.

Tip 2. Sex makes you younger. Studies show that older people who have sex twice a week look ten years younger than their peers who do not have sex at all.

People who draw conclusions from such studies usually do not understand statistics. Correlation doesn’t mean causality. Let me use the above-mentioned studies to reach different conclusions by reversing the causality assumption of the unknown writer of the mass email. You can compare results and make your own decisions.

Case 1. Studies show that people who retire at 65 live longer than people who retire at 60. Reversed causality: People who live longer are healthier, so they are able to keep working and to retire later in life.

Case 2. Studies show that older people who have sex twice a week look ten years younger than their peers who do not have sex at all. Reversed causality: Older people who look ten years younger than their peers can get laid easier, so they have sex more often.


Shannon Entropy Rescues the Tuesday Child

My son Alexey Radul and I were discussing the Tuesday’s child puzzle:

You run into an old friend. He has two children, but you do not know their genders. He says, “I have a son born on a Tuesday.” What is the probability that his second child is also a son?

Here is a letter he wrote me on the subject. I liked it because unlike many other discussions, Alexey not only asserts that different interpretations of the conditions in the puzzle form different mathematical problems, but also measures how different they are.

by Alexey Radul

If you assume that boys and girls are symmetric, and that days of the week are symmetric (and if you have no information to the contrary, assuming anything else would be sheer presumption on your part), then you can be in one of at least two states.

1) You say that “at least one son born on a Tuesday” is all the information you have, in which case your distribution including this information is uniform over consistent cases, in which case your answer is 13/27 boy and your information entropy is

− ∑27 (1/27) log(1/27) = − log(1/27) = 3.2958.

2) You say that the information you have is “The guy might have said any true thing of the form ‘I have at least one {boy/girl} born on a {day of the week}’, and he said ‘boy’, ‘Tuesday’.” This is a different mathematical problem with a different solution. The solution: By a symmetry argument (see below [*]) we must assign uniform probability of him making any true statement in any particular situation. Then we proceed by Bayes’ Rule: the statement we heard is d, and for each possible collection of children h, the posterior is given by p(h|d) = p(h)p(d|h)/p(d). Here, p(h) = 1/142 = 1/196; p(d) = 1/14; and p(d|h) is either 1 or 1/2 according as whether his other child is or is not another boy also born on a Tuesday (or p(d|h) = 0 if neither child is a boy born on a Tuesday). There are 1 and 26 of these situations, respectively. The answer they lead to is of course 1/2; but the entropy is

− ∑ p log p = − 1/14 log 1/14 − 26/28 log 1/28 = 3.2827

Therefore that assumed additional structure really is more information, which is only present at best implicitly in the original problem. How much more information? The difference in entropies is 3.2958 – 3.2827 = 0.0131 nats (a nat is to a bit what the natural log is to the binary log). How much information is that? Well, the best I can do is to reproduce an argument of E.T. Jaynes’, which may or may not really apply to this situation. Suppose you have some repeatable experiment with some discrete set of possible outcomes, and suppose you assign probabilities to those outcomes. Then the number of ways those probabilities can be realized as frequencies counted over N trials is proportional to eNH, where H is the entropy of the distribution in nats. Which means that the ratio by which one distribution is easier to realize is approximately eN(H1-H2). In the case of N = 1000 and H1 – H2 = 0.0131, that’s circa 5×105. For each way to get a 1000-trial experiment to agree with version 2, there are half a million ways to get a 1000-trial experiment to agree with version 1. So that’s a nontrivial assumption.

[*] The symmetry argument: We are faced with the following probability assignment problem

Suppose our subject’s first child is a boy born on a Tuesday, and his second child is a girl born on a Friday. What probability must we assign to him asserting that he has at least one boy born on a Tuesday?

Good question. Let’s transform our coordinates: Let Tuesday’ be Friday, Friday’ be Tuesday, boy’ be girl, girl’ be boy, first’ be second and second’ be first. Then our problem becomes

Suppose our subject’s second’ child is a girl’ born on a Friday’, and his first’ child is a boy’ born on a Tuesday’. What probability must we assign to him asserting that he has at least one girl’ born on a Friday’?

Our transformation necessitates p(boy Tuesday) = p(girl’ Friday’), and likewise p(girl Friday) = p(boy’ Tuesday’). But our state of complete ignorance about what’s going on with respect to the man’s attitudes about boys, girls, Tuesdays, Fridays, and first and second children has the symmetry that question and question’ are the same question, and must, by the desideratum of consistency, have the same answer. Therefore p(boy Tuesday) = p(boy’ Tuesday’) = p(girl Friday) = 1/2.