Chances are, you heard this month about thediscovery of a tiny fundamental physics particle that may be the long-sought Higgs boson. The phrase five-sigma was tossed about by scientists to describe the strength of the discovery. So, what does five-sigma mean?
In short, five-sigma corresponds to a p-value, or probability, of 3x10-7, or about 1 in 3.5 million. This is not the probability that the Higgs boson does or doesn't exist; rather, it is the probability that if the particle does not exist, the data that CERN scientists collected in Geneva, Switzerland, would be at least as extreme as what they observed. "The reason that it's so annoying is that people want to hear declarative statements, like 'The probability that there's a Higgs is 99.9 percent,' but the real statement has an 'if' in there. There's a conditional. There's no way to remove the conditional," says Kyle Cranmer, a physicist at New York University and member of the ATLAS team, one of the two groups that announced the new particle results in Geneva on July 4.
Scientists use p-values to test the likelihood of hypotheses. In an experiment comparing some phenomenon A to phenomenon B, researchers construct two hypotheses: that "A and B are not correlated," which is known as the null hypothesis, and that “A and B are correlated,” which is known as the research hypothesis.
The researchers then assume the null hypothesis (because it's the most conservative supposition, intellectually) and calculate the probability of obtaining data as extreme or more extreme than what they observed, given that there is no relationship between A and B. This calculation, which yields the p-value, can be based on any of several different statistical tests. If the p-value is low, for example 0.01, this means that there is only a small chance (one percent for p=0.01) that the data would have been observed by chance without the correlation. Usually there is a pre-established threshold in a field of study for rejecting the null hypothesis and claiming that A and B are correlated. Values of p=0.05 and p=0.01 are very common in many scientific disciplines.
High-energy physics requires even lower p-values to announce evidence or discoveries. The threshold for "evidence of a particle," corresponds to p=0.003, and the standard for "discovery" is p=0.0000003.
The reason for such stringent standards is that several three-sigma events have later turned out to be statistical anomalies, and physicists are loath to declare discovery and later find out that the result was just a blip. One factor is the "look elsewhere effect:" when analyzing very wide energy intervals, it is likely that you will see a statistically improbable event at some particular energy level. As a concrete example, there is just under a one percent chance of flipping an ordinary coin 100 times and getting at least 66 heads. But if a thousand people flip identical coins 100 times each, it becomes likely that a few people will get at least 66 heads each; one of those events on its own should not be interpreted as evidence that the coins were somehow rigged.
So where do the sigmas come in? The Greek letter sigma is used to represent standard deviation. Standard deviation measures the distribution of data points around a mean, or average, and can be thought of as how "wide" the distribution of points or values is. A sample with a high standard deviation is more spread out—it has more variability, and a sample with a low standard deviation clusters more tightly around the mean. For example, a plot of dogs' heights would probably have a larger standard deviation than a plot of heights of dogs from a particular breed, even if that breed had the same average height as dogs in general.
For particle physics, the sigma used is the standard deviation arising from a normal distribution of data, familiar to us as a bell curve. In a perfect bell curve, 68% of the data is within one standard deviation of the mean, 95% is within two, and so on.
