On the BBC's More or Less podcast (previously), Tim Harford and his team carefully unpick the numerical claims made by both sides in the UK/EU referendum debate. (more…)

]]>On the BBC's More or Less podcast (previously), Tim Harford and his team carefully unpick the numerical claims made by both sides in the UK/EU referendum debate. (more…)

]]>Lawyer-turned-data-scientist David Colarusso analyzed 2.2 million sentencing records from Virginia to determine the relationship between race, income and treatment in the criminal justice system. (more…)

]]>Lawyer-turned-data-scientist David Colarusso analyzed 2.2 million sentencing records from Virginia to determine the relationship between race, income and treatment in the criminal justice system. (more…)

]]>A new Pew Research report finds that the number of single adults still living with their parents is at historically high levels -- in the US, the number of singles still at home outnumber the cohort of those living out of the house, something last seen in the 1880s. (more…)

]]>A new Pew Research report finds that the number of single adults still living with their parents is at historically high levels -- in the US, the number of singles still at home outnumber the cohort of those living out of the house, something last seen in the 1880s. (more…)

]]>
In The Association Between Income and Life Expectancy in the United States, 2001-2014, published in the *Journal of the American Medical Association*, economists from Stanford, MIT and Harvard analyzed 1.4 million US tax records to see how income correlated with lifespan.
(more…)

In The Association Between Income and Life Expectancy in the United States, 2001-2014, published in the *Journal of the American Medical Association*, economists from Stanford, MIT and Harvard analyzed 1.4 million US tax records to see how income correlated with lifespan.
(more…)

]]>Extending on an analysis by the academic Kieran Healy, I calculated the rate of U.S. homicide deaths by racial group, based on the CDC WONDER data.3 From 2010 through 2012, the annual rate of homicide deaths among non-Hispanic white Americans was 2.5 per 100,000 persons, meaning that about one in every 40,000 white Americans is a homicide victim each year. By comparison, the rate of homicide deaths among non-Hispanic black Americans is 19.4 per 100,000 persons, or about 1 in 5,000 people per year. Black Americans are almost eight times as likely as white ones to be homicide victims, in other words.

]]>Extending on an analysis by the academic Kieran Healy, I calculated the rate of U.S. homicide deaths by racial group, based on the CDC WONDER data.3 From 2010 through 2012, the annual rate of homicide deaths among non-Hispanic white Americans was 2.5 per 100,000 persons, meaning that about one in every 40,000 white Americans is a homicide victim each year. By comparison, the rate of homicide deaths among non-Hispanic black Americans is 19.4 per 100,000 persons, or about 1 in 5,000 people per year. Black Americans are almost eight times as likely as white ones to be homicide victims, in other words.

Economist Tim Harford attacks three of the statistics being widely cited in the campaigns over the upcoming referendum on the UK remaining in the EU, two from the "leave" camp and one from the "stay" camp. (more…)

]]>Economist Tim Harford attacks three of the statistics being widely cited in the campaigns over the upcoming referendum on the UK remaining in the EU, two from the "leave" camp and one from the "stay" camp. (more…)

]]>Once again the MPAA has released its box-office numbers for the year, and once again, this year has smashed all records (as has been the case throughout our young century) (really!). As always, the astronomical rise-and-rise of their fortunes is somehow used to launch a call for more publicly subsidized enforcement against "piracy." (more…)

]]>Once again the MPAA has released its box-office numbers for the year, and once again, this year has smashed all records (as has been the case throughout our young century) (really!). As always, the astronomical rise-and-rise of their fortunes is somehow used to launch a call for more publicly subsidized enforcement against "piracy." (more…)

]]>Police violence is America is a statistical black hole, where data collection on shootings and killings are kept in haphazard or nonexistent form across local, state and federal levels, leaving scholars to piece together statistical pictures using techniques developed to reconstruct genocides from survivors' accounts. (more…)

]]>Police violence is America is a statistical black hole, where data collection on shootings and killings are kept in haphazard or nonexistent form across local, state and federal levels, leaving scholars to piece together statistical pictures using techniques developed to reconstruct genocides from survivors' accounts. (more…)

]]>Statistician Patrick Ball runs an NGO called the Human Rights Data Analysis Group, which uses extremely rigorous, well-documented statistical techniques to provide evidence of war crimes and genocides; HRDAG's work has been used in the official investigations of atrocities in Kosovo, Guatemala, Peru, Colombia, Syria and elsewhere. (more…)

]]>Statistician Patrick Ball runs an NGO called the Human Rights Data Analysis Group, which uses extremely rigorous, well-documented statistical techniques to provide evidence of war crimes and genocides; HRDAG's work has been used in the official investigations of atrocities in Kosovo, Guatemala, Peru, Colombia, Syria and elsewhere. (more…)

]]>Wonkblog runs the numbers on the counties with the strongest support for Trump and finds that the typical Trump supporter is likely to live in a place with higher-than-normal mortality for whites (middle-aged white mortality has been increasing since the 1990s at a rate unseen in the developed world since the collapse of the Soviet Union), lower-than-usual rates of university eduction, higher-than-normal rates of unemployment, where manufacturing jobs have vanished due to offshoring. (more…)

]]>Wonkblog runs the numbers on the counties with the strongest support for Trump and finds that the typical Trump supporter is likely to live in a place with higher-than-normal mortality for whites (middle-aged white mortality has been increasing since the 1990s at a rate unseen in the developed world since the collapse of the Soviet Union), lower-than-usual rates of university eduction, higher-than-normal rates of unemployment, where manufacturing jobs have vanished due to offshoring. (more…)

]]>If you've read Darell Huff's seminal 1954 book How to Lie With Statistics, you've learned an important rule of thumb: any chart whose Y-axis doesn't start at zero is cause for suspicion, if not alarm. (more…)

]]>If you've read Darell Huff's seminal 1954 book How to Lie With Statistics, you've learned an important rule of thumb: any chart whose Y-axis doesn't start at zero is cause for suspicion, if not alarm. (more…)

]]>Matthew Hankins catalogs 500 phrases used in scientific articles that researchers use to figleaf the fact that their results aren't statistically significant, and to hand-wave-away the fact that they're publishing anyway. (more…)

]]>Matthew Hankins catalogs 500 phrases used in scientific articles that researchers use to figleaf the fact that their results aren't statistically significant, and to hand-wave-away the fact that they're publishing anyway. (more…)

]]>The rash of high-profile journal retractions, revelations of systematic frauds in peer-review, and journals publishing deliberately bogus papers (e.g. "Get Me Off Your Fucking Mailing List") -- are we experiencing a crisis in science? (more…)]]>

The rash of high-profile journal retractions, revelations of systematic frauds in peer-review, and journals publishing deliberately bogus papers (e.g. "Get Me Off Your Fucking Mailing List") -- are we experiencing a crisis in science? (more…)]]>

Wichita State University's Beth Clarkson (who is also chief statistician of WSU's National Institute for Aviation Research) discovered "odd patterns" in Kansas electoral voting records, so she requested public docs to help her get to the bottom of things -- requests that state officials ignored, dodged, and stalled. (more…)]]>

Wichita State University's Beth Clarkson (who is also chief statistician of WSU's National Institute for Aviation Research) discovered "odd patterns" in Kansas electoral voting records, so she requested public docs to help her get to the bottom of things -- requests that state officials ignored, dodged, and stalled. (more…)]]>

Peer review and replication are critical to the scientific method, but in medical trials, a combination of pharma company intransigence and scientists' fear of being pilloried for human error means that the raw data that we base life-or-death decisions upon is routinely withheld, meaning that the errors lurk undetected in the data for years -- and sometimes forever. (more…)]]>

Peer review and replication are critical to the scientific method, but in medical trials, a combination of pharma company intransigence and scientists' fear of being pilloried for human error means that the raw data that we base life-or-death decisions upon is routinely withheld, meaning that the errors lurk undetected in the data for years -- and sometimes forever. (more…)]]>

The police will tell you that the reason they're arming up with surplus military gear and pursuing a shoot-first posture to their job is that being a cop is deadly business -- but as the saying goes, you're entitled to your own opinion, but not your own facts. (more…)]]>

The police will tell you that the reason they're arming up with surplus military gear and pursuing a shoot-first posture to their job is that being a cop is deadly business -- but as the saying goes, you're entitled to your own opinion, but not your own facts. (more…)]]>

It's not just the students: despite my own background in mathematics (I teach linear and abstract algebra), I sometimes find myself uncertain about advising my students about their data analysis and also in conflict with some colleagues about what counts as being statistically valid. Typically, I turn to statistical textbooks and other colleagues for advice.

An article in the April 16, 2015 edition of *Scientific American* boldly claimed that research psychologists are wringing their hands over the inadequacy of the statistical tools they have been using. It seems that the use of *p* values as gold standard tests for significance has gone into disrepute as a consequence of over-reliance and inadequacy in determining the quality of the results. This is where Alex Reinhart comes in.

Reinhart is a physicist turned statistician who has set out to write a book whose aim is to improve the quality of statistical education and understanding that researchers need to have. Statistics Done Wrong is not a textbook. It is a highly informed discussion of the frequent inadequacy of published statistical results and confronts the sacred cow: the *p* value. Here is what he has to say on page 2.

Since the 1980s, researchers have described numerous statistical fallacies and misconceptions in the popular peer-reviewed scientific literature and have found that many scientific papers -- perhaps more than half -- fall prey to these errors. Inadequate statistical power renders many studies incapable of finding what they're looking for, multiple comparisons and misinterpreted

pvalues cause numerous false positives, flexible data analysis makes it easy to find a correlation where none exists, and inappropriate model choices bias important results. Most errors go undetected by peer reviewers and editors, who often have no specific statistical training, because few journals employ statisticians to review submissions and few papers give sufficient statistical detail to be accurately evaluated.

Astonishing to my eyes was his conclusion that

The methodological complexity of modern research means that scientists without extensive statistical training may not be able to understand most published research in their fields.

Reinhart advises users of statistics to replace point estimates (*p* values) with confidence intervals (estimates of uncertainty). He discusses statistical power, (a way of determining the degree of confidence associated with statistical tests using the null hypothesis). He discusses and illustrates with clear and uncomplicated examples such things as the effects of sample size and reasonable estimates of bias (suggestive of the Bayesian approach).
(more…)

It's not just the students: despite my own background in mathematics (I teach linear and abstract algebra), I sometimes find myself uncertain about advising my students about their data analysis and also in conflict with some colleagues about what counts as being statistically valid. Typically, I turn to statistical textbooks and other colleagues for advice.

An article in the April 16, 2015 edition of *Scientific American* boldly claimed that research psychologists are wringing their hands over the inadequacy of the statistical tools they have been using. It seems that the use of *p* values as gold standard tests for significance has gone into disrepute as a consequence of over-reliance and inadequacy in determining the quality of the results. This is where Alex Reinhart comes in.

Reinhart is a physicist turned statistician who has set out to write a book whose aim is to improve the quality of statistical education and understanding that researchers need to have. Statistics Done Wrong is not a textbook. It is a highly informed discussion of the frequent inadequacy of published statistical results and confronts the sacred cow: the *p* value. Here is what he has to say on page 2.

Since the 1980s, researchers have described numerous statistical fallacies and misconceptions in the popular peer-reviewed scientific literature and have found that many scientific papers -- perhaps more than half -- fall prey to these errors. Inadequate statistical power renders many studies incapable of finding what they're looking for, multiple comparisons and misinterpreted

pvalues cause numerous false positives, flexible data analysis makes it easy to find a correlation where none exists, and inappropriate model choices bias important results. Most errors go undetected by peer reviewers and editors, who often have no specific statistical training, because few journals employ statisticians to review submissions and few papers give sufficient statistical detail to be accurately evaluated.

Astonishing to my eyes was his conclusion that

The methodological complexity of modern research means that scientists without extensive statistical training may not be able to understand most published research in their fields.

Reinhart advises users of statistics to replace point estimates (*p* values) with confidence intervals (estimates of uncertainty). He discusses statistical power, (a way of determining the degree of confidence associated with statistical tests using the null hypothesis). He discusses and illustrates with clear and uncomplicated examples such things as the effects of sample size and reasonable estimates of bias (suggestive of the Bayesian approach).
(more…)

In 2012, Jim Henley got tongue cancer, but it was the good kind -- his odds are like making a save-against-death throw on a D8 and needing to beat a one. (more…)]]>

In 2012, Jim Henley got tongue cancer, but it was the good kind -- his odds are like making a save-against-death throw on a D8 and needing to beat a one. (more…)]]>

Patrick Ball and the Human Rights Data Analysis Group applied the same statistical rigor that he uses in estimating the scale of atrocities and genocides for Truth and Reconciliation panels in countries like Syria and Guatemala to the problem of estimating killing by US cops, and came up with horrific conclusions. (more…)]]>

Patrick Ball and the Human Rights Data Analysis Group applied the same statistical rigor that he uses in estimating the scale of atrocities and genocides for Truth and Reconciliation panels in countries like Syria and Guatemala to the problem of estimating killing by US cops, and came up with horrific conclusions. (more…)]]>

The commonly cited "800,000" number is from a 2002 study of 1999 data. This information is widely misstated, and the data hasn't been updated in the era of ubiquitous mobile access and Amber Alerts. The National Incidence Studies of Missing, Abducted, Runaway, and Thrownaway Children (NISMART) surveyed about 16,000 adults, examined a subset of law-enforcement agencies (roughly 25% of the total), and a sampling of juvenile facilities, including detention and treatment centers. No comprehensive study or survey of this scope has since been conducted in America.

(more…)]]>The commonly cited "800,000" number is from a 2002 study of 1999 data. This information is widely misstated, and the data hasn't been updated in the era of ubiquitous mobile access and Amber Alerts. The National Incidence Studies of Missing, Abducted, Runaway, and Thrownaway Children (NISMART) surveyed about 16,000 adults, examined a subset of law-enforcement agencies (roughly 25% of the total), and a sampling of juvenile facilities, including detention and treatment centers. No comprehensive study or survey of this scope has since been conducted in America.

(more…)]]>“Overrated” and “underrated” are slippery terms to try to quantify. An interesting way of looking at this, I thought, would be to compare the reviews of film critics with those of Joe Public, reasoning that a film which is roundly-lauded by the Hollywood press but proved disappointing for the real audience would be “overrated” and vice versa.

To get some data for this I turned to the most prominent review aggregator: Rotten Tomatoes...

On the whole it should be noted that critics and audience agree most of the time, as shown by the Pearson correlation coefficient between the two scores (0.71 across >1200 films). [But] using our earlier definition it’s easy to build a table of those films where the audience ending up really liking a film that was panned by critics:

Here we’re looking at those films which the critics loved, but paying audiences were then less enthused:

Explore an interactive version of the chart at the top of this post here; and read more of Moore's methodology and findings here.]]>
*The Godfather* is the "best film" among the ~2600 films considered on Rotten Tomatoes, with a 100% score among professional reviewers and a 98% score from the audience. It is perhaps somewhat more surprising to learn which films divide those two groups; thanks to Benjamin Moore, we can contemplate that...

“Overrated” and “underrated” are slippery terms to try to quantify. An interesting way of looking at this, I thought, would be to compare the reviews of film critics with those of Joe Public, reasoning that a film which is roundly-lauded by the Hollywood press but proved disappointing for the real audience would be “overrated” and vice versa.

To get some data for this I turned to the most prominent review aggregator: Rotten Tomatoes...

On the whole it should be noted that critics and audience agree most of the time, as shown by the Pearson correlation coefficient between the two scores (0.71 across >1200 films). [But] using our earlier definition it’s easy to build a table of those films where the audience ending up really liking a film that was panned by critics:

Here we’re looking at those films which the critics loved, but paying audiences were then less enthused:

Explore an interactive version of the chart at the top of this post here; and read more of Moore's methodology and findings here.]]>

The latest installment in Randall Munroe's XKCD "What If?" series is called Paint the Earth and it is *amazing*. One of Munroe's readers wanted to know "Has humanity produced enough paint to cover the entire land area of the Earth?" and Munroe uses this as a springboard for explaining Fermi estimation, a powerful, counter-intuitive tool that has applications in many fields.
(more…)

The latest installment in Randall Munroe's XKCD "What If?" series is called Paint the Earth and it is *amazing*. One of Munroe's readers wanted to know "Has humanity produced enough paint to cover the entire land area of the Earth?" and Munroe uses this as a springboard for explaining Fermi estimation, a powerful, counter-intuitive tool that has applications in many fields.
(more…)

In Frequency, the latest XKCD cartoon, Randall Munroe has assembled a grid of animated GIFs representing various events in the universe, each keyed to blink in the frequency in which they occur in reality. As with the best of Munroe's work, it's a mix of the trenchant and the silly, and the juxtapositions are smart and provocative. There's real genius in putting "50,000 plastic bottles are produced" and "50,000 plastic bottles are recycled" next to each other, the former blinking much more often than the latter -- but the best part is "A Sagittarius named Amelia drinks a soda," just above them, mixing up the alarming and the humorous.

The other juxtapositions are just as delicious -- one birth/one death; China builds a car/Japan builds a car/Germany builds a car/US builds a car/someone else builds a car; someone buys "To Kill a Mockingbird"/someone's cat kills a mockingbird -- and so on. This being XKCD, you can be sure that Munroe has an absurdly well-thought-through process for establishing and documenting his numbers, too.

The tool-tip notes that he wanted to include pitch-drops in the chart, but "it turns out the gif format has some issues with decade-long loops."
Frequency
(*via IO9*)
]]>

In Frequency, the latest XKCD cartoon, Randall Munroe has assembled a grid of animated GIFs representing various events in the universe, each keyed to blink in the frequency in which they occur in reality. As with the best of Munroe's work, it's a mix of the trenchant and the silly, and the juxtapositions are smart and provocative. There's real genius in putting "50,000 plastic bottles are produced" and "50,000 plastic bottles are recycled" next to each other, the former blinking much more often than the latter -- but the best part is "A Sagittarius named Amelia drinks a soda," just above them, mixing up the alarming and the humorous.

The other juxtapositions are just as delicious -- one birth/one death; China builds a car/Japan builds a car/Germany builds a car/US builds a car/someone else builds a car; someone buys "To Kill a Mockingbird"/someone's cat kills a mockingbird -- and so on. This being XKCD, you can be sure that Munroe has an absurdly well-thought-through process for establishing and documenting his numbers, too.

The tool-tip notes that he wanted to include pitch-drops in the chart, but "it turns out the gif format has some issues with decade-long loops."
Frequency
(*via IO9*)
]]>

Catriona tumbled these enraging statistics about gender and representation in games and films for 2013:
(more…)

Catriona tumbled these enraging statistics about gender and representation in games and films for 2013: (more…)]]>

If you're the type of person who really needs some good visuals to make a concept stick in your head, this series of YouTube videos made by the British Psychological Society Media Centre will help you remember the meanings behind statistical concepts like "correlation", "frequency distributions", and "sampling error". There are four videos in the series so far, and they do a great job of painting pictures around abstract ideas. Bonus: Soothing music.

Via Openculture

]]>

If you're the type of person who really needs some good visuals to make a concept stick in your head, this series of YouTube videos made by the British Psychological Society Media Centre will help you remember the meanings behind statistical concepts like "correlation", "frequency distributions", and "sampling error". There are four videos in the series so far, and they do a great job of painting pictures around abstract ideas. Bonus: Soothing music.

Via Openculture

]]>“The numbers had floated around between 20,000 and 30,000 people killed and disappeared,” says Daniel Manrique-Vallier. “But nobody knew what the composition was. Non-governmental organizations were estimating that 90% of the deaths were the responsibility of state agents.”

Manrique-Vallier, a post-doc in the Duke University department of statistical science, was part of a team that researched the deaths for Peru’s Truth and Reconciliation Commission. Their results were completely different from those early estimates. Published in 2003, the final report presented evidence for nearly 70,000 deaths, 30% of which could be attributed to the Peruvian government.

How do you find 40,000 extra dead bodies? How do you even start to determine which groups killed which people at a time when everybody with a gun seemed to be shooting civilians? The answers lie in statistics, data analysis, and an ongoing effort to use math to cut through the fog of war.

(more…)]]>“The numbers had floated around between 20,000 and 30,000 people killed and disappeared,” says Daniel Manrique-Vallier. “But nobody knew what the composition was. Non-governmental organizations were estimating that 90% of the deaths were the responsibility of state agents.”

Manrique-Vallier, a post-doc in the Duke University department of statistical science, was part of a team that researched the deaths for Peru’s Truth and Reconciliation Commission. Their results were completely different from those early estimates. Published in 2003, the final report presented evidence for nearly 70,000 deaths, 30% of which could be attributed to the Peruvian government.

How do you find 40,000 extra dead bodies? How do you even start to determine which groups killed which people at a time when everybody with a gun seemed to be shooting civilians? The answers lie in statistics, data analysis, and an ongoing effort to use math to cut through the fog of war.

(more…)]]>Business Insider's Walter Hickey did the math on Monopoly, calculating the most frequently landed-up squares (taking into account dice probability, Go To Jail events, and Community Chest/Chance cards) and conceived of a supposedly optimal strategy for buying and building upon property. I still hate Monopoly, but I suspect that this would make it less boring (for a while, at least).

How To Use Math To Crush Your Friends At Monopoly Like You've Never Done Before
(*via MeFi*)
]]>

Business Insider's Walter Hickey did the math on Monopoly, calculating the most frequently landed-up squares (taking into account dice probability, Go To Jail events, and Community Chest/Chance cards) and conceived of a supposedly optimal strategy for buying and building upon property. I still hate Monopoly, but I suspect that this would make it less boring (for a while, at least).

How To Use Math To Crush Your Friends At Monopoly Like You've Never Done Before
(*via MeFi*)
]]>

As a woman, you do become less fertile as you get older, eventually culminating in menopause and the end of your potential babymaking years. But what does "less fertile" mean, and at what age, and how quickly does the drop-off in fertility happen?

According to this really fascinating piece by Jean Twenge at The Atlantic, some of the commonly cited scare stats — that one in three women ages 35 to 39 will not be pregnant after a year of trying, say — are based on extremely old data collected from historical birth records that don't necessarily reflect what's happening with real women who are alive right now. That statistic mentioned above, for instance, comes from French records (likely those collected by local church baptismal registries) for the years 1670 to 1830.

That matters because fertility is affected by things like quality of nutrition, infection rates, and even childhood illnesses — all of which have changed drastically for the average Western woman since the 19th century.

Look at more modern records, and the outlook for post-30 babymaking is completely different.

(more…)]]>As a woman, you do become less fertile as you get older, eventually culminating in menopause and the end of your potential babymaking years. But what does "less fertile" mean, and at what age, and how quickly does the drop-off in fertility happen?

According to this really fascinating piece by Jean Twenge at The Atlantic, some of the commonly cited scare stats — that one in three women ages 35 to 39 will not be pregnant after a year of trying, say — are based on extremely old data collected from historical birth records that don't necessarily reflect what's happening with real women who are alive right now. That statistic mentioned above, for instance, comes from French records (likely those collected by local church baptismal registries) for the years 1670 to 1830.

That matters because fertility is affected by things like quality of nutrition, infection rates, and even childhood illnesses — all of which have changed drastically for the average Western woman since the 19th century.

Look at more modern records, and the outlook for post-30 babymaking is completely different.

(more…)]]>