Blog

Perfect Part 2: I’m one of 360,000!

Did I mention that I was in the stands for Felix Hernandez’s perfect game?

Oh, I did? Cool.

On Wednesday, I asked a question that my friend Jon had asked me soon after that 2012 game: How many people alive today have seen a perfect Major League Baseball game in person?

In Wednesday’s post, I outlined an approach to solving this seemingly-difficult problem. To repeat:

  1. We have the total attendance for each game. (It varies from the 6,298 people who saw Catfish Hunter’s perfect game in 1968 to the 64,519 who saw Don Larsen’s in the 1956 World Series.
  2. Assume that the percentage of people at each age and sex at the game was the same as the percentage of people at each age and sex in the U.S. as a whole. Again, this is almost certainly not true, but I’m not sure how to do better.
  3. All those people are older by the amount of years that have passed since the game. Someone who saw David Wells’s 1998 perfect game at age 48 would be 68 today.
  4. Assume that anyone whose age today turns out to be greater than 76 (for men) or 81 (for women) has gone to the big game in the sky.
  5. Add up the number of people still alive who saw each game to get the total alive across all games. That is the answer to Jon’s question.
  6. Additional step! This extra step was suggested by my other friend Ed, who asked “What about the weirdos that have seen more than one perfect game?” Are there cases where adding up the attendance would double-count some people, and if so, how do we account for that?

I closed Wendesday’s post asking how to get the data we would need for this approach. We can get attendance figures from the Wikipedia article on MLB perfect games, and I suggested that we should be able to find demographic data (age and sex) from the U.S. Census. Did anyone think of how to find that data? No one commented, but that doesn’t mean you weren’t thinking about it.

But to jump to the reveal:

I got the U.S. population by age and sex from 1900 to 2000 from Demographic Trends in the 20th Century (table 5 on page A-7), and for 2010 from 2010 Census Briefs: Age and Sex Composition. Both are free to download, and use for any purpose, by following the links.

Graph of male and female population in the U.S. census from 1900 to 2010.  Population scale goes from 0 to 180 million. The numbers stay equal until about 1950, then the number of women begins to exceed the number of men.
From the census data: number of men (blue) and women (orange) living in the U.S. in each decade from 1900 to 2010. Click for a larger version.

BONUS QUESTION: Notice how the number of men and women is about 50/50 until 1950, and after that, women exceed men. Why is that?

Of course, the U.S. census is only done every 10 years, and perfect games occurred in interim years like 1965 and 2004. So to get the age/sex populations in those interim years, I interpolated between the once-a-decade censuses by assuming a linear population growth, with slope (P2 – P1) / 10 for each of the categories. Note that this assumes that populations shifted within categories only every 10 years, which is not a good assumption but is good enough for this quick analysis. For games in 2011 and 2012, I carried the 2000-2010 growth rate forward another two years.

The census doesn’t publish population counts for every age; instead, they report a single count for all ages within five-year intervals (bins), starting with “0-5” and ending with “85+”. Within each age bin, they report the number of men and the number of women. They also give the total number of men and women within overall, and the U.S. bottom-line population total.

From these numbers, I calculated the percentage of people in each age/sex bin in years in which perfect games took place. I then multiplied this number by the attendance of each perfect game to find the number of people expected in each age/sex bin (as predicted by our simple model in which spectators at a baseball game is a representative sample of the U.S. population).

Then I subtracted the number of years between that perfect game and today, and added that number to each of the bins. That shows us how old that game’s crowd would be today. I added up the counts only the bins for people who are today 76 or younger (for men) or 81 or younger (for women).

So, calculate the percentage of people in each age/sex bin in each census. Multiply that percentage by the total number of people at each game to figure out how many members of each age/sex bin there were at the game. Increase the age of the population to today, and remove any bins that would be greater than 76/81 today. For example, Don Larsen’s perfect game took place 62 years ago, so I added up population counts for men who were then younger than (76 – 62 =) 14. That includes everyone from the age bins 0-4, 5-9, and 10-14. From a similar analysis of women, I added up everyone from age bins 0-4, 5-9, 10-14, 15-19. (In some cases the years overlapped bins – for example, Kenny Rogers’s perfect game in 1994 returns women 57 or younger, which overlaps the 55-59 age bin. So I added 2/5 of the count from that age bin.)

Adding up all remaining age bins leaves the number of people who attended each perfect game who are probably still alive in 2018. For example, of the 64,519 people who attended Don Larsen’s perfect game (it was in the World Series!), an estimated 21,422 are still alive today. We’ve made so many assumptions that I wouldn’t trust that number to be anywhere near exact, so let’s report it as “about 21,000”.

Doing this for all perfect games and then adding up for the total gives 363,000. But then, there’s Ed’s question: were there some people who saw multiple perfect games, meaning that we’ve double-counted some people? Do we need to adjust our estimate down to make up for it?

If this were 2011, I’d say no. Before then, perfect games had happened in different cities in different years. But then in 2012, Philip Humber’s and Felix Hernandez’s perfect games took place at Safeco Field in Seattle. Thus, a Mariners 2012 season ticket holder would have seen metaphorical lightning strike twice – two perfect games in one stadium in one season.

So to account for this effect, we need to estimate the percentage of the crowd at a Major League Baseball game that are season ticket holders who attend every game (or at least attended two games, and were incredibly lucky in choosing the two games they attended). I have no idea how to estimate that percentage. But consider that we know for sure two people that attended Hernandez’s perfect game but not Humber’s – me and my lovely spouse. And I know several Baltimore Orioles season ticket holders, none of them have attended every single game this season. So, let’s pick an estimate that I think is on the high side of reasonable: 10 percent.

That means that to account for the double-counting, we need to subtract about 10% of the crowd for Hernandez’s perfect game because we suspect they had already seen Humber’s perfect game four months before. Ten percent of 21,889 is (rounding off to remind ourselves this is just an estimate) about 2,000. So subtract out 2,000 people from our preliminary estimate of 363,000 to leave 361,000 people who have seen one or more perfect games.

Perfect Game number Pitcher Date

People who saw the game People who saw the game who are alive today
1 Lee Richmond 6/12/1880 unknown 0
2 John Montgomery Ward 6/17/1880 unknown 0
3 Cy Young 5/5/1904 10,267 0
4 Addie Joss 10/2/1908 10,598 0
5 Charlie Robertson 4/30/1922 25,000 0
6 Don Larsen 10/8/1956 64,519 21,000
7 Jim Bunning 6/21/1964 32,026 14,000
8 Sandy Koufax 9/9/1965 29,139 13,000
9 Catfish Hunter 5/8/1968 6,298 3,000
10 Len Barker 3/15/1981 7,290 4,000
11 Mike Witt 9/30/1984 8,375 5,000
12 Tom Browning 9/16/1988 16,591 11,000
13 Dennis Martinez 7/28/1991 45,560 33,000
14 Kenny Rogers 7/28/1994 46,581 34,000
15 David Wells 5/17/1998 49,820 38,000
16 David Cone 7/18/1999 41,930 33,000
17 Randy Johnson 5/18/2004 23,381 19,000
18 Mark Buerhrle 7/23/2009 28,036 24,000
19 Dallas Braden 5/9/2010 12,288 11,000
20 Roy Halladay 5/29/2010 25,086 22,000
21 Philip Humber 4/21/2012 22,472 20,000
22 Matt Cain 6/13/2012 42,298 38,000
23 Felix Hernandez 8/15/2012 21,889 20,000
  (remove double-counts from Seattle 2012)   -2,000 -2,000
  TOTAL   567,444 361,000

Again, there are a lot of potential sources of systematic error in this analysis, so I don’t think we can be confident enough in our estimates to go down to the level of 1,000 people either way. So again, let’s round to 360,000.

Thus, our estimate shows that about 360,000 people alive today have seen a Major League Baseball perfect game. And I am one of them.

This has been a quick analysis of a small, self-contained question, but it showcases many features of the thought process that data scientists go through each day. I hope it’s been fun to follow along. The most important part is to always keep in the back of your mind: How might I be wrong?

With that in mind, then: how might this analysis be wrong? What assumptions did we make that we should not have made? What can we do to improve our estimates?

I’d love to hear your thoughts on this, no matter what your experience with math and science have been. The entire point of this blog is to bring the excitement of science to everyone.

Don’t make Eeyore sad, comment below with your thoughts!

Dank meme: A picture of Eeyore with the caption "No one commented"

Perfect Part 1: How many people alive today have seen a perfect MLB game?

Me in the stands at Safeco Field for Felix Hernandez's perfect game on August 15, 2012Today marks the sixth anniversary of a baseball accomplishment: the most recent perfect game in Major League Baseball history. Since 1876, Major League Baseball teams have played 216,449 games, and only 23 have ended in perfect games (and here is a list of all of them). And I was there for the most recent one.

Which was nice.

A “perfect game” is when a pitcher goes an entire game without allowing any hits, walks, or errors – and thus no one from the opposing team even reaches first base. This photo shows me looking gleeful at the conclusion of the one six years ago. My lovely spouse and I were on vacation in Seattle, and decided to go watch a Seattle Mariners game. Or rather, I wanted to watch the game, so my lovely spouse came along and brought a book to read. (Did I mention my spouse is lovely?)

Someday the baseball gods will extract their terrible revenge on me for all the gloating I have done in the last six years. But the reason I tell the story today is not to further gloat (although: did I mention I saw a perfect game???), but rather to use it as an opportunity to answer a question that my friend Jon posed when he first heard the story:

How many people alive today have seen a perfect game?

The scoreboard at the end of the game reads: "Felix Hernandez, first perfect game in Mariners history"
Did I mention I saw Felix Hernandez’s perfect game six years ago? Which was nice.

So what approach can we take to find an answer to Jon’s question? The question can be divided into two parts: how many people have seen a perfect game, and how many of those people are alive today?

The first part of that question is straightforward, mostly. All MLB games today have their attendance recorded as part of their official scoring, and nearly all baseball scorecards ever are available from the Baseball Reference website. For example, the box score from perfect game #4 – Addie Joss’s in Cleveland on October 2, 1908 – shows that there were 10,598 people in attendance. There are no attendance figures for the first two perfect games, which were both pitched in 1880, but there are for the other 21.

I looked up the attendance from the box scores for each game of the 21 for which figures are available. The total attendance is 570,144. That means that, excluding the first two games in the 1800s (for which the attendance is unknown), 570,144 people have seen an MLB perfect game in person.

So now for the more interesting part: how many of those people are alive today? There’s no way to know for sure, of course – at least not without making the ridiculous effort to track down every person who was there. But we can get a decent estimate based on population and lifespan data. (Note: this is why we don’t need to worry about attendance those first two games – no one born in the 1800s is alive today).

The approach is this: make a guess of who would have been at each game. How many men and how many women? How many people of each age – children, seniors, etc.? Then follow those people into their futures, using population statistics to figure out how many of them are still alive in 2018.

For example: when I saw Felix Hernandez’s perfect game in 2012, I was a 34-year-old male. Now I am a 40-year-old male. But if the perfect game I had seen was instead Cy Young’s in 1904, I would today have been a 148-year-old male, and there are no 148-year-old males alive.

The scorecard for the Tampa Bay Rays in that game: 27 batters, 27 out
My scorecard for the game: F. Hernandez (SEA): 27 batters faced, 27 outs

This approach requires a major starting assumption: what are the demographic characteristics of a Major League Baseball audience? As a starting point, let’s assume that the stadium audience for these perfect games was representative of the whole U.S. population at the time. That is almost certainly not true, but it’s an easy starting point. Toward the end of this post, I give some thoughts on how you might improve estimates.

Next, assuming that the people at each game were representative of the U.S. population, let’s “age the population,” by applying lifespan data for this distribution. The average lifespan in the U.S. today is 76 for men and 81 for women. But, of course, some people live longer than that, and some less long. At first, I thought I would need to calculate simulated lifespans for everyone who might have been in the stadium. For example, if we wanted to see how many people alive today saw Charlie Robertson’s perfect game in 1922, we would need to estimate the likelihood that someone who was there – say a one-year-old baby, is still alive at age 97. But then I realized – no need!

If the research question were about that 1922 game specifically, then we would indeed have to use such a detailed statistical approach. But the research question is about how many people have seen ANY Major League Baseball perfect game. This is where the math gets slightly depressing, sorry, but if that now-97-year-old baby is still alive, s/he is balanced out by someone who saw one of the later games and died young. So we can just use the average lifespan for men and women as an estimate of the probability that someone is still alive – if they would today be older than 76-for-men-81-for-women, we can assume they are no longer with us. That also means that we don’t need to consider games for which the youngest person who might have been there is now older than 81. So we can start with Don Larsen’s perfect game in 1956.

So here is the approach we will use:

  1. We have the total attendance for each game. (It varies from the 6,298 people who saw Catfish Hunter’s perfect game in 1968 to the 64,519 who saw Larsen’s in the 1956 World Series.
  2. Assume that the percentage of people at each age and sex at the game was the same as the percentage of people at each age and sex in the U.S. as a whole. Again, this is almost certainly not true, but I’m not sure how to do better.
  3. All those people are older by the amount of years that have passed since the game. Someone who saw David Wells’s 1998 perfect game at age 48 would be 68 today.
  4. Assume that anyone whose age today turns out to be greater than 76 (for men) or 81 (for women) has gone to the big game in the sky.
  5. Add up the number of people still alive who saw each game to get the total alive across all games. That is the answer to Jon’s question.

So the only additional data we need, besides the attendance for each game, is the percentage of people at each age and sex in the U.S. population at the time each game occurred. We can multiply the percentages of each age and sex by the total number at each game to find the number of people at each age and sex at each game. Data for age and sex for the entire history of the U.S. is easily obtainable from the U.S. Census at www.census.gov.

So what’s the answer?

Tune in Friday to find out!

Except they weren’t: Joe Magarac

A statue of Joe Magarac bending a steel beamI’m fascinated with stories of people who are not what they seem, or are not what they claim to be. History is full of stories of people who hide or change their identities, for all sorts of reasons both good and not-so-good — as well as stories of fictional people who were invented for all sorts of reasons both good and not-so-good. But sometimes the story is so much more interesting, and I’d love to share some of these stories with you. If truth is stranger than fiction, then true fiction is even stranger than that. Welcome to a new occasional series: Except They Weren’t.

What John Henry was to African-American rail workers, Joe Magarac was to steelworkers in Pittsburgh.

In 1931, Pittsburgh author Owen Francis (a former steelworker himself) interviewed a group of immigrant steelworkers from Croatia. They told him of the legend of Joe Magarac, a strong and hardworking folk hero figure who shows up just in time so save his fellow workers from calamity.

Francis wrote up the story for the November 1931 issue of Scribner’s magazine as “The Saga of Joe Magarac: Steelman.” Like the more famous John Henry, Joe Magarac might not have been a real person, but he was a real inspiration to the downtrodden workers of America.

Except he wasn’t.

There is no historical record of Joe Magarac before the 1931 article, and when anthropologists interviewed Pittsburgh immigrant steelworkers in the 1940s and 1950s, no one had heard of him. We’ll never know for sure, but it sure looks like the guys were just playing a fun game of troll the reporter. The clincher? Check out what “magarac” means in Croatian.

A photo of a cute donkey grazingNot Joe Magarac

But in one of those Bizarre Things That Happen Sometimes, the article proved so popular that Joe Magarac actually became a local folk hero, and an unofficial symbol of the city of Pittsburgh. Here he is, hard at work on a mural of a downtown building. And the photo above is of his statue in front of U.S. Steel’s headquarters.

Joe Magarac is an example of what historians and anthropologists call “fakelore” — stories that have the structure and purpose of folklore, but were created for another purpose. What other examples of fakelore do you know?

Image credit for the statue photo: Flickr user Devon Christopher Adams

Grandpa vs. Nazis

A photo of Mike Raddick, Sr. in front of a car wearing a U.S. Army uniform
Detroit, January 1942: On furlough after training

I recently learned a cool story about my grandfather, Michael Raddick, Sr.

In late 1941, like many young men of his generation, he volunteered for the U.S. Army. At age 24 and married, he had never left the American Midwest, and suddenly he was on a train to Camp Beauregard in Louisiana for Basic training. The cool story comes at the end of Basic.

The Sergeant addressed the company and asked who could operate construction equipment. Grandpa raised his hand. Sure, he’d never actually operated construction equipment, but he’d driven cars, and it couldn’t be that different. Right?

Thus he became a member of the Army Corps of Engineers. Suddenly the young man who had never left the Midwest was shipped off to Iran, where he worked on the Trans-Iranian Railway (still in use today) to supply the Soviet Union in its fight against the Nazis. He worked there for three years, met the Shah, and was honorably discharged at the end of the war as a Technician fifth grade (TEC 5), at the time the Engineers’ equivalent of a Corporal. You can read about the operation here at the National Museum of the U.S. Army and at the U.S. Army Corps of Engineers websites.

He returned home and settled in Neville Township, PA, where he became a principal and community leader, and had two children – Aunt Elaine and my Dad. He died in 2005 at age 88, and is buried in Salem, Ohio next to his beloved wife (my grandmother), who died 12 years later.

Headstone reading: Michael Raddick, TEC 5 US Army, World War II, Aug 11 1917 - Sep 24 2005
Rest in peace, Grandpa. You earned it.

This story illustrates what has become something of a trademark strategy in the Raddick family: volunteer for something you are not technically “qualified” for because it sounds cool, then learn fast.

That was how my Dad became a salesman in the lumber industry, and went on to start a successful cabinet supply business.

It’s also how I became a writer/educator/data scientist.

Kill moose and squirrel: Russians pretend to be Americans online

Boris, Natasha, and Fearless Leader from an episode of The Adventures of Rocky and Bullwinkle and Friends“Justin” set his Twitter location as Austin, Texas, but his time zone was set to Moscow Standard Time.

When a Smart Data Science Friend (hi Scott!) shared this in October 2017, I knew that organizations in Russia had mounted used social media to support Donald Trump’s presidential campaign, but I hadn’t realized the scale or effectiveness of their efforts. Looking back, we can see that “Justin” wasn’t alone. How many other Russians were out there on Twitter pretending to be Americans?

I’d like to find out.

Last Wednesday, I wrote about an article published on FiveThirtyEight.com: “Why We’re Sharing 3 Million Russian Troll Tweets. As I said there, I downloaded the dataset that they generously made available on GitHub (technically, I forked the repository into my own GitHub space) and then loaded it into the SciServer online science suite. Let me know if you’d like to join this effort.

I used Python’s regular expression features to do a quick search through 1,849,687 million English-language tweets in the FiveThirtyEight dataset. I looked for tweets that showed evidence of claiming to be Americans – featuring “I’m” or “I am” plus some form of “USA” or “America” or “American”. The screenshot below shows me running that command inside a Jupyter notebook in SciServer Compute:

Python commands using the re and pandas modules
Python commands to find tweets like “I’m American”

The search returned 177 tweets from 84 separate authors – counts that should in no way be considered scientific or used for any kind of analysis, either quantitative or qualitative. I then read through all 177 tweets and selected only those that unambiguously claimed to be American citizens/voters.

I was left with 29 tweets from 20 separate Twitter handles, covering the period from December 2014 to August 2017. (Of course, there is no reason to think that Russians impersonating Americans suddenly stopped in August 2017.) Here are five selected randomly:

  • @ISRAEL_WILLS on February 9, 2015: Hope everyone had a great day yesterday. I’m happy we don’t have a war on the American soil. Thank you to all the military serving today. 🙂
  • @JANI_S_JAC on July 4, 2015: #HappyIndependenceDay I’m a patriot and it’s sad for me to see what’s happening to America today
  • @TEN_GOP on February 3, 2016: If I were a dem I’d be embarrassed by who represents my party. But I could never be a dem. For I’m American! #TCOT
  • @TEN_GOP on March 25, 2016: ‘@COJeepGirl well b/c I’m American and Hussein is the President atm’
  • @JANSKEESTR on August 15, 2017: I voted for Trump because I knew he’s the only man who could save America from liberal degeneracy. I’m still sure that I made a right choice

It’s worth reiterating: all of these people are claiming — directly, unambiguously — to be American citizens. All of them are Russian. It is not illegal in most circumstances to claim that you are someone else online. Nor is it illegal for a foreign citizen to have an opinion on a U.S. election. But I find it profoundly disturbing that we are only now realizing the full extent of these Russian operations. The best we can do right now is to try to understand how these trolls have operated in the past in hopes of preventing similar incidents in the future.

One thing is clear: America has never been so ready for your Rocky & Bullwinkle references. Russian catfisher @TEN_GOP says it best:

Trump was a strong & fearless LEADER today. I’m proud to be an American.

P.S. Here is an Excel spreadsheet containing the tweets that I identified, if you’d like to play with the data yourself.

The wolf should be obvious: why I think we really found water on Mars this time

As I mentioned on Friday, when I first heard about the Italian Space Agency’s announcement of water on Mars, I was skeptical. Various space agencies have cried wolf on major discoveries before – most famously, with “NASA Confirms Evidence That Liquid Water Flows on Today’s Mars (it’s actually sand) and Discovery of “Arsenic-bug” Expands Definition of Life (it wasn’t, and it doesn’t). This is not a conspiracy — it’s just overexcitement. Scientists work hard to keep themselves free of confirmation bias, but they’re still human, and sometimes they still see what they want to see.

Given this history, how do we know that it really is a wolf this time? I’ve found that it helps to ask the obvious question.

Aside… This is what bothers me most about global warming deniers. They will go on for pages and pages about July temperatures in Paraguay, without even trying to answer the obvious question: why did global temperatures start to increase at exactly the time when we started releasing into the atmosphere a gas that is known to increase temperatures?

In the case of water on Mars, here is the obvious question. We know for sure that there is liquid water on one of the nine planets in the Solar System: here on Earth. The research team claims that there is liquid water under the polar ice caps on Mars. Could the same techniques they used have detected water under Earth’s polar ice caps, where we know there is water?

It’s right there in the second sentence of the paper that published the announcement (Orosei et al. 2018): “Radio echo sounding (RES) is a suitable technique to resolve this dispute, because low-frequency radars have been used extensively and successfully to detect liquid water at the bottom of terrestrial polar ice sheets.”

The technique they used is the IN SPAAAAAAAAAAAACE version of a commonly-used technique called ground-penetrating radar (GPR). GPR involves transmitting radio waves into the ground, then listening for the echoes of those waves reflecting off various underground layers. The strength of the return signals reflected off each layer tells you what the layer is made of, and nothing reflects quite like water. And that water-related pattern is exactly that kind of reflection that the research team saw on Mars.

The radar image that proves there is water under Mars's south polar cap; it shows up as an underground layer that strongly reflects radio waves
(A) The radar reflection profile found by Mars Express. “Surface reflection” shows the radio waves reflecting off Mars’s surface, while “Basal reflection” shows the radio waves reflecting off the water layer

(B) The same reflection measurements shown as a more traditional graph.

Source: Orosei et al. 2018. Click on the image for a larger version.

Obvious question answered, wolf found. We really did it this time!