This is one of the project in datacamp where we can explore the relationship between names & age of the people born in U.S. This is a good exercise for the usage of Pandas in python.
1. Introduction to Baby Names Data¶
What’s in a name? That which we call a rose, By any other name would smell as sweet.
In this project, we will explore a rich dataset of first names of babies born in the US, that spans a period of more than 100 years! This suprisingly simple dataset can help us uncover so many interesting stories, and that is exactly what we are going to be doing.
Let us start by reading the data.
# Import modules
import pandas as pd
# Read names into a dataframe: bnames
bnames = pd.read_csv('datasets/names.csv.gz')
2. Exploring Trends in Names¶
One of the first things we want to do is to understand naming trends. Let us start by figuring out the top five most popular male and female names for this decade (born 2011 and later). Do you want to make any guesses? Go on, be a sport!!
# bnames_top5: A dataframe with top 5 popular male and female names for the decade
import pandas as pd
bnames = pd.read_csv('datasets/names.csv.gz')
bnames_2010 = bnames.loc[bnames['year']> 2010]
bnames_2010_agg = bnames_2010.groupby(['sex','name'], as_index=False)['births'].sum()
bnames_top5 = bnames_2010_agg.\
sort_values(['sex','births'], ascending=[True, False]).\
groupby('sex').head().reset_index(drop=True)
print(bnames_top5)
3. Proportion of Births¶
While the number of births is a useful metric, making comparisons across years becomes difficult, as one would have to control for population effects. One way around this is to normalize the number of births by the total number of births in that year.
bnames2 = bnames.copy()
# Compute the proportion of births by year and add it as a new column
total_births_by_year = bnames.groupby('year')['births'].transform('sum')
bnames2['prop_births'] = bnames2['births'] / total_births_by_year
bnames2.head()
4. Popularity of Names¶
Now that we have the proportion of births, let us plot the popularity of a name through the years. How about plotting the popularity of the female names Elizabeth
, and Deneen
, and inspecting the underlying trends for any interesting patterns!
# Set up matplotlib for plotting in the notebook.
%matplotlib inline
import matplotlib.pyplot as plt
def plot_trends(name, sex):
data = bnames[(bnames.name == name) & (bnames.sex == sex)]
ax = data.plot(x="year", y="births")
ax.set_xlim(1880, 2016)
return ax
# Plot trends for Elizabeth and Deneen
for name in ('Elizabeth', 'Deneen'):
plt.axis = plot_trends(name, 'F')
plt.xlabel('Year')
plt.ylabel('Births')
plt.show()
# How many times did these female names peak?
num_peaks_elizabeth = 3
num_peaks_deneen = 1
5. Trendy vs. Stable Names¶
Based on the plots we created earlier, we can see that Elizabeth is a fairly stable name, while Deneen is not. An interesting question to ask would be what are the top 5 stable and top 5 trendiest names. A stable name is one whose proportion across years does not vary drastically, while a trendy name is one whose popularity peaks for a short period and then dies down.
There are many ways to measure trendiness. A simple measure would be to look at the maximum proportion of births for a name, normalized by the sume of proportion of births across years. For example, if the name Joe
had the proportions 0.1, 0.2, 0.1, 0.1
, then the trendiness measure would be 0.2/(0.1 + 0.2 + 0.1 + 0.1)
which equals 0.5
.
Let us use this idea to figure out the top 10 trendy names in this data set, with at least a 1000 births.
# top10_trendy_names | A Data Frame of the top 10 most trendy names
names = pd.DataFrame()
name_and_sex_grouped = bnames.groupby(['name', 'sex'])
names['total'] = name_and_sex_grouped['births'].sum()
names['max'] = name_and_sex_grouped['births'].max()
names['trendiness'] = names['max'] / names['total']
top10_trendy_names = names.loc[names['total'] > 999].sort_values('trendiness', ascending=False).\
head(10).reset_index()
print(top10_trendy_names)
6. Bring in Mortality Data¶
So, what more is in a name? Well, with some further work, it is possible to predict the age of a person based on the name (Whoa! Really????). For this, we will need actuarial data that can tell us the chances that someone is still alive, based on when they were born. Fortunately, the SSA provides detailed actuarial life tables by birth cohorts.
year | age | qx | lx | dx | Lx | Tx | ex | sex |
---|---|---|---|---|---|---|---|---|
1910 | 39 | 0.00283 | 78275 | 222 | 78164 | 3129636 | 39.98 | F |
1910 | 40 | 0.00297 | 78053 | 232 | 77937 | 3051472 | 39.09 | F |
1910 | 41 | 0.00318 | 77821 | 248 | 77697 | 2973535 | 38.21 | F |
1910 | 42 | 0.00332 | 77573 | 257 | 77444 | 2895838 | 37.33 | F |
1910 | 43 | 0.00346 | 77316 | 268 | 77182 | 2818394 | 36.45 | F |
1910 | 44 | 0.00351 | 77048 | 270 | 76913 | 2741212 | 35.58 | F |
You can read the documentation for the lifetables to understand what the different columns mean. The key column of interest to us is lx
, which provides the number of people born in a year
who live upto a given age
. The probability of being alive can be derived as lx
by 100,000.
Given that 2016 is the latest year in the baby names dataset, we are interested only in a subset of this data, that will help us answer the question, "What percentage of people born in Year X are still alive in 2016?"
Let us use this data and plot it to get a sense of the mortality distribution!
%matplotlib inline
import matplotlib.pyplot as plt
# Read lifetables from datasets/lifetables.csv
lifetables = pd.read_csv('datasets/lifetables.csv')
# Extract subset relevant to those alive in 2016
lifetables_2016 = lifetables[lifetables['year'] + lifetables['age'] ==2016]
# Plot the mortality distribution: year vs. lx
lifetables_2016.plot(x='year', y='lx')
plt.show()
lifetables_2016.head()
7. Smoothen the Curve!¶
We are almost there. There is just one small glitch. The cohort life tables are provided only for every decade. In order to figure out the distribution of people alive, we need the probabilities for every year. One way to fill up the gaps in the data is to use some kind of interpolation. Let us keep things simple and use linear interpolation to fill out the gaps in values of lx
, between the years 1900
and 2016
.
# Create smoothened lifetable_2016_s by interpolating values of lx
import numpy as np
year = np.arange(1900, 2016)
mf = {"M": pd.DataFrame(), "F": pd.DataFrame()}
for sex in ["M", "F"]:
d = lifetables_2016[lifetables_2016['sex'] == sex][["year", "lx"]]
mf[sex] = d.set_index('year').\
reindex(year).\
interpolate().\
reset_index()
mf[sex]['sex'] = sex
lifetable_2016_s = pd.concat(mf, ignore_index = True)
lifetable_2016_s.head()
8. Distribution of People Alive by Name¶
Now that we have all the required data, we need a few helper functions to help us with our analysis.
The first function we will write is get_data
,which takes name
and sex
as inputs and returns a data frame with the distribution of number of births and number of people alive by year.
The second function is plot_name
which accepts the same arguments as get_data
, but returns a line plot of the distribution of number of births, overlaid by an area plot of the number alive by year.
Using these functions, we will plot the distribution of births for boys named Joseph and girls named Brittany.
def get_data(name, sex):
name_sex = ((bnames['name'] == name) &
(bnames['sex'] == sex))
data = bnames[name_sex].merge(lifetable_2016_s)
data['n_alive'] = data['lx']/(10**5)*data['births']
return data
def plot_data(name, sex):
fig, ax = plt.subplots()
dat = get_data(name, sex)
dat.plot(x = 'year', y = 'births', ax = ax,
color = 'black')
dat.plot(x = 'year', y = 'n_alive',
kind = 'area', ax = ax,
color = 'steelblue', alpha = 0.8)
ax.set_xlim(1900, 2016)
return
# Plot the distribution of births and number alive for Joseph and Brittany
#get_data("Brittany", "F")
plot_data('Brittany', 'F')
9. Estimate Age¶
In this section, we want to figure out the probability that a person with a certain name is alive, as well as the quantiles of their age distribution. In particular, we will estimate the age of a female named Gertrude. Any guesses on how old a person with this name is? How about a male named William?
# Import modules
from wquantiles import quantile
# Function to estimate age quantiles
def estimate_age(name, sex):
data = get_data(name, sex)
qs = [0.75, 0.5, 0.25]
quantiles = [2016 - int(quantile(data.year, data.n_alive, q)) for q in qs]
result = dict(zip(['q25', 'q50', 'q75'], quantiles))
result['p_alive'] = round(data.n_alive.sum()/data.births.sum()*100, 2)
result['sex'] = sex
result['name'] = name
return pd.Series(result)
# Estimate the age of Gertrude
estimate_age('Gertrude', 'F')
10. Median Age of Top 10 Female Names¶
In the previous section, we estimated the age of a female named Gertrude. Let's go one step further this time, and compute the 25th, 50th and 75th percentiles of age, and the probability of being alive for the top 10 most common female names of all time. This should give us some interesting insights on how these names stack up in terms of median ages!
# Create median_ages: DataFrame with Top 10 Female names,
# age percentiles and probability of being alive
top_10_female_names = bnames.\
groupby(['name', 'sex'], as_index = False).\
agg({'births': np.sum}).\
sort_values('births', ascending = False).\
query('sex == "F"').\
head(10).\
reset_index(drop = True)
estimates = pd.concat([estimate_age(name, 'F') for name in top_10_female_names.name], axis = 1)
median_ages = estimates.T.sort_values('q50').reset_index(drop = True)
median_ages.head()