Kaggle has a number of datasets that one can download and play with. I was pleasantly surprised to find one that contains detailed script data for all episodes of The Simpsons up to the beginning of season 28. There are three files with information about the characters, the episodes, and all lines of dialog. In particular, the dialog file is quite detailed. Scenes are defined, and the speaker and timestamp are given for each spoken line. This is much more detailed than other datasets I have looked for in the past for other shows, some of which have all spoken words, but no speaker attached, making analysis much more difficult.

There is a lot that can be done with this dataset, but in this post, I'd like to look at the speech of the various characters and determine whether there are significant differences in their vocabularies or sophistication of their speech.

You can download the dataset here.

Firest, let's see what we can get from the characters file.

A pretty simple file that gives us character names and genders. It turns out that there are an astounding 6722 characters! Of course, most of these are minor characters that are only seen briefly. Also, looking more thoroughly at the file, you will notice that there are many variant characters based on the main characters. For example, here are some interesting versions of Homer:

For this study, I am only going to look at the top 30-40 characters in terms of number of spoken lines or words, and I will only consider words spoken by the main character and not any variant characters.

Dialog¶

The simpsons_script_lines.csv file contains all spoken lines and scenes from every episode. Here are the first few lines (not all columns are shown).

In this file, we get non-speaking lines, which establish scene locations, each followed by spoken lines of the characters in the scene. Timestamps are provided, along with character name, all spoken words, and word count. A normalized text column is also provided, in which all punctuation has been stripped from the spoken words, and all words are lowercase. Let's determine the most prolific speakers.

There is an interesting discrepancy depending on which column we use for counting. For Homer, for example, you get anywhere from 27917 lines up to 29850. The discrepancy must result from some NA values in certain columns.

Now we see where the difference comes from. Homer has a lot of lines that are just grunts and groans with no spoken words! Let's go ahead and count those anyway.

With nearly 30,000 lines, Homer is by far the most prolific speaker. He is followed by his three immediate family members, all of whom have many times as many lines as any other character. If we look at number of words (no grunts allowed), we see a similar picture.

Either way you look at it, Homer absolutely dominates the dialog in The Simpsons. All four of the main family characters are well above Mr. Burns, who comes in at fifth place. In fact, the main family speaks 604,467 words, which accounts for more than 46% of the total 1.3M words!

Sophistication of Speech¶

Any long time viewer of The Simpsons knows how central Homer is to the show. The more interesting question to me is whether there are big differences in the sophistication of speech or vocabulary size between the characters. Many measures have been devised to estimate the difficulty of reading text. Two of the most well known are the Flesch Reading Ease and Flesch-Kincaid Grade Level. Although these were designed for written text, I am not worried about the accuracy of these measures for the spoken words in this case. Rather, I want to see how characters compare to each other.

These measures require counts of words, sentences, and syllables. Word counts are provided, and sentence counts are easy to obtain. For syllable counts, I'm using a well-known method that counts the vowel sounds from the Carnegie Mellon Pronouncing Dictionary. Words not found in the dictionary are given 1 syllable, which will sometimes be incorrect, but I don't expect it to affect the results significantly.

Here, I have plotted the Flesch readability and Flesch-Kincaid grade level for the 30 characters with the most total words spoken. The two measures have an inverse relationship. A higher readability should correspond to a lower grade level. All characters would be rated as highly readable. By either measure, three characters stand apart from the rest - Kent Brockman, the Announcer, and Mayor Joe Quimby. We see a bit more variability in the grade level. While most characters fall somewhere in the 1st-3rd grade levels, Kent Brockman reaches almost to 5th grade level. A few other characters are above 3rd grade level, including the Announcer, Mayor Joe Quimby, Gary Chalmers, Dr. Hibbert, Principal Skinner, Prof. Frink, and Comic Book Guy. Within the Simpsons family, Homer and Bart come in at the lower end of the range, while Marge and Lisa fare a bit better. Nelson Muntz comes in last among the top 30 most heard characters. No surprise there.

Vocabulary¶

Now, we will look at the vocabularies of the different characters. A bit of processing is required. First, the lines data will be grouped by character, and for each, all of their spoken lines will be joined to create one long string. The string will be tokenized to form a list of individual words. English stop words are removed to simplify calculations later. Total words spoken, number of unique words, and vocabulary size as a percentage of total words are determined.

The following table shows the top 10 characters for total words.

The list looks a bit different if we sort by vocab size.

Homer has, by far, the greatest number of unique words, but mostly because he dominates the dialog of The Simpsons.

This plot shows vocab size vs. total words.

There is a definite correlation between total words spoken and unique vocabulary. Homer, of course, is the point in the upper right corner of the plot.

The following plot shows the vocab:total ratio vs. total words for characters with more than 500 total words spoken.

The unique vocab to total words ratio would be expected to decline with more words because additional words are more likely to have been used before.

Homer has the greatest vocabulary, mostly because he speaks so much more than any other character. Given his Flesch-Kincaid score, however, it wouldn't seem right to crown him the vocab king. Perhaps there is a better way to measure vocabulary as it relates to sophistication of speech by taking into account the number of words spoken. Instead of summarizing the size of vocabulary for each character, let's plot a curve showing how many unique words have been used vs. the total number of words spoken. I remove English stop words and limit the calculation at 20,000 total words because most characters don't surpass that number and because the set calculations for much larger lists were taking a long time.

In the early range up to ~5000 words, there appears to be 3 groups. In the lowest group, we find some of the most important characters, including Homer, Marge, Bart, and Moe. I am not surprised that Homer has the shallowest vocab curve, but I am a bit surprised that Marge runs neck-and-neck with Homer most of the way. I guess I expected her to be significantly ahead of him. Another surprise to me was that Moe begins to overtake all main Simpsons family members except Lisa. Bart is also well ahead of Homer and Marge, but not Lisa, of course. Lisa falls in the middle group early on, but ends with a respectable vocabulary. Within the middle group, Mr. Burns and Principal Skinner are at the top, and at the 17,500 word mark, they easily have the greatest vocabularies. The high fliers early on include the Announcer, Kent Brockman, Comic Book Guy, Prof. Frink, and Mayor Joe Quimby. All except Kent Brockman have far fewer total words than other characters, but add new words at the greatest pace. Kent Brockman really stands out, adding new words at a faster pace all the way up to the 10,000 word mark. We saw Kent Brockman, the Announcer, and Mayor Quimby before, in the Flesch measures, so it is no surprise to see them in the highest group here. Prof. Frink doesn't surprise me, but Comic Book Guy sure did!

I think this curve is a better way of measuring sophistication of speech because it lets you compare characters on an equal-word basis. The slope of the line represents the rate of using new words, which I think is a better measure of sophistication of speech.

This graph only covers up to 20,000 words per character. For the main Simpsons family members, this only accounts for the early seasons. Over the course of a show that has been on air for this long, however, characters can develop, and the writing can change. It would be interesting to check if Homer is getting any smarter over time. Let's break down Homer's 175,000 words into 10 groups of 17,500 and plot them.

This is actually quite interesting because there is a definite difference between groups, and there is a pretty clear upward trend in the early groups. It would appear that Homer is indeed getting smarter over time! What about some of the other characters?

Marge shows a similar pattern. Let's check Bart and Lisa to be sure.

Except for a small anomaly where Lisa's second group is below her first, it seems to be the general case that all the Simpsons characters become a bit more sophisticated after the first few seasons.

Final Thoughts¶

I really enjoyed working with this dataset, as The Simpsons is one of my favorite shows. The dialog data is really rich, with a lot of interesting things to discover. Many of the results confirmed my suspicions, such as the fact that Homer spoke the most, but there were also some surprises. I would have expected Prof. Frink to come out on top for new vocabulary rate, but in fact the clear winner was Kent Brockman. I never would have suspected that Comic Book Guy would be so sophisticated. Given that I didn't watch the show in the first few seasons, I didn't expect to see such a difference in new vocab rates over time. Maybe I should go back and watch some of the early episodes to see if it is really noticeable.

One of the great things about this dataset is that it defines each scene before the lines of dialog of the characters in it. I hope to use that to follow this up with an analysis of character relationships.