Training Artificial Intelligence: Bags of Words

Students explore song recommendation, another example of data-driven algorithms at work. Students consider how data aggregation is a key component of machine learning.

It’s likely that Michelle could get better suggestions without Spotify making any changes to the code base. But it’s also possible that changing the code would improve Michelle’s experience! Let’s consider song recommendation in more depth and explore the idea that sometimes it is helpful for algorithms to change.

Very broadly, a song recommendation system does two things:

What does building the profile for a listener entail?

As you discovered, song recommendation systems need training in order to make recommendations.

In machine learning, we generally start with a large chunk of data. A model is then generated from the data. That model is generally expected to be a lot smaller than the original chunk of data (but it may still be huge!). The model can be queried from to get answers.

In the case of Spotify, the model is a summary of the users' listening habits. Using this model, Spotify answers questions about new, unseen data (e.g. Do we expect Michelle to like the latest Taylor Swift song…​ or not?).

When Michelle observed that Spotify must have updated its algorithms, she may have been correct…​

But we have no way of knowing what actually goes on behind the scenes at Spotify, because 1) these are trade secrets (which companies don’t talk about) and 2) there is a huge team actively working on Spotify so how it works could change from day to day!

Students practice thinking like a hacker to determine the basic requirements of a successful plagiarism detection program. They consider the "bag-of-words" model, which a user can query to better understand how similar or different two documents are.

A student in your class has been accused of plagiarism because an AI plagiarism detector flagged their essay.

Your teacher seems to believe that the plagiarism detector can be trusted with 100% certainty — but the student is your friend, and they assure you that they did not plagiarize.

Let’s learn how a plagiarism detector works so that we can defend the student.

To understand the workings of plagiarism detection, we’ll start by looking at a simple detector.

Plagiarizers usually alter at least a few words of the original document. Sometimes they change the ordering of the text, and sometimes they delete a sentence or word here and there.

We need a more sophisticated plagiarism detector!

Rather than detecting identicality, we need to determine the closeness of two documents. To do that, we summarize each document, and then compute the distance between the summaries.

One standard way to summarize a document is by creating a "bag of words" model. Let’s try it on two documents (below); each document is an example of jazz "scatting", when a vocalist improvises with nonsense syllables.

The bag-of-words summary for Document a looks like this: "be": 2, "doo": 3

As you can see, we’ve taken the original sentence and disregarded word order, creating a collection that focuses solely on word frequency.

Note: We could have written these bag-of-words summaries as "doo": 3, "be": 2, but once we decide on a word order for one document, adhering to that same order is required. The simplest way to be consistent is to use alphabetical order.

The bag-of-words summary for both documents is exactly the same!

A plagiarism detector that uses this model, taking stock of word frequency and word order, could compare the bags instead of the documents. If it did so, it would conclude that the two bags of words are a perfect match…​ and that Document a and Document b are suspiciously similar.

Checking if two bags of words are identical is an improvement from checking if two texts are identical.

Students explore the importance of data normalization, when we organize data to follow a standard pattern.

Here are some discoveries we have made so far:

What we need is a way to check if bags are similar!

One strategy programmers use for this is to represent bags of words as points in space.

Let’s see how that would work for Documents a and b.

When we plot a point on the coordinate plane, we typically locate x on the horizontal axis and y on the vertical axis. Similarly, bags must use same word order if we want meaningful results.

The decision we just made — to use "be" then "doo" for both bags of words — is an example of data normalization. Data normalization is the act of adapting and modifying disparate data so that they all have the same characteristics (making them easy to compare and otherwise compute with).

Failure to normalize data can lead to useless and confusing outputs.

Let’s look at some slightly more complicated documents and consider how to plot their points in a multi-dimensional space.

We have a problem. We can plainly see that Documents c and d are not the same …​ but their points are…​

In the example above, we forgot the data normalization. How can we fix it?

To solve this problem, let’s start by taking a closer look at our data.

We must revise our bag-of-words summaries and our points!

Normalizing data requires that we consider all the words; when a word occurs zero times in a document, we acknowledge it. Instead of glossing over the dimension, we indicate that a given word occurred zero times. When we include all of the words from both documents, we produce a model with the correct dimensionality. For the bag-of-words model, the dimensionality equals how many different words are in the corpus.

It is a bit trickier to envision plotting these points, but not impossible!

Let’s recap:

Compressing text into bags of words gives us a coarse-grained notion of similarity. Let’s explore how to produce a more refined notion of similarity.

When we ask people whether two documents are the same, they rarely give us a black-and-white "yes" or "no" answer. Instead they tend to speak about shades of similarity. Likewise, we would like our computer to give us a range of values that give us a sense of how similar the two documents are. In other words, we would like the output to be a Number, not just a Boolean (identical, not identical).

Now that we know how to represent our bag of words summaries as points in space, we can draw a ray from the origin through each of those points and ask: What is the angle between the two rays?

Take, for example, this comparison between two strings: stringA ("doo doo doo doo") and stringB ("be be be be").

If two documents are identical, they will be at the same point in space, and have the same ray extending from the origin to that point. That means the angle between those rays will be 0°. Even if one document just rearranges the other, their bags of words will be identical—thereby again making the angle between the lines 0°.

As the documents contain different words, the angles between the rays will grow. To reflect this, we can use the angle-difference function. It will give us a value between 0° (if the two are identical) and 90° (if the two have nothing in common).

When two documents are represented as two points in space, angle difference is the difference between the angles formed by rays from the origin through those points.

The contract for angle-difference is below.