Video Transcript (English)

I know you're excited to implement the autocomplete functionality in our movie catalog app. We'll get to that shortly. First let's get some basics out of the way.

In this video, we’ll go over some familiar topics such as analyzers, tokenization, minGrams, and maxGrams. But they work differently in autocomplete.

If you haven’t completed our unit on Atlas Search analyzers or need a refresher, be sure to check it out before proceeding with this unit.

The autocomplete functionality has four main components: an analyzer, tokenization strategy, minGrams, and maxGrams.

Let’s start with the tokenization strategy.

When using autocomplete, we have three options for tokenizing a string. We can use edgeGram, nGram, and rightEdgeGram.

You might remember edgeGram and nGram from our lesson on custom analyzers. So first let’s talk about rightEdgeGram.

rightEdgeGram is similar to edgeGram, but it starts on the right edge of the word instead of the left. This makes it ideal for languages that are read from right to left.

For that reason, we won’t have much use for it in our movie catalog app, which is primarily in English.

So let’s turn our attention to edgeGram.

edgeGram creates tokens based on minGrams and maxGrams.

minGrams is the minimum number of characters per token, and maxGrams is the maximum number of characters per token.

Let’s take a moment to refresh our understanding of how a pure tokenizer works.

We'll then look at how it differs when using autocomplete.

Take the movie title “Star Wars: Episode Three - Revenge of the Sith.”

We’ll use the edgeGram tokenizer with a minGrams of four and maxGrams of seven.

Since edgeGram begins at the leftmost edge and minGrams is four, it produces “star” as the first token.

The next token includes the first one but is now larger by one, so “star {SPACE}”.

The process repeats until the length of the token reaches the set maxGrams of seven, then it stops.

Simple enough, right? Well, autocomplete actually performs tokenization a bit differently. The tokenizer for autocomplete is more akin to a token filter. Let me show you what I mean.

With autocomplete, tokenization is a two-step process.

First, the string goes through the defined analyzer that creates tokens.

Second, the tokens provided by the analyzer get further filtered based on the tokenization strategy to generate sub-tokens, or grams.

So what does this look like? Let’s go through the two-step process together.

We'll use the same title as before with the standard analyzer to keep it simple.

The standard analyzer will create a token for each word but remove any punctuation and capitalization.

Once the analyzer has finished its job, these tokens are passed to the tokenization strategy and broken down further.

Like earlier, we’ll use edgeGram with minGrams set to four and maxGrams set to seven.

The first token provided by the analyzer is “star”. This token is to be broken down into smaller sub-tokens based on the tokenization strategy.

In this case, because minGrams is set to four, the first sub-token is still “star”

Since that's the entire analyzer token we're done breaking this one down, right? Well - we are, however, we are not done generating sub-tokens. Bear with me while we finish with "star."

Instead of stopping, something called shingling occurs. Shingling is the process of overlapping or combining tokens.

For example, if the token from the analyzer is smaller than the maxGrams, the tokenizer will use the following tokens until it reaches maxGrams.

This results in the next tokens being “star {SPACE}” followed by "star-space-w". This pattern continues until we reach “star-space-w-a,” because the maximum number of grams is seven.

Now that we have arrived at the maxGrams, what do you think is going to happen next?

Instead of stopping, the tokenizer moves on to the left edge of the next analyzer token. In this case, the next analyzer token is “wars”.

This process repeats itself until edgeGram goes through every token provided by the analyzer.

Now, looking at all of the generated tokens, you may notice something interesting.

We have tokens that are smaller than the set minGrams of four, such as “of”, “the”, and the roman numeral three.

Any tokens created by the analyzer that are less than the set minGrams are automatically included in the sub-token result set.

With the tokenization process complete, these sub-tokens will be stored in our Atlas Search index for use in autocomplete queries.

The good news for us is we don’t have to do any of this ourselves - Atlas Search automatically takes care of it for us. But more on that later.

For now, let’s take a moment to recap what we learned.

First, we learned that autocomplete has four main components: an analyzer, the tokenization strategy, minGrams, and maxGrams.

After that, we learned that autocomplete tokenization is a two-step process.

First, the string goes through an analyzer that creates tokens.

Then, the tokens provided by the analyzer are filtered further based on the tokenization strategy.

Now that you understand how autocomplete works behind the scenes, let’s move on to the fun part: putting it into action. In the next lesson, we’ll begin to do that by configuring our index for autocomplete. See you there.