Video Transcript (English)

By now we know that the transformer model is made up of encoders and decoders. The encoder is used to understand the meaning and context of a given input. On the other hand the decoder uses the context provided by the encoder to make predictions.

But did you know that we can use each of these components separately? That’s right, we can leverage each part of the transformer model individually to create models for very specific needs. Let’s take apart the Transformer model to see how this works.

In this lesson, we’ll learn about the different models that can be created using the various components of the transformer model. We’ll also identify the models best suited for Atlas Vector Search.

Up until this point, we have been discussing sequence to sequence models. But what do we mean by sequence to sequence?

Sequence to sequence is a model type that uses both encoders and decoders.

Sequence to sequence models excel at tasks that involve taking an input and generating a sentence based on that input.

These models are ideal for translation, text summarization, and answering questions.

Some popular sequence to sequence models are BART and T5.

Next, we have encoder models which only use the encoder from the transformer model. Encoders take in an input and return embeddings.

Also, encoders have bi-directional attention. In other words, they are globally aware and this allows them to examine text that comes before and after the target section.

This makes encoder models particularly good at understanding text. Common use cases include: Word classification, recognizing entities, and extracting answers to questions from provided documents.

Some popular encoder models are BERT, ALBERT, and ELECTRA.

Finally, we have decoder models which use the decoder side of the transformer. Unlike the encoder, decoders take embeddings and generate an output based on the context provided by the embeddings.

Decoders are unidirectional which means the prediction for each output token can only depend on the tokens that have been generated before it

Decoder models are best suited for text generation.

Some common decoder only models include all of the GPT models.

By now you are probably wondering how all of this pertains to Atlas.

While Atlas can’t store models or generate embeddings we can use the rich document model to store those embeddings alongside the rest of your data.

Once we have them, Atlas Vector Search can use them to provide semantic searches.

We can also use the Retrieval Augmented Generation architecture to build applications that leverage both Atlas Vector Search and decoder models.

Sounds cool, right? We’ll learn how to do all of this and more throughout this course.

Let’s recap what we covered in this lesson. First, we learned that the entire transformer model, also referred to as a sequence-to-sequence model, is used for different types of GenAI applications. Alternatively, some models use only the encoder or decoder components of the transformer individually.

Next, we learned that sequence to sequence models are best suited for translation, text summarization, and Q/A.

After that, we learned that encoders are bi-directional. They are best suited for word classification, entity recognition, and extracting answers to questions from provided documents.

Finally, we learned that decoders are unidirectional and are best suited for text generation.

But don't go anywhere because the fun is just starting. See you soon.