First, What Is an 'Encoder-Only' Model?
The original transformer architecture that kicked off the modern AI boom had two parts: an encoder and a decoder. The encoder's job is to read an input sequence (like a sentence) and build a rich, contextual understanding of it. The decoder's job is to take
that understanding and generate a new sequence, like a translation. Encoder-only models like BERT are specialists; they are built using only the encoder part. Their entire design is optimized for one thing: deeply understanding the relationships between words in a text. This is achieved by looking at the entire sentence at once, both left-to-right and right-to-left, which is why it's called "bidirectional." This design choice makes BERT fantastic for tasks that require comprehension, but it’s also the root of many surprises.
Surprise 1: It's Terrible at Writing Sentences
A common first mistake is trying to use BERT like a chatbot or a story writer. You give it a prompt and expect it to generate a coherent paragraph of new text. It can't, and the results are often nonsensical. This is by design. Because BERT is bidirectional, it sees all the words at once, which is great for understanding context but breaks the step-by-step, word-by-word logic needed for generation. Models like GPT, which are "decoder-only," are built for generation because they are unidirectional, predicting the next word based only on the words that came before. BERT is made for analysis tasks: sentiment classification, question answering, and named entity recognition. It understands language; it doesn't speak it.
Surprise 2: The Output Isn't a Simple Answer
When you run a sentence through BERT, you don't get a neat label like 'positive' or 'negative.' Instead, you get a massive array of numbers—a high-dimensional vector for every single word in your input. This is the model's rich, contextual representation. For a beginner, this can be confusing. What are you supposed to do with all these numbers? The key is that BERT is almost always used as a base model. To get a useful output, you have to add a task-specific "head" on top of it. For sentiment analysis, you'd add a classification layer that takes the vector from the special [CLS] token and learns to map it to 'positive' or 'negative.' For question answering, you'd add a different head that learns to identify the start and end tokens of the answer in a passage. BERT provides the understanding; the head provides the final answer.
Surprise 3: It Was Trained to Play Mad Libs
BERT's deep understanding comes from its unique pre-training process, which involves two main tasks. The first, and most important, is Masked Language Modeling (MLM). During this phase, the model is fed billions of sentences where 15% of the words have been randomly hidden, or "masked." BERT's only job is to predict those missing words by using the surrounding context. Think of it as a giant game of Mad Libs. This forces it to learn intricate grammatical and semantic relationships. The second task, Next Sentence Prediction (NSP), trains the model to determine if two sentences logically follow each other. Understanding these training objectives is key, as they explain why BERT is so good at context but requires fine-tuning on a specific, labeled dataset to become useful for a real-world application.
Surprise 4: It Can Be Slow and Expensive
For all its power, BERT is computationally heavy. The self-attention mechanism, which allows every word to look at every other word, has a cost that grows quadratically with the length of the input. This means processing long documents can be very slow and resource-intensive, requiring powerful GPUs or TPUs. Furthermore, the original BERT model has a fixed input size of 512 tokens, making it unsuitable for tasks involving very long texts like summarizing a book. This has led to the development of many smaller, "distilled" versions of BERT and more efficient architectures. For many simpler NLP problems, a massive model like BERT can be overkill, and a lighter model might provide a better balance of performance and cost.
