The AI That Connects Words and Pictures
At its core, CLIP (Contrastive Language-Image Pre-training) is a neural network trained on a massive dataset of 400 million image-text pairs scraped from the internet. Instead of learning to classify images into predefined categories like "cat" or "dog,"
it learns the relationship between an image and the words used to describe it. It does this through a process called contrastive learning, where it's trained to match a given image to its correct text description and push it away from incorrect ones. The result is a shared 'embedding space' where the concept of a dog in a picture and the text "a photo of a dog" are mathematically close. This allows for incredible flexibility, most notably "zero-shot" classification, where the model can identify objects it wasn't explicitly trained to name.
Surprise 1: It's Incredibly Literal and Lacks Common Sense
One of the first shocks for new users is CLIP's extreme literalism. The model doesn't 'understand' context or intent the way a human does; it just matches patterns it learned from its training data. This can lead to unexpected results. For example, the model can be tricked by placing text within an image. In a famous example from OpenAI's initial paper, attaching a piece of paper that says "iPod" to an apple can make CLIP confidently classify the apple as an iPod. It also struggles with abstract concepts. It's notoriously bad at counting objects, understanding spatial relationships (like "on top of" or "behind"), or processing negative prompts ("a man with no hat"). It has learned what a hat looks like, but not the concept of absence.
Surprise 2: It's Full of Hidden Biases
Because CLIP was trained on a vast, unfiltered dataset from the internet, it inherited all the biases present in that data. This manifests in ways that can be subtle or stark. Studies have shown the model associates certain jobs with specific genders and struggles to perform accurately on images from lower-income households, whose daily items might look different from the more common examples in its dataset. For example, a "light source" in a low-income country might be a fire, which looks very different from the ceiling lamps more prevalent in Western-centric web data. These biases aren't a bug but a direct reflection of the data it learned from, making it a critical consideration for any real-world application, especially content moderation or classification of people.
Surprise 3: It Can Be Manipulated by Gibberish
Perhaps the most alien aspect of CLIP is its vulnerability to what are known as adversarial attacks or, more colloquially, 'typoglycemia' or 'gibberish' prompts. Researchers have discovered that certain nonsense words or even random strings of characters can reliably make the model 'see' specific objects. This reveals that CLIP isn't 'reading' in a human sense. Instead, it's responding to statistical patterns in the text tokens. A specific, meaningless token might happen to be located near the cluster for "bird" in the model's vast multidimensional map of concepts, so feeding it that token can trigger the 'bird' concept. This highlights the gap between its sophisticated output and its fundamentally mathematical, non-cognitive process.
Surprise 4: It Favors the Big and the First
When presented with images containing multiple objects, CLIP exhibits clear preferences. Research shows its image encoder is biased toward larger objects in a scene, often giving them more weight in its analysis. Simultaneously, its text encoder tends to prioritize words that appear earlier in a prompt. If you ask for "a man and a dog," the model may focus more on the man than the dog, simply because he was mentioned first. This bias can directly impact the performance of text-to-image models like Stable Diffusion that use CLIP's text encoder to interpret prompts. For practitioners, this means the order of words in a prompt is not just a matter of style but a key factor in guiding the model's attention.
