It Treats Images Like Sentences
The first and most fundamental surprise is how a ViT “sees” an image. Practitioners accustomed to CNNs think in terms of pixels and filters that scan for local patterns like edges and textures. A ViT, however, takes a completely different approach inspired
by Natural Language Processing (NLP). It chops an image into a grid of fixed-size patches, flattens them, and treats the resulting sequence of patches just like a sentence made of words. This shift from a pixel-based grid to a sequence of “image words” is a radical departure. Instead of building understanding from local details outwards, a ViT aims to understand the relationships between all these patches at once, much like a language model understands how words in a sentence relate to each other to form meaning.
It Lacks Common Sense Assumptions
CNNs have what’s called an “inductive bias” baked into their architecture. They are inherently designed to assume that nearby pixels are related (locality) and that an object remains the same even if it moves within the frame (translation invariance). This built-in knowledge helps them learn efficiently from relatively small datasets. ViTs, on the other hand, throw most of that away. They have a very weak inductive bias, meaning they don't make these assumptions about how images work. This is a major surprise for practitioners, as it feels like starting from a less informed position. The model must learn the importance of locality from scratch, a task that requires a colossal amount of data.
Its Insatiable Appetite for Data
A direct consequence of having a weak inductive bias is that ViTs are incredibly data-hungry. A new practitioner might try to train a ViT on a moderately sized dataset and be shocked by its poor performance compared to a standard CNN. This is because without the built-in assumptions of a CNN, a ViT needs to see an overwhelming number of examples to learn the fundamental properties of images for itself. The real power of ViTs only becomes apparent at a massive scale. When pretrained on gigantic datasets, sometimes containing hundreds of millions of images, they begin to outperform their CNN counterparts, often by a significant margin.
It Sees the Big Picture Immediately
The way CNNs and ViTs build context is starkly different. A CNN starts small, identifying edges in early layers, combining them into textures and simple shapes in middle layers, and finally recognizing complex objects in deeper layers. It builds a global understanding hierarchically. A ViT, thanks to its self-attention mechanism, works in the opposite way. From the very first layer, every image patch can attend to every other patch. This allows the model to capture global relationships across the entire image right from the start. For instance, it can immediately associate a patch of a dog's ear with a patch of its tail on the other side of the image, a feat that would take a CNN many layers to accomplish.
Performance Scales Dramatically
Perhaps the most rewarding surprise for those who stick with ViTs is their incredible scalability. While a CNN's performance might start strong and then plateau, a ViT's performance continues to improve dramatically with more data and more computing power. The initial frustration of training a ViT on a small dataset gives way to awe as developers witness its capabilities unlock at scale. This scalability is why ViTs have become the backbone for many state-of-the-art, large-scale vision systems. They may be more computationally expensive and demanding to train initially, but their potential for high performance on complex tasks once trained is a key reason for their rapid adoption in the field.
