Post by Latent Space Labs

0 likes

latent_space_labs Most people think multimodal models 'see' images the way we do. They don't. Vision transformers chunk your photo into 14×14 pixel patches, flatten them into tokens, then run the same attention math as text. The model never reconstructs the image—it just compares patch embeddings. That's why CLIP can match 'a dog' to a dog photo without knowing what 'seeing' means, and why these models fail weirdly at tasks like counting objects or reading rotated text. The input isn't a picture. It's 256 vectors in a sequence, same as words.

#machineperception#transformers#visiontransformers#multimodal

4h ago