Vision-language models (VLMs) support visual understanding alongside text input. They are typically built by passing visual tokens from a pretrained vision encoder through a projection layer into a pretrained large language model, letting the encoder’s rich visual representations combine with the LLM’s world knowledge and reasoning. Applications range from accessibility assistants and UI navigation to robotics and gaming.
VLM accuracy generally improves with input image resolution — which forces a tradeoff between accuracy and efficiency. Many production use cases need both: low latency for real-time applications and on-device operation for privacy-preserving AI. In a paper accepted to CVPR 2025, Apple ML researchers shared their answer: FastVLM, a vision-language model of elegant design that significantly improves the accuracy-latency tradeoff. FastVLM uses a hybrid-architecture vision encoder purpose-built for high-resolution images, delivering accurate, fast, efficient visual query handling suited to real-time on-device apps. Inference code, model checkpoints, and an MLX-based iOS/macOS demo app are available on GitHub.
Resolution Versus the Accuracy-Latency Tradeoff
VLM accuracy typically rises with image resolution, especially for detail-heavy tasks — document analysis, UI recognition, answering natural-language questions about an image. In Apple’s figure 1 example, asked to read a street sign, the model given a low-resolution image fails; the same model given high resolution correctly identifies the “Do Not Enter” sign.
But high resolution explodes time-to-first-token (TTFT) in two ways at once: higher-resolution images take longer for the vision encoder to process, and the encoder emits more visual tokens, inflating the LLM’s prefill time. TTFT is their sum, and at high resolutions the vision encoder becomes the dominant bottleneck. The numbers make it vivid — for a 1.5B-parameter VLM (fp16), vision-encoding latency across input resolutions of 256/512/768/1024/1536 pixels runs 6.8ms, 21.2ms, 54.8ms, 116.4ms, and then 883.7ms — a cliff, while LLM prefill grows far more gently (49.8ms to 281.9ms). FastVLM is the new VLM designed to break exactly this cliff without sacrificing accuracy.
The Hybrid Encoder Wins the Accuracy-Latency Fight
To find which architecture yields the best tradeoff, Apple systematically compared pretrained vision encoders with everything else held constant — training data, recipe, LLM — changing only the encoder. Transformer-based ViT-L/14 and SigLIP-SO400 were plotted at their native resolutions, alongside the fully-convolutional ConvNeXT and the hybrid FastViT (combining convolution and transformer blocks, from Apple’s earlier FastViT, ICCV 2023, and MobileCLIP, CVPR 2024). FastViT delivered the best accuracy-latency balance of all — roughly 8 times smaller and 20 times faster than ViT-L/14.
FastViT’s hybrid trunk was ideal for efficient VLMs, but challenging tasks needed a larger encoder. Naively scaling up each FastViT layer made it less efficient at high resolutions than the fully-convolutional encoder. So the team designed a new trunk from scratch: FastViTHD, purpose-built for high-resolution images. It adds a stage compared to FastViT and uses the MobileCLIP recipe for pretraining, producing fewer but higher-quality visual tokens. FastViTHD posts lower latency than FastViT at high resolutions — and paired against different LLM sizes (0.5B, 1.5B, 7B) across image resolutions, the Pareto-optimal curves show FastVLM on FastViTHD beating FastViT-based models on the accuracy-latency tradeoff, up to 3 times faster at equal accuracy. A subtle finding worth internalizing: very high resolution with a smaller LLM is not always optimal — sometimes switching to a larger LLM beats raising resolution.
FastViTHD is a hybrid convolution-transformer architecture — a convolutional trunk, three convolution stages, and two following transformer stages, each preceded by a patch-embedding layer halving spatial dimensions. FastVLM uses FastViTHD as its encoder with a simple MLP module projecting visual tokens into the LLM’s embedding space.
Beating Token Pruning and Merging at Their Own Game
Previous acceleration work used elaborate token-merging or pruning techniques to cut visual token counts and speed LLM prefill. FastVLM outperforms all of them on overall accuracy at every visual-token budget: at 16 tokens it averages 68; at 256 tokens, 72.4 — while FastV at 192 tokens reaches only 60.9, SparseVLM at 192 reaches 68.4, and VisionZip at 192 reaches 69.7. The advantage flows from FastViTHD’s high-quality tokens — and because FastVLM needs no complex pruning or merging machinery, it deploys more simply too.
Against dynamic tiling (the AnyRes approach of splitting ultra-high-resolution images into tiles, encoding each separately, and sending all tokens to the LLM), FastVLM’s native high-resolution strength again wins: without tiling, the accuracy-latency balance is better across the range — tiling only pays at the very highest resolutions with the smallest 2×2 grids. The final comparison against mainstream VLMs is the headline: FastVLM is 85 times faster than LLaVA-OneVision (0.5B LLM), 5.2 times faster than SmolVLM, and 21 times faster than Cambrian-1 (7B) — at higher accuracy on the seven-benchmark average (ChartQA, TextVQA, DocVQA, OCRBench, AI2D, MMMU, ScienceQA): FastVLM 7B at 641ms TTFT and 76.2 average, versus Cambrian-1’s 5,085ms at 68.8 and LLaVA-Next’s 20,347ms at 63.1.
On-Device, Near-Real-Time
To demonstrate on-device efficiency, Apple shipped an MLX-based iOS/macOS demo app: FastVLM 0.5B runs locally on an iPhone 16 Pro’s GPU with time-to-first-token displayed on screen — near-real-time visual question answering in your hand. The conclusion Apple draws is understated but large: VLMs power a wide range of practical applications, their accuracy normally tracks input resolution, and that tradeoff has limited their value for anything needing both precision and speed. FastVLM’s FastViTHD encoder resolves the tension with a simple design that beats existing methods on both axes.
The strategic reading: FastVLM is Apple staking its corner of the VLM world on latency and on-device privacy rather than scale — the opposite bet to the datacenter giants, and arguably the right one for a company whose products are the devices. Fewer, better tokens beat more, pruned tokens; a purpose-built encoder beats post-hoc compression; and on-device near-real-time vision is now a shipping reality rather than a paper claim. For developers, the open-sourced checkpoints and MLX app make this the fastest path to building camera-aware, privacy-preserving AI experiences on Apple hardware. Paper: Apple ML Research. Code: github.com/apple/ml-fastvlm.
