Prompt engineering has been declared dead more than once. Google DeepMind’s latest paper does not bury it — it moves it into another modality, from text to vision.
The paper’s proposal, VIPE (Visual Prompt Engineering), rests on a blunt observation: for video models, editing the input image works better than editing the prompt. When an abstract sketch was rewritten into a photorealistic scene, Veo 3.1’s accuracy on physical reasoning tasks jumped from 41.3% to 59.3% — without touching a single word of the prompt.
What Visual Prompt Engineering Is
The logic underneath prompt engineering is that whenever the model’s input space is rich enough, optimizing the input improves downstream performance. Conventionally that “input” meant text instructions — but video models are becoming the foundation models of visual work, and their input already contains an image.
VIPE rewrites the task image into a visual form the model understands and reasons about more easily, without altering the task’s logic. For a physics puzzle like “which bucket does the ball land in after rolling past the obstacle,” the abstract ball-and-ramp diagram becomes billiard balls on a wooden plank — with every spatial relationship preserved.
The whole process runs automatically through three modules:
-
Conceptualizer: a vision-language model generates rewrite instructions that pin down which task elements must survive — object counts, positions, orientations, relative relations.
-
Editor: rewrites the original image per the instructions, producing one or more candidates — via an image editing model or by hand.
-
Filter: an optional step that picks the highest-quality candidate that still carries the key task information.
The Numbers: How Big Is the Gain
Across test suites including the VPCT physical reasoning benchmark:
-
Veo 3.1: accuracy rose from 41.3% to 59.3% with single-sample inference.
-
Omni Flash: from 56.3% to 67.5%.
-
Automated free-form rewriting: error rates fell by more than 75% on some tasks.
-
Sort 3 Numbers: selecting the best visual version per sample drove the error rate to zero.
The interaction with test-time scaling deserves special attention. Without VIPE, Veo 3.1’s multi-sampling ceiling was 50.0%, up from 41.3%. With VIPE, a single sample already hits 59.3%, and stacking multi-sampling reaches 68.0%. One input rewrite buys more than burning extra compute on sampling — and the two effects compound.
The paper also documents a systematic preference: video models do better with realistic inputs, and every rewrite step toward realism improved generation consistency. Effective rewrites share three traits — a more realistic scene, more prominent targets, and more direct semantics.
The Framework: Prompt Engineering Is Input Representation Optimization
Set VIPE in a wider frame and it validates a general rule: prompt engineering was never about phrasing — it is about optimizing the model’s input representation. In the text era, that representation was a token sequence, so wording mattered. In the video-model era, the heaviest component of the input is the visual content itself, which makes “the comprehensibility of the scene” the new optimization surface. The preference for photographic inputs is really the gap between abstract inputs and the training data distribution — close the gap and reasoning improves on its own.
There is also a cost law worth memorizing: image editing is roughly an order of magnitude cheaper than video generation. Spending one cheap image-edit call to reduce the failure probability of one expensive video generation is structural arbitrage in compute. The paper says it plainly: a cheap image edit can reduce the probability of an expensive video generation failing.
Prompt Engineering Won’t Die — It Changes Battlefields
First, the definition of “prompt” is expanding. Prompts for multimodal models will increasingly be joint optimizations over image plus video plus instruction, with text as one component. The team notes that as video models accept richer multimodal context, visual prompt engineering may extend from single images to multiple reference images and even video.
Second, image editing models get repriced. They stop being “generation tools” and become input preprocessing chips that run before video inference — a role change that alters how such models are evaluated and deployed.
Third, automated prompt engineering goes visual. The full pipeline — VLM writes instructions, the editor applies them, an automatic scorer filters — already runs end to end, which makes inference-time adaptive input optimization a research direction of its own rather than a hand-tuning chore.
The boundary is clear too: the native image generation models tested (the Nano Banana family) did not reliably benefit — they are equally at home with abstract and photorealistic styles — and on some tasks text prompts remain stronger. The accurate formulation is that visual prompt engineering does not replace text prompt engineering; it fills in an optimization surface video models had left untouched.
What To Do With This
-
If you build or evaluate video-model applications: audit the realism of your input frames first. Rewriting abstract diagrams and line art as photorealistic scenes before inference may be the cheapest accuracy gain available.
-
Self-check against the three traits: Is the scene realistic enough? Are target objects prominent enough? Is the semantics direct enough? When all three hold, the rewrite is done.
-
Run the cost math: when video retries are expensive, try edit-then-generate. One cheap edit can save an entire round of expensive regeneration.
-
Don’t apply it blindly: for native image generation models and tasks where text already dominates, validate on a small set before scaling.
-
Watch the automation: the VIPE pipeline is already fully automatic, and inference-time input optimization is likely to become a standard capability of video agents — worth building for early.
Prompt engineering is not dead. It just learned to see.
Paper: Visual prompt engineering for video models (arXiv:2607.25537, Google DeepMind, 2026-07-28). Project page: visual-prompt-engineering.github.io
