Retrospective Physical Process Reasoning

RetroHolmes: When Semantic Plausibility Fails Retrospective Physical Process Reasoning

RetroHolmes is a real-world benchmark that asks vision-language models to reason backward from outcomes: is a transformation physically reachable, and what causal steps could have produced it?

Explore the benchmark ↓ Paper PDF ↗

RetroHolmes taxonomy of physical transitions

FROM ENDPOINTS → CAUSES

9,530image pairs

14real-world domains

5diagnostic dimensions

3aligned metrics

01 / The gap

A plausible procedure is not necessarily a possible one.

Modern VLMs can describe what a scene looks like and generate fluent instructions. But fluency can hide impossible physics: a chopped ingredient cannot become whole again, and a sealed container cannot be filled without opening it.

Can a model distinguish “sounds right” from “could really happen”?

TASK 01

World reachability judgment

Given two sparse visual endpoints, determine whether a valid physical transition exists.

TASK 02

Inverse step reconstruction

If reachable, infer the causal execution chain that bridges the initial and target states.

02 / RetroHolmes

Object-centric, process-oriented, deliberately diagnostic.

The benchmark moves beyond endpoint recognition and tests whether models can reason over hidden physical transitions.

Four transformation families.

RetroHolmes covers physical deformation, material and state change, topological assembly, and hybrid processes across cooking, woodworking, robotics, crafting, and other real-world domains.

Reachable / unreachableSparse endpointsCausal step chainsHuman verification

Five evaluation dimensions in RetroHolmes

Five evaluation dimensions probe causal modeling, long-range reasoning, belief conflict, temporal reversibility, and irreversible micro-states.

03 / Simulate-and-Verify

Ground abstract reasoning in simulated state evolution.

A lightweight analysis-by-synthesis instantiation turns a verbal hypothesis into a visual trajectory that can be checked.

Initial planner

Propose a complete candidate execution chain from the two endpoint images.

Physical world simulator

Synthesize intermediate visual states while anchoring the start and target images.

Verifier

Use the simulated evolution to verify reachability and refine the prediction.

04 / Experimental results

Strong perception does not guarantee physical reasoning.

Complete results from the main experiment. JA = Judgment Accuracy, BCS = Belief-Conflict Sensitivity, and SC = Step Completeness.

Model	Judgment Accuracy (JA) ↑					ED3 / BCS ↓	Step Completeness (SC) ↑
Model	All	ED1	ED2	ED4	ED5	ED3 / BCS ↓	All	ED1	ED2
Random Choice	50.03	50.07	49.87	26.64	50.50	—	—	—	—
Closed-source models
GPT-5.2	72.58	72.03 ↓0.55	74.29 ↑1.71	40.62 ↓31.96	60.00 ↓12.58	16.35	17.99	17.57 ↓0.42	18.98 ↑0.99
GPT-4o	76.58	74.24 ↓2.34	83.33 ↑6.75	51.37 ↓25.21	55.00 ↓21.58	14.64	19.11	19.21 ↑0.10	18.91 ↓0.20
Gemini 3 Pro	84.51	83.87 ↓0.64	86.56 ↑2.05	59.62 ↓24.89	92.50 ↑7.99	14.49	18.75	17.70 ↓1.05	20.69 ↑1.94
Open-source models
Qwen3-VL-30B	71.99	72.87 ↑0.88	69.30 ↓2.69	36.20 ↓35.79	69.85 ↓2.14	44.02	18.79	18.65 ↓0.14	19.13 ↑0.34
Qwen3-VL-8B	70.28	73.16 ↑2.88	62.63 ↓7.56	34.64 ↓35.64	87.00 ↑16.72	50.32	18.32	18.37 ↑0.05	18.19 ↓0.13
Qwen3-VL-4B	71.26	71.91 ↑0.65	69.79 ↓1.47	38.91 ↓32.35	71.50 ↑0.24	54.37	15.74	15.55 ↓0.19	16.15 ↑0.41
InternVL3.5-4B	53.38	50.00 ↓3.38	62.52 ↑9.14	11.59 ↓41.79	11.00 ↓42.38	99.67	16.96	16.97 ↑0.01	16.95 ↓0.01
InternVL3.5-8B	52.59	53.16 ↑0.57	51.27 ↓4.32	20.93 ↓31.66	46.25 ↓6.34	78.69	17.27	17.56 ↑0.29	16.58 ↓0.69
LLaVA-OV-8B	53.26	55.73 ↑2.47	46.69 ↓6.57	11.52 ↓41.74	74.75 ↑21.49	95.27	17.30	17.40 ↑0.10	17.06 ↓0.24
Cosmos-Reason2-8B	64.73	68.92 ↑4.19	53.95 ↓10.78	26.54 ↓38.19	81.00 ↑16.27	70.24	13.86	13.83 ↓0.03	13.95 ↑0.09
Ours
Qwen3-VL-8B (SFT)	83.43	83.49 ↑0.06	83.33 ↓0.10	63.49 ↓19.94	82.50 ↓0.93	14.33	25.78	26.86 ↑1.08	23.61 ↓2.17
Ours w/o Stage 2 (Qwen)	65.04	69.00 ↑3.96	54.15 ↓10.89	25.95 ↓39.09	96.00 ↑30.96	42.98	11.97	13.08 ↑1.11	9.12 ↓2.85
Ours (Qwen)	85.72	85.03 ↓0.69	87.83 ↑2.11	67.35 ↓18.37	84.50 ↓2.98	7.53	27.45	28.14 ↑0.69	26.13 ↓1.32

ED1: Basic Causal Modeling · ED2: Long-Range Modeling · ED3: Belief-Conflict Sensitivity · ED4: Temporal Reversibility · ED5: Irreversible Micro-States Dynamics. Values in parentheses in the paper indicate evaluation-set sizes.