EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy

Abstract

Humans constantly reason about 3D proximity, the relations between their body and surrounding objects, to guide perception and action in daily life. Whether multimodal large language models (MLLMs) can perform such embodied 3D reasoning remains unclear. To this end, we introduce EgoProx, a benchmark for egocentric 3D proximity reasoning. We organize our tasks along a cognitive chain, covering intention, exploration, exploitation, and chain-of-actions reasoning. We also design an agent based data engine that produces diverse and consistent QA pairs at scale. We benchmark prevailing MLLMs on EgoProx and conduct additional analyses with dataset specific and task specific instruction tuning. We observe large cross-domain gains, indicating that current MLLMs contain some spatial knowledge; however, they still struggle to effectively leverage it for spatial reasoning VQA.

Key Contributions

Cognitive Hierarchy of EgoProx

EgoProx organizes egocentric 3D proximity reasoning as a cognitive hierarchy. Intention captures immediate gaze and head-orientation shifts toward goals; Exploration studies navigation steps toward objects or locations; Exploitation evaluates upcoming human-object interactions; and Chain of Actions asks models to reason over multi-step future behavior and the spatial relationships among action locations. Approximate proximity models coarse metric rotations and translations, while relative proximity captures directional relationships.

Four-Level Egocentric 3D Proximity Reasoning

The benchmark requires models to use long-term contextual cues, spatial dependencies, and action-state changes from first-person videos, moving from local intention cues toward higher-order action chains.

Task Distribution

EgoProx contains 2,405 VQA samples collected from two complementary egocentric datasets: 1,016 samples from Aria Digital Twin and 1,389 samples from EgoExo4D. The benchmark covers a broad spectrum of proximity reasoning scenarios and is organized according to a four-level cognitive hierarchy consisting of Intention (30.27%), Exploration (15.71%), Exploitation (46.37%), and Chain of Actions (7.65%).

Intention: 30.27%

Exploration: 15.71%

Exploitation: 46.37%

Chain of Actions: 7.65%

Agentic Data Engine

Overview of our agent-based data construction pipeline. The agent first identifies salient moments with an interaction- and fixation-based sampler, then uses the 3D Analysis Toolset to extract spatial cues such as object positions, gaze targets, occupancy maps, and action chains. It then invokes the Spatial Calculator to derive 3D distances, orientations, and proximity relations, producing structured 3D proximity ground truth. Final benchmark question-answer pairs are compiled through necessary post-processing.

From Video to Structured 3D Proximity QA

The Agentic Data Engine converts egocentric video streams and raw annotations into scalable 3D proximity reasoning VQA data.

Leaderboard

Red cells indicate the best score among evaluated MLLMs for each metric. Human Level is shown as a reference.

Even strong proprietary models remain far below human performance, especially on Chain of Actions. The results also show that model scaling alone brings limited gains for 3D proximity reasoning.

Cross-Category Experiment

Red cells indicate the best score in each evaluation column.

Training on a small amount of data from one category often improves performance on other categories. Intention tuning transfers especially well, supporting the paper's cognitive hierarchy: intentional cues guide location reasoning and action-conditioned 3D reasoning. Exploration tuning also improves chain-of-actions metrics because navigation supervision provides useful signals about action locations.

Cross-Dataset Experiment

Red cells indicate the best score in each evaluation column.

Fine-tuning on one source dataset improves proximity reasoning on the other despite the recording-domain gap between ADT and EgoExo4D. The smaller gain on ADT Exploration is consistent with the data-source design: EgoExo4D training data does not contain exploration-type questions.

Qualitative Results

Intention

Exploration

Exploitation

Chain of Actions

BibTeX

@misc{li2026egoproxevaluatingmllmsegocentric,
      title={EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy}, 
      author={Jinzhao Li and Yinuo Chen and Dongxu Piao and Panwang Pan and Yifan Yu and Dong Wang and Honglei Yan and Liang Yue and Shaofei Wang and Yixin Chen and Siyuan Huang and Miao Liu},
      year={2026},
      eprint={2605.24456},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.24456}, 
}