PhyModels
Physical-world models perform grounded perception, reasoning, prediction, or decision-making about physical scenes.
NeurIPS 2026 · Evaluations & Datasets Track
Configure physical-world tasks, test methods, defenses, and victim models or agents. Compare functional robustness and collect reusable test datasets for future alignment.

Adversarial stress testing is the main focus. The same interfaces also support non-adversarial input perturbations and obfuscations without an attack objective.
Physical-world models perform grounded perception, reasoning, prediction, or decision-making about physical scenes.
Physical-world agents use these models with planning or tool use to carry out grounded tasks.
Scope of the current paper. The study focuses on vision-language PhyModels and tool-using PhyAgents built on vision-language backbones. Evaluation uses scene images, questions, and task-level responses. These terms describe physical-world task roles across model architectures.
Project repository: github.com/loopleap-lab/RedPhys



Failures are wrong actions and judgments in grounded tasks, not harmful text.
A scenario agent builds each target from its scene, the same for every method.
One dataset, two metrics and a black-box threat model for attacks and defenses.
Driving, manipulation, grasping, physics, egocentric, diagnostics and scene QA.
832 physical-world cases, one-shot black-box transfer. ASR is the attack success rate.
Models that answer clean images less accurately tend to be easier to attack, though not deterministically.
Lower is better. BlueSuffix cuts PromptInject and FreqPure cuts FOA. PAD changes almost nothing, and no defense lowers FigStep.
The toolkit and the benchmark are out. An automatic platform is in progress, then agents that research new attacks and defenses.
One library to run, defend against and score red-teaming methods.
Agents build a target for each scene; the benchmark scores 13 PhyModels.
Fully automatic. The user picks a model or a use case; agents do the rest.
Agents design new attacks and defenses and test them.
Yifei Zhao, Qian Lou, and Mengxin Zheng · LoopLeap Lab
RedPhys is accepted at NeurIPS 2026 (Evaluations & Datasets Track). The citation below reflects the accepted conference paper; the arXiv version remains available. Download BibTeX.
@inproceedings{zhao2026redphys,
title={{RedPhys}: An Extensible Red-Teaming Platform for Physical-World Models and Agents},
author={Zhao, Yifei and Lou, Qian and Zheng, Mengxin},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2026},
note={Accepted, Evaluations and Datasets Track},
doi={10.48550/arXiv.2606.23892},
url={https://arxiv.org/abs/2606.23892}
}