Continuum robots are advancing bronchoscopy procedures by accessing complex lung airways and enabling targeted interventions. However, their development is limited by the lack of realistic training and test environments: Real data is difficult to collect due to ethical constraints and patient safety concerns, and developing autonomy algorithms requires realistic imaging and physical feedback.
We present ROOM (Realistic Optical Observation in Medicine), a comprehensive simulation framework designed for generating photorealistic bronchoscopy training data. By leveraging patient CT scans, our pipeline renders multi-modal sensor data including RGB images with realistic noise and light specularities, metric depth maps, surface normals, optical flow and point clouds at medically relevant scales.
We validate the data generated by ROOM through monocular depth estimation experiments, demonstrating diverse challenges that state-of-the-art methods must overcome to transfer to these medical settings. Furthermore, we show that the data produced by ROOM can be used to fine-tune existing depth estimation models to overcome these challenges, enabling downstream applications such as vision-based navigation with Model Predictive Control.
Accurate continuum robot dynamics with realistic tissue interactions
Path-traced imagery with endoscopic lighting and sensor noise modeling
RGB, depth, normals, optical flow, and point clouds synchronized
Automated pipeline from CT scans to anatomically accurate 3D models
Our pipeline consists of four main stages: (1) Medial Axis Extraction from segmented CT lung models, (2) Automated Sampling along skeletal branches with higher density at bifurcations and high-curvature regions, (3) Data Synthesis generating synchronized multi-modal sensor streams, and (4) Sensor Noise Modeling applying realistic noise characteristics matching real bronchoscopy imagery through frequency-domain analysis.
Left: Real bronchoscopy data captured from a continuum robot showing specular highlights from wet mucosal surfaces and directional lighting. Center: ROOM's photorealistic rendering using Blender's path tracing with Principled BSDF shaders, accurately reproducing tissue surface properties and lighting conditions. Right: Naive PyBullet-based rendering lacking photorealistic materials and lighting.
Interactive demonstrations of continuum robot navigation through airways using ROOM-generated data. The simulator provides realistic physics-based interactions and photorealistic rendering for training autonomous navigation algorithms.
Models fine-tuned on ROOM synthetic data show significant improvements on real bronchoscopy images, validating our simulation framework's effectiveness for domain adaptation. The comparison demonstrates clear performance gains across multiple state-of-the-art depth estimation methods.
Demonstration of Model Predictive Control (MPC) for vision-based navigation through airways. The planner uses depth predictions from models trained on ROOM data to generate collision-free paths, showcasing the practical application of our simulation framework for autonomous bronchoscopy.
If you find our work useful in your research, please consider citing:
@article{esposito2025room,
title={ROOM: A Physics-Based Continuum Robot Simulator for Photorealistic Medical Datasets Generation},
author={Esposito, Salvatore and Mattamala, Matías and Rebain, Daniel and
Zhang, Francis Xiatian and Dhaliwal, Kevin and Khadem, Mohsen and
Ramamoorthy, Subramanian},
journal={arXiv preprint arXiv:2025.xxxxx},
year={2025}
}