Determ Technology brings core 3D calibration and fusion capabilities from the Flowbit Platform to mobile phones and XR headsets, turning remote performances, creative workflows, and cultural exhibitions into live full-color 3D spaces that headset wearers can step into and explore.
Small-scale performances, craft making, and cultural exhibitions draw their appeal from physical presence and fine spatial detail. Traditional video livestreams confine viewers to a flat frame from a fixed camera angle, making it impossible to step closer or look from another side. Meanwhile, professional 3D spatial capture typically requires expensive dedicated hardware and complex on-site calibration, keeping it out of reach for everyday creative studios and live venues. How can readily available mobile devices turn a remote site into a live 3D space that viewers can walk into? Determ Technology recently demonstrated a technical exploration prototype that uses several everyday mobile devices, a local workstation, and Apple Vision Pro to reconstruct a remote physical space as a live full-color 3D scene, allowing viewers in the headset to walk around freely, step closer for a nearer look, and shift perspectives naturally as they turn their heads.
Behind this prototype are the multi-camera automatic calibration and centralized 3D fusion capabilities developed for the Flowbit, a real-time spatial data platform. This prototype extracts those two core capabilities and runs them across several mobile phones and an Apple Vision Pro headset, delivering a live multi-view full-color 3D space without dedicated 3D cameras or calibration boards. Downstream segmentation, annotation, and inference modules from the full Flowbit Platform design are not enabled in this prototype, though the central server retains the capacity to activate them on demand.
System pipeline diagram showing mobile devices (color + LiDAR depth) streaming over Wi-Fi (H.265, 12 Mbps per device) to a local workstation (NVIDIA GPU: multi-camera fusion + 3D point cloud rendering) and via NVIDIA CloudXR to Apple Vision Pro (3D display + head pose feedback)
Capture: LiDAR-Equipped iPhones
This test uses three iPhone 16 Pro devices. Each iPhone simultaneously captures color video and LiDAR depth and streams the data over Wi-Fi to a local workstation. Capture and encoding run on the iPhone itself, and the system sets no extra compute requirements for the device, so other LiDAR-equipped iPhones (iPhone 12 Pro and later) can connect as well:
Color video is compressed and transmitted using H.265 encoding.
Depth data and camera parameters are packaged into the same data stream rather than sent over separate connections, staying synchronized with the color video.
Each device automatically registers with the local workstation upon launch and starts streaming once registered, simplifying on-site setup for multi-person teams.
Fusion: Markerless Multi-Camera Calibration
Conventional mobile streaming setups typically rely on a single device and cover one viewpoint. While 360 or VR180 cameras capture a wider field of view or stereoscopic video, they lack true depth data and restrict viewers to a fixed-position, with only three degrees of freedom (3DoF) view, so they cannot walk through the scene, and they still require dedicated hardware.
The system tested by Determ Technology fuses streams from multiple active devices in real time into a single full-color 3D point cloud. Designed for lightweight, portable multi-camera setups, the system automatically initiates markerless joint calibration whenever two or more devices are online, executing three steps in sequence:
The system matches visual features across camera views to estimate each device's initial spatial pose.
Because pure visual reconstruction yields geometry without real-world dimensions, the system incorporates LiDAR depth measurements to restore true physical scale.
The system performs global optimization across the fused data to reduce alignment error between devices and merge the multiple viewpoints into a unified 3D scene.
The entire process requires no calibration boards and no manual measurement. As long as the camera views share sufficient overlapping regions, calibration and alignment complete automatically. In a small indoor performance or craft recording session, for instance, on-site staff do not need to survey the room; they simply place three phones around the stage or workbench with overlapping fields of view, and the system automatically completes spatial alignment and 3D capture fusion (as shown in the video, and the pictures below).
Rendering and Streaming: Head-Tracked 3D Scenes
Once the multiple data streams reach the local workstation, an NVIDIA GPU fuses them into a single full-color 3D point cloud and renders the scene in real time. The rendered output streams to Apple Vision Pro over NVIDIA CloudXR. The link runs both ways: head movement from the Apple Vision Pro wearer is sent back to the workstation in real time. As the viewer turns their head, steps closer, or walks around to the side, the workstation updates the rendered viewpoint accordingly. Viewers can move through the scene as if on site and choose where to look, leaving fixed-camera viewing behind.
Tested Configuration and Performance
Under the configuration used in this test, the capture and reconstruction pipeline runs at the following specifications:
Up to 3 iPhone 16 Pro devices connected simultaneously in a single scene.
3D point cloud fusion and update rate of 30 FPS.
Wireless transmission bitrate of 12 Mbps per iPhone.
These figures describe the capture and point cloud reconstruction stages. Headset display frame rate and end-to-end latency are separate metrics measured independently.
What's Next
This technical exploration prototype targets a range of creative and remote presentation scenarios. Potential applications include, but are not limited to, the following three areas:
Free-viewpoint remote viewing for small-scale music, theater, and puppetry performances.
3D documentation of creative processes such as woodcarving, clay sculpture, calligraphy, and painting.
Multi-angle spatial presentation for galleries and cultural exhibitions.
When full spatial data meets creative ideas, many more live scenarios and new forms of expression remain to be explored. Determ Technology is opening this prototype for collaboration with creators, media and arts organizations, spatial computing and visionOS developers, and researchers working in related fields.
For technical details or collaboration inquiries, contact contact@determtech.com.
NVIDIA and CloudXR are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Apple, iPhone, Apple Vision Pro, and visionOS are trademarks of Apple Inc., registered in the U.S. and other countries and regions.