Set up the detection system

Live 3D object segmentation on top of the synced multi-device cloud, and what to do when things go wrong.

How 3D segmentation works

2D detection (Python). Each camera's color frame goes to a small, standalone detector service - FastAPI plus Ultralytics YOLO11-seg - that returns per-pixel instance masks.
3D fusion (C++, in-process). The flowbitserverdetect module lifts those masked pixels into a shared world voxel grid and returns typed world-space instances the viewer overlays on the cloud. Detection runs on one synchronized frame group, so an object seen by several cameras fuses into a single instance instead of ghosting across views.

The detector is the only Python piece and is optional. If it is not running, the viewer still renders the live cloud and reports segmentation unavailable.

Full notes are in <prefix>/detector/INSTALL.md. This page is the summary.

GPU requirements for the detector

The detector is a separate PyTorch process, so the SDK's RTX 3060 rule does not bind it.

Aspect	Requirement
GPU needed?	Optional. Run setup with `TORCH_INDEX_URL=""` (`-TorchIndexUrl ""` on Windows) for a CPU-only install — it still works, just slowly, at seconds per frame rather than milliseconds
Recommended GPU	Any NVIDIA CUDA GPU with compute capability ≥ 7.5 (RTX 20-series / T4 or newer) for the default wheels (Windows `cu128`, Ubuntu `cu130`). Older cards need an older wheel — for example `-TorchIndexUrl https://download.pytorch.org/whl/cu121`
VRAM	`yolo11s-seg` is small — inference fits in about 2 GB; 4 GB or more is comfortable
System CUDA toolkit	Not required. PyTorch bundles its own CUDA runtime. Only the NVIDIA driver must be recent enough for that wheel's CUDA (`cu128` needs driver ≥ ~550)
Shares the SDK's GPU?	Yes. It can also run on a separate machine — point `Detector URL` at that host

One-time detector setup

Installs a private .venv with CUDA PyTorch and Ultralytics, and fetches the YOLO11-seg weight.

[!NOTE] The weights are AGPL-3.0 and are therefore not shipped. Running setup fetches them locally and accepts the AGPL terms on this machine.

# Windows
cd

# Windows
cd

# Windows
cd

# Ubuntu
cd

# Ubuntu
cd

# Ubuntu
cd

[!TIP] If PowerShell refuses with "File …\setup.ps1 cannot be loaded. The file is not digitally signed", run this first, in the same terminal:
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

setup.ps1 installs a CUDA build of PyTorch, default wheel index cu128. Override with -TorchIndexUrl "https://download.pytorch.org/whl/cu126", or force CPU-only with -TorchIndexUrl "".

Run the detector service

Leave it in its own terminal. It listens on port 8901.

# Windows
cd <prefix>\detector
.\.venv\Scripts\python.exe -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Windows
cd <prefix>\detector
.\.venv\Scripts\python.exe -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Windows
cd <prefix>\detector
.\.venv\Scripts\python.exe -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Ubuntu
cd <prefix>/detector
./.venv/bin/python -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Ubuntu
cd <prefix>/detector
./.venv/bin/python -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Ubuntu
cd <prefix>/detector
./.venv/bin/python -m uvicorn app.api:app --host 127.0.0.1 --port 8901

Verify

[ ] http://127.0.0.1:8901/health returns {"status":"ok"}

The first detection loads the weights lazily and takes a few seconds; later requests are fast.

Run the segment viewer

viewer_opengl_segment is built with the other examples — no extra flag. It takes the same fixed-device-list arguments as the streaming viewer.

# two named cameras
viewer_opengl_segment --devices=<id-1>,<id-2> --config=<prefix>/config

# or from a client-list JSON
viewer_opengl_segment --clients=<prefix>/example/streaming_client_info_example.json --config

# two named cameras
viewer_opengl_segment --devices=<id-1>,<id-2> --config=<prefix>/config

# or from a client-list JSON
viewer_opengl_segment --clients=<prefix>/example/streaming_client_info_example.json --config

# two named cameras
viewer_opengl_segment --devices=<id-1>,<id-2> --config=<prefix>/config

# or from a client-list JSON
viewer_opengl_segment --clients=<prefix>/example/streaming_client_info_example.json --config

Key	Action
`G`	Segment once, on demand
`H`	Toggle continuous segmentation
`R` / `S`	Start / stop recording

Configure `detect_config.json`

Loaded at startup - edit and restart, no rebuild. Keys prefixed with ? are inline documentation.

Key	Default	Meaning
`Detector URL`	`http://127.0.0.1:8901`	Base URL of the Python service. Point it elsewhere to run the detector remotely
`Model`	`yolo11_seg`	Must attach masks for the voxel path. A box-only model produces no masks and fusion sees nothing
`Classes`	`person, chair, table`	Whitelist of COCO classes run and returned. Empty means detector defaults
`Rotate 180`	`false`	Default for all cameras. `true` for upside-down mounts — the detector rotates upright and fusion un-rotates the masks
`Rotate 180 Overrides`	`{}`	Per-device override for a mixed-orientation rig: `{ "<device-id>": true }`
`Trigger Mode`	`on_demand`	`on_demand` (press `G`) or `continuous`
`HTTP Timeout Ms`	`10000`	Timeout for one detector request
`Voxel Seg`	—	Fusion tunables: `voxel_size_m` (0.04), `min_voxels` (30), `use_calibrated_floor` (true), `depth_band_m` (0.60)

Troubleshooting

Symptom	Likely cause	Fix
Nothing detected; log warns "detections carried no mask"	A box-only model is running	Set `Model` to `yolo11_seg`
0 detections on a scene that clearly has objects	Camera mounted upside-down	Set `"Rotate 180": true`
Some cameras detect, others do not after `Rotate 180`	Mixed mounting orientations	List the differing cameras in `Rotate 180 Overrides`
Detector logs N detections but the viewer logs `0 instance(s)`	3D fusion discarding masks, usually floor removal cutting low objects	Set `"use_calibrated_floor": false`; try lowering `min_voxels` or widening `depth_band_m`
"Segmentation unavailable" at launch	Detector not started, or wrong `Detector URL`	Start the detector or fix the URL, then press `G`

Build, link, runtime and streaming problems are on Use the SDK in your own project.

Next steps

Use the SDK in your own project

Run examples