Set Up the Detection System

How 3D segmentation works

The pipeline has two halves:

1. 2D detection. Each camera's colour frame goes to a small, standalone detector service that returns per-pixel instance masks over HTTP.

2. 3D fusion (C++, in-process). The flowbitserverdetect module lifts those masked pixels into a shared world voxel grid and returns typed world-space instances the viewer overlays on the cloud. Detection runs on one synchronized frame group, so an object seen by several cameras fuses into a single instance instead of ghosting across views.

The SDK is detector-agnostic. It never launches, imports or links the detector. It only calls it over HTTP at the address you set in Detector URL, so any service that answers the Set Up the Detection System - detector HTTP contract will drive segmentation, and the detector can run on this PC or on another host.

Segmentation is optional. If the detector is not running, the viewer still renders the live cloud and reports segmentation unavailable.

Download and install the detector

WARNING

The detector is a separate download and is not part of the SDK export. This is deliberate: it keeps the SDK free of any third-party detection model, and nothing model-specific ships in the SDK tree.

Download the detector flowbit-server-SDK-1.5.0-detector.tar, unpack it, then follow its own INSTALL.md. In outline, the setup script creates a private Python .venv, installs the dependencies, and fetches the model weights on this machine, which is where you accept the licence terms of the model you choose.

# Windows
cd detector
./setup.ps1

# Windows
cd detector
./setup.ps1

# Windows
cd detector
./setup.ps1

# Ubuntu
cd

# Ubuntu
cd

# Ubuntu
cd

The detector's INSTALL.md covers GPU and CPU installs, which models are available, and how the weights are fetched.

Run the detector service

Leave it running in its own terminal. It listens on port 8901.

# Windows
cd detector
.\.venv\Scripts\python.exe -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Windows
cd detector
.\.venv\Scripts\python.exe -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Windows
cd detector
.\.venv\Scripts\python.exe -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Ubuntu
cd detector
./.venv/bin/python -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Ubuntu
cd detector
./.venv/bin/python -m uvicorn app.api:app --host 127.0.0.1 --port 8901

# Ubuntu
cd detector
./.venv/bin/python -m uvicorn app.api:app --host 127.0.0.1 --port 8901

Verify

• http://127.0.0.1:8901/health returns {"status":"ok"}

The first detection loads the model lazily and takes a few seconds; later requests are fast.

The detector can share the SDK's GPU, or run on a separate machine. To run it elsewhere, point Detector URL at that host.

Run the segment viewer

viewer_opengl_segment is built with the other examples, with no extra flag. It takes the same fixed-device-list arguments as the streaming viewer.

# two named cameras
viewer_opengl_segment --devices=<id-1>,<id-2> --config=<prefix>/config

# or from a client-list JSON
viewer_opengl_segment --clients=<prefix>/example/streaming_client_info_example.json --config

# two named cameras
viewer_opengl_segment --devices=<id-1>,<id-2> --config=<prefix>/config

# or from a client-list JSON
viewer_opengl_segment --clients=<prefix>/example/streaming_client_info_example.json --config

# two named cameras
viewer_opengl_segment --devices=<id-1>,<id-2> --config=<prefix>/config

# or from a client-list JSON
viewer_opengl_segment --clients=<prefix>/example/streaming_client_info_example.json --config

Configure detect_config.json

Loaded at startup. Edit and restart, no rebuild. Keys prefixed with ? are inline documentation.

Detector HTTP contract

The SDK depends only on this wire shape, not on any particular model, so you can supply your own detector. Any service that answers these two endpoints can drive segmentation.

POST /api/detect/run takes multipart/form-data: a text_prompt field holding the class list, with the classes separated by dots or commas, plus one image_<device_id> JPEG per device. It returns:

{
  "detections": {
    "<device_id>": [
      { "label": "person", "confidence": 0.94, "bbox": [0, 0, 0, 0], "polygon": [] }
    ]
  }
}

{
  "detections": {
    "<device_id>": [
      { "label": "person", "confidence": 0.94, "bbox": [0, 0, 0, 0], "polygon": [] }
    ]
  }
}

{
  "detections": {
    "<device_id>": [
      { "label": "person", "confidence": 0.94, "bbox": [0, 0, 0, 0], "polygon": [] }
    ]
  }
}

bbox is [x1, y1, x2, y2]. polygon may be omitted, but without it the 3D voxel path has nothing to fuse. Every detection you want in 3D must carry one.

GET /health returns {"status":"ok"} once the service is ready. The viewer polls this.

Troubleshooting

Build, link, runtime and streaming problems are on Use the SDK in your own project.

Next steps

Use the SDK in your own project