Unity Plugin

What's new in 1.5.x: the streaming viewer is time-synchronized (multiple cameras aligned by capture timestamp); recording is always saved on the server (the on-device recording + FTP download feature was removed); and a bundled 3D segmentation detector can be started/stopped from inside Unity. Some screenshots below may still show the earlier 1.4 UI, but the steps are for current release.

1. Prerequisites (install once)

1.1 CUDA

• Version > 10.2. An NVIDIA GPU is required; RTX 3060 or newer is recommended (needed for the capture kernels, and used by the detector).

1.2 GStreamer

• Download from the GStreamer download page.

• Install both the runtime and development MSVC x86_64 packages. 1.18–1.24 all work.

• Note: install to the default path (C:\gstreamer or C:\Program Files\gstreamer).

• Note: add path\to\gstreamer\1.0\msvc_x86_64\bin to the system Path environment variable, then restart the PC.

1.3 Bonjour

• Download and install Bonjour print services - used to discover camera devices on the LAN.

1.4 Python (only if you use 3D segmentation)

• Python 3.10+ on PATH, for the bundled detector service (set up in §3).

2. Import the plugin

• Recommended: Unity 6000.0.37f1, Universal 3D (URP) project.

  • The plugin's shaders render via URP's SRPDefaultUnlit path, and the camera controls use the Input System package.

  • On other Unity versions (e.g. 2022) create a Universal 3D project, or a Built-In 3D project and install Universal RP from the Package Manager.

  • Install the Input System package (Window > Package Manager > Unity Registry). When prompted to enable the new input backend, press Yes to restart Unity.

  

• Assets > Import Package > Custom Package, then select the Flowbit Unity plugin .unitypackage.

  • When the TMP Importer window appears (on import or when first opening a scene), press Import TMP Essentials, then re-open the scene.

  • The plugin bundles its native DLLs under Assets/DetermFlowbitServer/Plugins/win64.

  

Note: the plugin requires unsafe code enabled:

• Edit > Project Settings > Player > Other Settings, tick Allow 'unsafe' Code.

3. Example scenes

3.1 Streaming example (time-synchronized)

1. Open FlowbitServerStreamingExample in the Scenes folder.

   

2. Select the FlowbitServer object. In the inspector set Config Path to Assets/DetermFlowbitServer/Config. This object also carries the Flowbit Server Sync Manager that time-aligns every camera by capture timestamp.

   

3. Under FlowbitServer there is one child object per camera (Camera_0123, Camera_0138, …), each with a Flowbit Server Sync Client component. On each, set Client Device ID to your real device id (e.g. SZVIDS-250214-923020-3002B0-38D5B8) and assign a Vertex Material from Materials/PointCloud - PointCloud_Vertex (points) or PointCloud_RectPlane (small squares). Add more cameras by duplicating a Camera object and changing its Client Device ID.

   

4. In Assets/DetermFlowbitServer/Config, open the <Client Device ID>/capture_config file for each device and set its Device IP to that camera's IP address.

5. Make sure the cameras are online, then press Play to view the synchronized point clouds. For a complete multi-camera group the device hosts must share a clock (PTP); a single camera always renders.

   

3.2 Manager example (runtime connect, stream, record, playback)

1. Open FlowbitServerManagerExample and press Play.

   

2. Press Scan Device to list the clients discovered on the LAN; each appears as a Client Info panel.

   

3. Use Connect / Disconnect to connect the selected clients (only panels with the Is Selected toggle on are affected). On connect, Manage Client Connect turns Yes.

   

4. Use Stream/Stop on the Streaming panel to start/stop point-cloud streaming. On stream, Capture Client Registered turns Yes and the cloud renders. Every streaming camera is automatically fed into the synchronizer, so multiple cameras render time-aligned.

   

5. Camera controls: W/A/S/D move, hold the right mouse button and drag to rotate, and the scroll wheel zooms.

6. Recording: press Start Record to record the selected streaming clients. In 1.5.x videos are always saved on the server, under Assets/<Video Folder Name>/<recording time>/ (default folder videos). Press Stop Record (the button toggles) to finish; streaming continues.

7. Playback: press Video File List to browse recordings by time, select one, and it appears on the Playback panel. Use Start/Pause/Stop to control playback; the slider shows progress.

   

8. Press Edit on a Client Info panel to change that device's exposure, depth cut range, and transform matrix (used to align the clouds from different devices in space).

   

9. Before you press Play again to stop the scene, if any device is connected/streaming, press Stop and Disconnect first.

   

3.3 3D segmentation (detection)

The manager scene has a 3D Segmentation panel (Panel_Detect) under the control panel. It runs live object segmentation on the synchronized clouds. Requires the detector set up in §3.

1. Press the Start Detector toggle - this launches the bundled Python detector (its .venv running on port 8901). The status line shows Detector: starting… then reachable (it loads the model first, a few seconds). The button label flips to Stop Detector; press it again to shut the service down (it is also killed automatically when you exit Play).

2. Press Segment Once for a single detection, or Continuous to detect continuously. The fused 3D instances are drawn as colored points floating on the live clouds - an object seen by several cameras fuses into one instance.

3. Press Clear to remove the drawn points and stop detecting.

Segmentation is optional: if the detector is not started (or setup.ps1 has not been run) the clouds keep streaming and only the overlay is unavailable. Tune the detector via Assets/DetermFlowbitServer/Config/detect_config.json (model, classes, Rotate 180 for upside-down cameras, voxel-fusion parameters) - edit and restart.