DvsenseDriver  1.3.0
The SDK for dvsense products.
Loading...
Searching...
No Matches
Fusion Camera User Guide

This guide covers DVS + RGB acquisition with DVSync and DVSyncPro. See DVSyncPro (Fusion Pro) Quick Start Guide for DVSyncPro USB host connections and its internal Linux system, and SDK 1.3.0 API Migration Guide for MIPI limitations and upgrade requirements. DVSLume, DVSLume-G1, DM3, and EVK4 use Use Dynamic Vision Camera.

Discovery, Opening, and Streaming

dvsense::DvsCameraManager manages all cameras. getCameraDescs() refreshes the device list; open a fusion device with openFusionCamera(desc.serial). USB DVSync reports product DVSync; the internal MIPI device reports DVSyncPro. Check the returned pointer: opening failures return null.

Register callbacks before start(FUSION_STREAM); a return value of 0 indicates success. Reference-board DVSyncPro currently accepts only FUSION_STREAM; DVS-only and APS-only start requests fail. Call stop() from a control thread rather than an acquisition callback.

Acquiring and Processing Fusion Data

dvsense::DvsApsFusionProccessor pairs APS frames in FIFO order with DVS groups closed by rising synchronization edges. Feed it using addDvsData(), addApsData() or addSharedApsData(), and addSyncSignal(). The latter receives the fusion camera's addSyncSignalCallback(), not an event camera trigger-in registration method.

The complete example below opens the first available fusion camera and connects all three inputs. Press Enter to stop. The fusion callback should hand matched data to a worker thread. See Samples/DvsenseSyncViewerSample/DVSyncViewer.cpp for actual rendering and processing.

#include <iostream>
#include <mutex>
#include <string>
#include "DvsenseDriver/camera/DvsCameraManager.hpp"
#include "DvsenseDriver/DataProcess/DvsApsFusionProccessor.hpp"
int main()
{
for (const auto &desc : manager.getCameraDescs()) {
if (desc.product == "DVSync" || desc.product == "DVSyncPro") {
camera = manager.openFusionCamera(desc.serial);
if (camera) break;
}
}
if (!camera) return 1;
std::mutex input_mutex;
[](dvsense::ApsFrame &frame, std::shared_ptr<dvsense::Event2DVector> events) {
// Enqueue a copy of frame and events for a worker; return promptly.
// Copying frame shares its owned pixels; do not retain the reference.
(void)frame;
(void)events;
});
const auto event_id = camera->addEventsStreamHandleCallback(
[&](dvsense::Event2D *begin, dvsense::Event2D *end) {
std::lock_guard<std::mutex> lock(input_mutex);
processor.addDvsData(begin, end);
});
const auto aps_id = camera->addApsFrameCallback(
[&](dvsense::ApsFrame frame) {
std::lock_guard<std::mutex> lock(input_mutex);
processor.addSharedApsData(frame);
});
const auto sync_id = camera->addSyncSignalCallback(
[&](const dvsense::EventTriggerIn &trigger) {
std::lock_guard<std::mutex> lock(input_mutex);
processor.addSyncSignal(trigger);
});
const int result = camera->start(dvsense::FUSION_STREAM);
if (result == 0) {
std::string line;
std::getline(std::cin, line);
}
camera->stop(dvsense::FUSION_STREAM);
camera->removeEventsStreamHandleCallback(event_id);
camera->removeApsFrameCallback(aps_id);
camera->removeSyncSignalCallback(sync_id);
processor.clear();
camera->destroy();
return result == 0 ? 0 : 1;
}
Contiguous BGR888 APS image with exposure timestamps.
Definition ApsTypes.h:34
Pair APS frames with completed DVS synchronization groups in FIFO order.
Definition DvsApsFusionProccessor.hpp:39
void addSyncSignal(const EventTriggerIn &trigger_in)
Add a synchronization signal and close the current DVS event group on a rising edge.
Definition DvsApsFusionProccessor.hpp:210
void addSharedApsData(const ApsFrame &frame)
Add an APS frame by sharing its pixel storage without a deep copy.
Definition DvsApsFusionProccessor.hpp:187
void addDvsData(const Event2D *begin, const Event2D *end)
Append DVS events to the current synchronization group.
Definition DvsApsFusionProccessor.hpp:138
void removeFusionDataCallback()
Remove both the non-owning and shared fusion-data callbacks.
Definition DvsApsFusionProccessor.hpp:121
void addSharedFusionDataCallback(SharedDsFusionDataCallback cb)
Register a callback that shares the fused DVS event buffer.
Definition DvsApsFusionProccessor.hpp:113
void clear()
Clear all queued APS frames, completed DVS groups, and the current DVS event buffer.
Definition DvsApsFusionProccessor.hpp:72
Camera Manager class.
Definition DvsCameraManager.hpp:36
FusionCameraDevice openFusionCamera(Serial serial)
Open and initialize a discovered DVSync or DVSyncPro fusion camera.
std::vector< CameraDescription > getCameraDescs()
Get the Camera Descriptions vector.
Definition EventTypes.hpp:22
Definition EventTypes.hpp:39
std::shared_ptr< FusionCamera > FusionCameraDevice
Definition DvsCameraManager.hpp:30
@ FUSION_STREAM
Definition DvsCameraUtils.hpp:47

The example serializes all inputs with input_mutex. Fusion callbacks execute synchronously within input calls while the processor queue mutex is held. Do not call clear() or feed APS/sync input into the same processor from those callbacks.

Data Lifetime and Preview Policy

The ordinary event callback's [begin, end) range is valid only during the callback; copy events for asynchronous use. Camera-level addSharedEventsStreamHandleCallback() permits retaining the shared buffer. The processor's addSharedFusionDataCallback() permits retaining the matched event shared_ptr, but its APS reference remains callback-scoped: copy the ApsFrame object before passing it to another thread.

Copies of ApsFrame share pixels; clone() performs a deep copy. addApsData() always deep-copies input pixels. addSharedApsData() does not; storage must be owned by the frame or kept alive by the caller for the entire queued lifetime.

Default construction preserves complete synchronization groups. For preview, DvsApsFusionProccessor(200000, true) limits events per group and discards older queued input to reduce latency. It may drop data and is unsuitable for complete-data analysis. See SDK 1.3.0 API Migration Guide.

Spatial Alignment

Optical paths cause spatial offsets between RGB images and DVS events. Read available device calibration using readCalibrationParam() and check its result, or load existing JSON calibration parameters. Do not treat the default mapping as device calibration when the device supplies no parameters.

#include "DvsenseDriver/Calibration/Calibrator.hpp"
// camera and aps_frame have already been acquired.
if (camera->readCalibrationParam(parameters)) {
calibrator.loadCalibrationParam(parameters);
dvsense::ApsFrame mapped = calibrator.mapApsToDvs(aps_frame);
}
Spatial APS/DVS calibration using affine mappings and cached APS pixel indexes.
Definition Calibrator.hpp:32
bool loadCalibrationParam(CalibratorParameters &param)
Definition Calibrator.hpp:49
ApsFrame mapApsToDvs(ApsFrame &src_frame)
Definition Calibrator.hpp:78
Definition TypeUtils.hpp:35

mapApsToDvs() returns a new DVS-resolution image. mapApsToDvsBgr() can write into a reusable BGR888 output buffer of at least dvs_cols * dvs_rows * 3 bytes. mapDvsToAps(begin, end) modifies event coordinates in place and does not return an image; copy events first to retain their original coordinates. Input images must match calibrated APS dimensions. Out-of-image mappings are filled black.

Separate DVS, APS, and Synchronization Callbacks

Use addEventsStreamHandleCallback(), addApsFrameCallback(), and addSyncSignalCallback() directly when needed. Each returns an ID for its matching removal method. APS callbacks accept ApsFrame by value or const ApsFrame &, not a mutable ApsFrame &.

Keep acquisition callbacks short and treat shared input as read-only. Stop acquisition and let SDK callback threads finish before removing callbacks, clearing the processor, and releasing data held by workers.

Recording and Playback

After starting acquisition, call startRecording(out_file_path, filename, FUSION_STREAM) and stopRecording(FUSION_STREAM). Supply a filename without an extension: DVS streams are recorded as raw and APS video as mp4. See the SDK FusionFileReader sample and File Reading for recording layout and playback.

Python and On-Camera WebRTC

Python uses DvsApsFusionProccessor.connect_camera(camera) to connect streams internally in C++ and Calibrator.render_fusion() to map and overlay events. Retain the connection object, then stop the camera before calling disconnect(). See python/samples/dvsync_camera_sample.py.

The optional Linux DVSyncPro WebRTC preview requires separately installed dependencies and a separate build. Its executable is omitted from the default sample package. See DVSyncPro (Fusion Pro) Quick Start Guide.