Fusion Camera User Guide

This section describes how to use DVSync through the API. DVSync is the only DVS + RGB fusion camera among the four current DVSense products. For DVSLume, DVSLume-G1, and DM3, refer to the event camera tutorial.

• Open Camera
• Get and Process Fusion Data
• Time Alignment
• Spatial Alignment
• Get Asynchronous Data
• Save Data

Example code can be found in the Sample directory under the SDK installation path.

---

Open Camera

In DvsenseDriver, all cameras are managed through the dvsense.DvsCameraManager class.

Find Cameras

// find all cameras
dvsense::DvsCameraManager cameraManager;
std::vector<dvsense::CameraDescription> cameraDescs = cameraManager.getCameraDescs();

dvsense.CameraDescription includes manufacturer, product model, and serial number.

You can use the product model to distinguish between an event camera or a fusion camera.

Open Fusion Camera

// Select and open DVSync.
dvsense::FusionCameraDevice fusion_camera;
for (const auto &cameraDesc : cameraDescs) {
    if (cameraDesc.product == "DVSync") {
        fusion_camera = cameraManager.openFusionCamera(cameraDesc.serial);
        break;
    }
}

dvsense::FusionCameraDevice is a convenience wrapper for dvsense::FusionCamera:

typedef std::shared_ptr<FusionCamera> FusionCameraDevice;

Start and Stop Streaming

By default, cameras do not start streaming immediately after being opened; you must start manually.

⚠️ Note: Before starting streaming, please register asynchronous event callbacks.

fusion_camera->start();    // Start streaming
fusion_camera->stop();     // Stop streaming

Supported stream modes:

• DVS_STREAM
• APS_STREAM
• FUSION_STREAM (Alpha version only supports fusion stream)

---

Get and Process Fusion Data

Since RGB and event data (DVS) are separate streams, time alignment and spatial alignment are required.

We provide sample code in:

• DvsenseDriver/include/DvsenseDriver/DataProcess/DvsApsFusionProccessor.hpp
• Samples/DvsenseSyncViewerSample/DVSyncViewer.cpp

Example

// Load calibration parameters from the device
dvsense::Calibrator calibrator;
dvsense::CalibratorParameters cali_param;
if(camera->readCalibrationParam(cali_param)) {
    dvsense::paramToJsonFile(cali_param, "./calibration.json");
    calibrator.loadCalibrationParam(cali_param);
}
 
dvsense::DvsApsFusionProccessor fusion_processor; // Fusion processor instance
 
// Register fusion callback
fusion_processor.addFusionDataCallback(
    [&sync_displayer](dvsense::ApsFrame &frame, const dvsense::Event2D *begin, const dvsense::Event2D *end) { 
        dvsense::ApsFrame aps_to_dvs_frame = calibrator.mapApsToDvs(frame);
        sync_displayer.processApsFrame(aps_to_dvs_frame); 
        sync_displayer.fusionDvsToAps(begin, end); 
    });

---

Time Alignment

The dvsense::DvsApsFusionProccessor class handles automatic time synchronization and delivers results via addFusionDataCallback.

You can also implement your own synchronization logic if needed.

---

Spatial Alignment

Due to optics, RGB and DVS images are not perfectly aligned. Calibration parameters are stored in the device.

We provide the Calibrator class for mapping and compensation, offering both APS→DVS and DVS→APS mapping.

Load Calibration Parameters

camera = cameraManager.openFusionCamera(open_camera_serial);
dvsense::Calibrator calibrator_;
dvsense::CalibratorParameters cali_param;
camera->readCalibrationParam(cali_param);    // Read preset parameters
calibrator_.loadCalibrationParam(cali_param); // Load calibration

APS to DVS Mapping

dvsense::ApsFrame aps_to_dvs_frame = calibrator_.mapApsToDvs(aps_frame);

DVS to APS Mapping

cv::Mat dvs_to_aps_frame = calibrator_.mapDvsToAps(dvs_frame);

See sample: Samples/DvsenseSyncViewerSample/DVSyncViewer.cpp

---

Get Asynchronous Data

Fusion cameras currently only support asynchronous data retrieval.

DVS Event Callback

Use dvsense.FusionCamera.addEventsStreamHandleCallback to access events as they arrive:

camera->addEventsStreamHandleCallback([&img](const dvsense::Event2D* begin, const dvsense::Event2D* end) {
    for (auto it = begin; it != end; ++it) {
        img.at<cv::Vec3b>(it->y, it->x) = (it->polarity) ? color_on : color_off;
    }
});

⚠️ Warning: Keep callbacks lightweight to avoid data backlog and crashes.

You can also add/remove multiple callbacks using callback IDs:

auto cb_id = camera->addEventsStreamHandleCallback([](const dvsense::Event2D* begin, const dvsense::Event2D* end) {
    // Process events
});
 
camera->removeEventsStreamHandleCallback(cb_id);

---

RGB Frame Callback

Use dvsense.FusionCamera.addApsFrameCallback to access RGB frames:

aps_image_ = cv::Mat(aps_height_, aps_width_, CV_8UC3);
camera->addApsFrameCallback([&aps_image_](dvsense::ApsFrame &aps_frame) {
    std::memcpy(aps_image_.data, aps_frame.data(), aps_frame.getDataSize());
});

⚠️ Warning: Keep callback execution time short.

You can add/remove multiple callbacks:

auto cb_id = camera->addApsFrameCallback([](dvsense::ApsFrame &) {
    // Process frame
});
 
camera->removeApsFrameCallback(cb_id);

---

Save Data

After starting streaming, you can save fusion camera data using:

• FusionCamera.startRecording
  
• FusionCamera.stopRecording

Event streams are saved as .raw files, video streams as .mp4 files.

camera->startRecording(out_file_path, filename);  // Filename without extension
camera->stopRecording();

---

Next Section

• Dynamic Vision Camera User Guide