Skip to content

Plumerai Video Intelligence C++ API

This document describes the C++ API for the Plumerai Video Intelligence library for videos.

The C++ API header files are self-documented. The main entrypoint is the plumerai::VideoIntelligence class which provides access to all functionality of the Plumerai software. It has a process_frame function that needs to be executed on each input frame. The various intelligence features such as object detection and familiar face identification are available through different interfaces, listed here.

Please refer to the minimal examples for example code to get started.

The API is re-entrant, i.e. you can instantiate several VideoIntelligence objects in different threads and use them independently. However, using the same instance from different threads at the same time is not supported.

VideoIntelligence

VideoIntelligence

VideoIntelligence(int height, int width);

Initializes a new Video Intelligence object.

This needs to be called only once at the start of the application.

Arguments:

  • height: The height of the input image in pixels.
  • width: The width of the input image in pixels.

process_frame

template <ImageFormat image_format>
ErrorCode process_frame(const ImagePointer<image_format> image_data,
                        float delta_t = 0.f);

Process a single frame from a video sequence.

Make sure the image is right side up. When it is upside down it can still work but accuracy is significantly degraded.

Arguments:

  • image_format: A template parameter which must be one of the ImageFormat enum values.
  • image_data: A pointer to the image data in the form of an ImagePointer helper struct.
  • delta_t: The time in seconds it took between this and the previous video frame (1/fps). If set to 0, then the system clock will be used to compute this value.

Returns:

  • An error code of type ErrorCode. See that enum for more details.

single_image

template <ImageFormat image_format>
ErrorCode single_image(const ImagePointer<image_format> image_data,
                       int height = 0, int width = 0);

Process a single image not part of a video sequence.

This should not be used for video data. It can be used for face enrollments from a set of images. The object detection box id values obtained after calling single_image are not related to those generated through process_frame or through other calls to single_image.

Arguments:

  • image_format: A template parameter which must be one of the ImageFormat enum values.
  • image_data: A pointer to the image data in the form of an ImagePointer helper struct.
  • height: The height of the input image in pixels. If height = 0 the height set in the constructor will be used.
  • width: The width of the input image in pixels. If width = 0 the width set in the constructor will be used.

Returns:

  • An error code of type ErrorCode. See that enum for more details.

set_night_mode

ErrorCode set_night_mode(bool night_mode);

Configure the video intelligence algorithm for either day mode color videos (default) or night mode IR videos.

This configures the video intelligence algorithm for optimal performance on day mode color data (default) or on night mode IR data.

After switching from day to night mode or back, the motion detection algorithm will need a couple of video frames to stabilize, so the motion-grid will not be immediately available.

This function does not have to be called before every frame, only when switching from RGB to IR video data or back.

Arguments:

  • night_mode: Set to true for night mode or false for day mode.

Returns:

  • Returns ErrorCode::SUCCESS on success.

store_state

ErrorCode store_state(std::vector<std::uint8_t>& state) const;

Store the current state of the algorithm to a byte array.

This function can be used when processing a video in chunks, doing different chunks at different times or on different machines. The state can be restored by calling restore_state with the data returned by store_state. When the library is built with support for familiar face identification, the state includes the face library. Constraints:

  • The delta_t parameter of process_frame can not be left to zero after restoring a previous state.

  • If familiar face identification is enabled, the state can only be stored and restored when not enrolling. Arguments:

  • state: A vector to store the serialized state in.

Returns:

  • Returns ErrorCode::SUCCESS on success.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);
std::vector<std::uint8_t> state;
auto error_code = pvi.store_state(state);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: store_state returned %s\n",
         plumerai::error_code_string(error_code));
}

restore_state

ErrorCode restore_state(const std::vector<std::uint8_t>& state);

Restore the state of the algorithm from a byte array.

See store_state for more information. The user must ensure that the height and width of the current object match the height and width of the state that is being restored.

Arguments:

  • state: A vector containing the serialized state.

Returns:

  • Returns ErrorCode::SUCCESS on success. Returns ErrorCode::STATE_CORRUPT or ErrorCode::STATE_SETTINGS_MISMATCH on error.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);
// The state as obtained by the store-state API, e.g. loaded from memory
std::vector<std::uint8_t> state = ...;
auto error_code = pvi.restore_state(state);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: restore_state returned %s\n",
         plumerai::error_code_string(error_code));
}

debug_next_frame

ErrorCode debug_next_frame(const std::uint8_t** debug_data_buffer,
                           std::size_t* debug_data_size);

Enable debug mode for the next frame.

The next time process_frame is called, this will dump the input image as well as internal data and final results to binary blob, meant to be stored to a file. This file can then be shared with Plumerai support for further analysis. These files contain uncompressed image data and can become several megabytes large.

The user will receive a pointer to the data, but this will only be available after the next call to process_frame, see the example code. The data is only valid for one frame, and will be invalidated after the second call to process_frame, or another call to debug_next_frame.

This function is only available on platforms that support dynamic memory allocation.

Arguments:

  • debug_data_buffer: An output parameter that receives a pointer to the debug data, after the next call to process_frame.
  • debug_data_size: An output parameter that receives the size of the debug data, after the next call to process_frame.

Returns:

  • Returns ErrorCode::SUCCESS if all went well. Returns ErrorCode::NOT_AVAILABLE on platforms where this functionality is not available. It can return ErrorCode::INTERNAL_ERROR if this method is called twice without calling process_frame.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);
const std::uint8_t* debug_data_buffer = nullptr;
std::size_t debug_data_size = 0;
auto error_code = pvi.debug_next_frame(&debug_data_buffer,
                                       &debug_data_size);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: debug_next_frame returned %s\n",
         plumerai::error_code_string(error_code));
}
// `debug_data_buffer` is still nullptr at this point
pvi.process_frame(...);
// `debug_data_buffer` and `debug_data_size` are now valid

// Write the data to a file for Plumerai support
const char* debug_file_name = "/tmp/plumerai_debug_frame.bin";
debug_file = fopen(debug_file_name, "wb");
fwrite(debug_data_buffer, 1, debug_data_size, debug_file);
fclose(debug_file);

// After this next `process_frame`, the memory of `debug_data_buffer` is
// automatically deallocated.
pvi.process_frame(...);

code_version

static const char* code_version();

Returns the version of the video intelligence code as a date and time.

For other version numbers, see also ObjectDetection::detector_version for the object detector, and FaceIdentification::embedding_version for the face embedder.

Returns:

  • The version of the code as YYYY.MM.DD.HH.MM date and time string.

object_detection

ObjectDetection object_detection();

Get the interface for the ObjectDetection video intelligence features.

motion_detection

MotionDetection motion_detection();

Get the interface for the MotionDetection video intelligence features.

detection_zones

DetectionZones detection_zones();

Get the interface for the DetectionZones video intelligence features.

face_identification

FaceIdentification face_identification();

Get the interface for the FaceIdentification video intelligence features.

face_enrollment_automatic

FaceEnrollmentAutomatic face_enrollment_automatic();

Get the interface for the FaceEnrollmentAutomatic video intelligence features.

face_enrollment_manual

FaceEnrollmentManual face_enrollment_manual();

Get the interface for the FaceEnrollmentManual video intelligence features.

Upgrade guide

From version 1.x to 2.0

  • The name 'People Detection' was changed to 'Video Intelligence' to reflect support for other detection classes and features such as advanced motion detection.

    • The library filename changed from libplumerai{peopledetection,faceidentification} to libplumeraivideointelligence. This should be updated in the relevant build scripts.
    • The main header file is now plumerai/video_intelligence.h instead of plumerai/people_detection.h, plumerai/face_identification.h or plumerai/face_identification_automatic.h.
    • The main class to use is now always plumerai::VideoIntelligence instead of plumerai::PeopleDetection, plumerai::FaceIdentification or plumerai::FaceIdentificationAutomatic.

  • Different features have been moved to separate 'feature classes', accessible from the main class VideoIntelligence. For example VideoIntelligence::motion_detection() provides access to all functionality related to motion detection, and VideoIntelligence::face_enrollment_automatic() provides access to automatic face enrollment functionality. Most functions that were in PeopleDetection, FaceIdentification or FaceIdentificationAutomatic have been moved to one of those other feature classes. For example:

    • PeopleDetection::add_detection_zone(...) is now VideoIntelligence::detection_zones().add_zone(...)
    • FaceIdentification::add_face_embedding(...) is now VideoIntelligence::face_enrollment_manual().add_embedding(...)
    • FaceIdentification::get_face_id(...) is now VideoIntelligence::face_identification().get_face_id(...)

  • The function process_frame no longer returns bounding boxes as output. Instead, the boxes are now accessible through VideoIntelligence::object_detection().get_detections(...) after the frame has been processed. Furthermore, image data pointers are now wrapped in a plumerai::ImagePointer struct, this is shown in the example below.

Code that looked like this before:

#include "plumerai/people_detection.h"

auto ppd = plumerai::PeopleDetection(height, width);

std::vector<BoxPrediction> predictions(0);
const auto error_code =
    ppd.process_frame<image_format>(image_data, predictions, delta_t);

Should be updated to this:

#include "plumerai/video_intelligence.h" // (1)

// Initialize the Video Intelligence object (2)
auto pvi = plumerai::VideoIntelligence(height, width);

// Process a video frame (3)
auto error_code = pvi.process_frame(
    plumerai::ImagePointer<image_format>(image_data), delta_t);

// Get bounding box results (4)
std::vector<BoxPrediction> predictions;
error_code = pvi.object_detection().get_detections(predictions);

  1. Replace plumerai/people_detection.h, plumerai/face_identification.h or plumerai/face_identification_automatic.h by plumerai/video_intelligence.h.

  2. Replace plumerai::PeopleDetection by plumerai::VideoIntelligence.

  3. Wrap the raw pointer argument in plumerai::ImagePointer. Remove the predictions output argument.

  4. Get bounding box results through the new ObjectDetection::get_detections function.

From version 1.14 to 1.15

The confidence_threshold argument has been removed from the single_image function. The return type of debug_next_frame changed from bool to ErrorCodeType.

In version 1.15 the API of start_face_enrollment and finish_face_enrollment changed compared to earlier versions:

  • The function get_cumulative_enrollment_score was removed.
  • The previous_embedding argument of start_face_enrollment was removed.
  • The resulting embedding of finish_face_enrollment is now returned via a reference argument.
  • The return type of both functions is now an error code, and finish_face_enrollment can indicate a low quality enrollment through this error code. There is no more need to check the enrollment score for the quality of the embedding.

If your code looked like this before:

const auto embedding = ffid.finish_face_enrollment();

Then it should be updated as follows:

std::vector<int8_t> embedding;
const auto error_code = ffid.finish_face_enrollment(embedding);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("Error code: %d\n", error_code);
}

From version 1.13 to 1.14

In version 1.14 the API of process_frame changed compared to earlier versions: the return type is now an error code, and the resulting boxes are now returned via a reference argument.

If your code looked like this before:

const auto predictions = ppd.process_frame<image_format>(image.data());

Then it should be updated as follows:

std::vector<BoxPrediction> predictions(0);
const auto error_code = ppd.process_frame<image_format>(image.data(), predictions);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("Error code: %d\n", error_code);
}