Skip to content

Plumerai Video Intelligence C API

This document describes the C API for the Plumerai Video Intelligence library for videos.

The C API wraps the C++ API and is as similar as possible. Since classes are not supported in C, a CPlumeraiVideoIntelligence object is introduced which needs to be passed to every function as first argument. This object can be created using plumerai_video_intelligence_create, after which each video frame should be processed by calling the process_frame function. The various intelligence features such as object detection and familiar face identification are documented on sub-pages found in the navigation menu.

Please refer to the minimal example for example code to get started.

The API is re-entrant, i.e. you can initialize several video intelligence objects in different threads and use them independently. However, using the same instance from different threads at the same time is not supported.

Video Intelligence

create

CPlumeraiVideoIntelligence
plumerai_video_intelligence_create(int height, int width);

Initializes a new Video Intelligence object.

This version of the constructor uses dynamic memory allocation. This is the default and recommended way to use the CPlumeraiVideoIntelligence object.

This needs to be called only once at the start of the application.

Arguments:

  • height: The height of the input image in pixels.
  • width: The width of the input image in pixels.

destroy

void plumerai_video_intelligence_destroy(
    CPlumeraiVideoIntelligence pvi);

Destroy the CPlumeraiVideoIntelligence object.

This needs to be called once at the very end to clean up.

Arguments:

  • pvi: An initialized CPlumeraiVideoIntelligence object.

create_static_mem

CPlumeraiVideoIntelligence
plumerai_video_intelligence_create_static_mem(int height, int width,
                                              uint8_t* buffer,
                                              size_t buffer_size);

Initializes a new Video Intelligence object.

This version of the constructor uses a user-provided memory buffer and is meant for advanced use-cases where dynamic memory allocation is not supported or not desired.

This needs to be called only once at the start of the application.

The CPlumeraiVideoIntelligence object will not take ownership of the memory buffer. The memory buffer must be at least as big as the size returned by plumerai_video_intelligence_get_required_memory_size and must stay valid for the lifetime of the CPlumeraiVideoIntelligence object.

If the size of the memory buffer is not big enough, the resulting object will be invalid, and all functions will return NOT_ENOUGH_MEMORY.

Arguments:

  • height: The height of the input image in pixels.
  • width: The width of the input image in pixels.
  • buffer: A pointer to the memory buffer, must stay valid for the lifetime of the CPlumeraiVideoIntelligence object.
  • buffer_size: The size of the memory buffer. The required size can be obtained through plumerai_video_intelligence_get_required_memory_size.

process_frame

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_packed_rgb888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_packed_rgba8888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_packed_bgra8888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_packed_rgb565(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_packed_yuyv(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_planar_rgb888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_planar_yuv420(
    CPlumeraiVideoIntelligence pvi, const uint8_t* plane0,
    const uint8_t* plane1, const uint8_t* plane2, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_planar_nv12(
    CPlumeraiVideoIntelligence pvi, const uint8_t* plane0,
    const uint8_t* plane1, float delta_t);

Process a single frame from a video sequence.

Make sure the image is right side up. When it is upside down it can still work but accuracy is significantly degraded.

Supported input formats:

  • Packed/interleaved data-types:
    • PACKED_RGB888: 8-bit red, green, and blue. Red first.
      • Example in memory: R0 G0 B0 R1 G1 B1 ...
    • PACKED_RGBA8888: 8-bit red, green, blue, and alpha. Red first. Also known as ABGR32.
      • Example in memory: R0 G0 B0 A0 R1 G1 B1 A1 ...
    • PACKED_BGRA8888: 8-bit blue, green, red, and alpha. Blue first. Also known as ARGB32.
      • Example in memory: B0 G0 R0 A0 B1 G1 R1 A1 ...
    • PACKED_RGB565: 5-bit red, 6-bit green, and 5-bit blue. Blue and first 3-bits green in the first byte, remaining 3-bits green and red in the second byte.
      • Example in memory: BG0 GR0 BG1 GR1 ...
    • PACKED_YUYV: ==YUY2, Luma (Y) and chroma (U, V) with 4:2:0 subsampling.
      • Example in memory: Y0 U01 Y1 V01 Y2 U23 Y3 V23 ...
  • Planar formats:
    • PLANAR_RGB888: 8-bit red, green, and blue. Red first. We assume the R, G, and B data is consecutive in memory without padding in between the three channels.
      • Example in memory: R0 R1 R2 ... G0 G1 G2 ... B0 B1 B2 ...
    • PLANAR_YUV420: 8-bit luma (Y) and chroma (U, V) with 4:2:0 subsampling. The Y, U, and V planes can be in 3 different memory locations.
      • Example in memory: Y0 Y1 Y2 Y3 ..., U01 U23 ..., V01 V23 ...
    • PLANAR_NV12: 8-bit luma (Y) and interleaved chroma (UV) with 4:2:0 subsampling. The Y and UV planes can be in 2 different memory locations.
      • Example in memory: Y0 Y1 Y2 Y3 ..., U01 V01 U23 V23 ...

Note that not all formats are available on every platform.

Arguments:

  • pvi: An initialized CPlumeraiVideoIntelligence object
  • image_data: A pointer to the image data (planeX for planar image formats).
  • delta_t: The time in seconds it took between this and the previous video frame (1/fps). If set to 0, then the system clock will be used to compute this value.

Returns:

  • An error code of type PlumeraiErrorCode. See that enum for more details.

single_image

PlumeraiErrorCode
plumerai_video_intelligence_single_image_packed_rgb888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_packed_rgba8888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_packed_bgra8888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_packed_rgb565(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_packed_yuyv(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_planar_rgb888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_planar_yuv420(
    CPlumeraiVideoIntelligence pvi, const uint8_t* plane0,
    const uint8_t* plane1, const uint8_t* plane2, int height, int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_planar_nv12(
    CPlumeraiVideoIntelligence pvi, const uint8_t* plane0,
    const uint8_t* plane1, int height, int width);

Process a single image not part of a video sequence.

This should not be used for video data. It can be used for face enrollments from a set of images. The object detection box id values obtained after calling plumerai_video_intelligence_single_image are not related to those generated through plumerai_video_intelligence_process_frame or through other calls to plumerai_video_intelligence_single_image.

See the documentation under plumerai_video_intelligence_process_frame for details about the formats.

Arguments:

  • pvi: An initialized CPlumeraiVideoIntelligence object
  • image_data: A pointer to the image data (planeX for planar image formats).
  • height: The height of the input image in pixels. If height = 0 the height set in the constructor will be used.
  • width: The width of the input image in pixels. If width = 0 the width set in the constructor will be used.

Returns:

  • An error code of type PlumeraiErrorCode. See that enum for more details.

set_night_mode

PlumeraiErrorCode plumerai_video_intelligence_set_night_mode(
    CPlumeraiVideoIntelligence pvi, bool night_mode);

Configure the video intelligence algorithm for either day mode color videos (default) or night mode IR videos.

This configures the video intelligence algorithm for optimal performance on day mode color data (default) or on night mode IR data.

After switching from day to night mode or back, the motion detection algorithm will need a couple of video frames to stabilize, so the motion-grid will not be immediately available.

This function does not have to be called before every frame, only when switching from RGB to IR video data or back.

Arguments:

  • pvi: An initialized CPlumeraiVideoIntelligence object.
  • night_mode: Set to true for night mode or false for day mode.

Returns:

  • Returns PlumeraiErrorCode::SUCCESS on success.

camera_is_unstable

PlumeraiErrorCode
plumerai_video_intelligence_camera_is_unstable(CPlumeraiVideoIntelligence pvi);

Signal that the camera/ISP is unstable.

In some situations, the camera or ISP may be adjusting its settings, resulting in unstable video frames. This can happen, for example, during auto-exposure or switching to and from IR mode. After calling this function, the algorithm will not run motion detection when frames are processed. When plumerai_video_intelligence_camera_is_no_longer_unstable is called, the algorithm will reset its internal motion state and continue processing frames as usual.

Arguments:

  • pvi: An initialized CPlumeraiVideoIntelligence object.

Returns:

  • Returns PlumeraiErrorCode::SUCCESS on success.

camera_is_no_longer_unstable

PlumeraiErrorCode
plumerai_video_intelligence_camera_is_no_longer_unstable(
    CPlumeraiVideoIntelligence pvi);

Signal that the camera/ISP is stable again.

Needs to be called some time after plumerai_video_intelligence_camera_is_unstable to re-enable the Plumerai motion detection. See plumerai_video_intelligence_camera_is_unstable for more information.

Arguments:

  • pvi: An initialized CPlumeraiVideoIntelligence object.

Returns:

  • Returns PlumeraiErrorCode::SUCCESS on success.

debug_mode_start

PlumeraiErrorCode plumerai_video_intelligence_debug_mode_start(
    CPlumeraiVideoIntelligence pvi, bool exclude_images);

Enable debug mode.

When plumerai_video_intelligence_process_frame is called while debug mode is active, this will store debug information. This data is meant to be stored to a file to be shared with Plumerai support for further analysis. These files contain uncompressed input image data and can become large (several megabytes per frame). If exclude_images is set to true, no image data will be included.

The resulting debug data can be retrieved using plumerai_video_intelligence_debug_mode_end.

Calling plumerai_video_intelligence_debug_mode_start invalidates the data pointer obtained from any previous calls to plumerai_video_intelligence_debug_mode_end.

Arguments:

  • pvi: An initialized CPlumeraiVideoIntelligence object.
  • exclude_images: Set to true to exclude input images in the debug data.

Returns:

  • Returns PlumeraiErrorCode::SUCCESS if all went well. Returns PlumeraiErrorCode::NOT_AVAILABLE on platforms where this functionality is not available. It can return PlumeraiErrorCode::INTERNAL_ERROR if this method is called twice without calling plumerai_video_intelligence_debug_mode_end.

debug_mode_end

PlumeraiErrorCode plumerai_video_intelligence_debug_mode_end(
    CPlumeraiVideoIntelligence pvi, const uint8_t** debug_data_buffer,
    size_t* debug_data_size);

Stop debug mode and retrieve the debug data.

The user will receive a pointer to the gathered data. The data will be invalidated after another call to plumerai_video_intelligence_debug_mode_start.

This function is only available on platforms that support dynamic memory allocation.

Arguments:

  • pvi: An initialized CPlumeraiVideoIntelligence object.
  • debug_data_buffer: An output parameter that receives a pointer to the debug data.
  • debug_data_size: An output parameter that receives the size of the debug data.

Returns:

  • Returns PlumeraiErrorCode::SUCCESS if all went well. Returns PlumeraiErrorCode::NOT_AVAILABLE on platforms where this functionality is not available. It can return PlumeraiErrorCode::INTERNAL_ERROR if this method is called twice without calling plumerai_video_intelligence_debug_mode_start, or PlumeraiErrorCode::INVALID_ARGUMENT when debug_data_buffer is null.

get_required_memory_size

size_t
plumerai_video_intelligence_get_required_memory_size(int height, int width);

Get the required memory size for the CPlumeraiVideoIntelligence object.

This optional function can be used to avoid any dynamic memory allocation within the CPlumeraiVideoIntelligence object. See the advanced plumerai_video_intelligence_create_static_mem function for details.

Arguments:

  • height: The height of the input image in pixels.
  • width: The width of the input image in pixels.

code_version

const char* plumerai_video_intelligence_code_version(void);

Returns the version of the video intelligence code as a date and time.

For other version numbers, see also plumerai_object_detection_detector_version for the object detector, and plumerai_face_identification_embedding_version for the face embedder.

Returns:

  • The version of the code as YYYY.MM.DD.HH.MM date and time string.

Upgrade guide

From version 2.2 to 2.3

The meaning of the confidence field in the BoxPrediction struct has been changed: the confidence values are now tuned so that for each code/model version and for each object class, sensible values range from 0.5 to 1. If you have been using the confidence field before, e.g. for thresholding, you must re-tune your thresholds accordingly and consider the new Plumerai API functions for this instead.

By default, the Plumerai software now sets a threshold (e.g. 0.6) to filter out values below a certain confidence level. The default value can be queried with a new API function plumerai_object_detection_get_confidence_threshold. To change the threshold, use the newly added API function plumerai_object_detection_set_confidence_threshold.

As a side effect of some internal changes, the track IDs produced by the algorithm are no longer guaranteed to be consecutive values, instead they will sometimes skip values.

From version 2.1 to 2.2

The track ID in the BoxPrediction struct has been updated, so code that uses track IDs needs to be updated accordingly:

  • It has been renamed from id to track_id.
  • It is now a signed 64-bit value (int64_t) instead of an unsigned 32-bit.
  • The track ID now starts counting from a unique random number instead of 0. This should likely not affect any code, but if it does, it is possible to re-create the old behaviour by saving the first track ID as a base value and subtracting it from subsequent track IDs.

The Automatic Face Enrollment functions have been updated:

  • The following functions have been renamed, code can be updated with a find-replace.

    • plumerai_face_enrollment_automatic_remove_embedding is now plumerai_face_enrollment_automatic_remove_face_id
    • plumerai_face_enrollment_automatic_remove_all_embeddings is now plumerai_face_enrollment_automatic_remove_all_face_ids
    • plumerai_face_enrollment_automatic_merge_embeddings is now plumerai_face_enrollment_automatic_merge_face_ids

  • The plumerai_face_enrollment_automatic_get_face_snapshot function now requires an additional track_id argument to retrieve snapshots for different tracks within an enrollment. Refer to the function's documentation for further details.

From version 2.0 to 2.1

Apart from adding new functionality, two minor changes were made to the existing API in 2.1:

  • All API calls providing constants (such as the version number or the embedding size) now no longer require a CPlumeraiVideoIntelligence pvi argument.
  • Because there is now also a static memory version of the API, the plumerai_face_enrollment_automatic_configure_face_snapshots now take additional optional arguments. Setting the buffer argument to NULL and the buffer_size to 0 will result in the same behaviour as in 2.0.

From version 1.x to 2.0

  • The name 'People Detection' was changed to 'Video Intelligence' to reflect support for other detection classes and features such as advanced motion detection.

    • The library filename changed from libplumerai{peopledetection,faceidentification} to libplumeraivideointelligence. This should be updated in the relevant build scripts.
    • The main header file is now plumerai/video_intelligence_c.h instead of plumerai/people_detection_c.h, plumerai/face_identification_c.h or plumerai/face_identification_automatic_c.h.

  • Most function have been renamed. They now follow the scheme plumerai_<feature>_<function-name> where <feature> is one of detection_zones, face_enrollment_automatic, face_enrollment_manual, face_identification, motion_detection, object_detection or video_intelligence.

  • The function plumerai_video_intelligence_process_frame no longer returns bounding boxes as output. Instead, the boxes are now accessible through plumerai_object_detection_get_detections after the frame has been processed.

Code that looked like this before:

#include "plumerai/people_detection_c.h"

CPeopleDetection ppd = PeopleDetectionInit(height, width);

BoxPrediction predictions[10];
int num_results = 0;
PlumeraiErrorCode error_code = PeopleDetectionProcessFrame(
    ppd, image, predictions, 10, &num_results, delta_t);

Should be updated to this:

#include "plumerai/video_intelligence_c.h" // (1)

// Initialize the video intelligence algorithm (2)
CPlumeraiVideoIntelligence pvi =
    plumerai_video_intelligence_create(height, width);

// Process a video frame (3)
PlumeraiErrorCode error_code =
    plumerai_video_intelligence_process_frame_packed_rgb888(pvi, image, delta_t);

// Get bounding box results (4)
const BoxPrediction *predictions = NULL;
size_t num_results = 0;
error_code = plumerai_object_detection_get_detections(pvi, &predictions,
                                                      &num_results);

  1. Replace plumerai/people_detection_c.h, plumerai/face_identification_c.h or plumerai/face_identification_automatic_c.h by plumerai/video_intelligence_c.h.

  2. Replace CPeopleDetection by CPlumeraiVideoIntelligence and PeopleDetectionInit by plumerai_video_intelligence_create.

  3. Replace PeopleDetectionProcessFrame by plumerai_video_intelligence_process_frame_<format>. Remove the predictions and num_results output arguments.

  4. Get bounding box results through the new plumerai_object_detection_get_detections function.

From version 1.14 to 1.15

The confidence_threshold argument has been removed from the single_image function. The type PlumeraiErrorCode was introduced. Most functions now return this error code. The process-frame and single-image functions now return the number of results through an output argument instead of the return value.

If your code looked like this before:

int num_results =
    PeopleDetectionProcessFrame(ppd, image, predictions, 10, delta_t);

Then it should be updated as follows:

int num_results = 0;
PlumeraiErrorCode error_code = PeopleDetectionProcessFrame(
    ppd, image, predictions, 10, &num_results, delta_t);
if (error_code != PLUMERAI_SUCCESS) {
  printf("Error: %s\n", PeopleDetectionErrorCodeString(error_code));
}