Plumerai Video Intelligence C API¶

This document describes the C API for the Plumerai Video Intelligence library for videos.

The C API wraps the C++ API and is as similar as possible. Since classes are not supported in C, a CPlumeraiVideoIntelligence object is introduced which needs to be passed to every function as first argument. This object can be created using plumerai_video_intelligence_create, after which each video frame should be processed by calling the process_frame function. The various intelligence features such as object detection and familiar face identification are documented on sub-pages found in the navigation menu.

Please refer to the minimal example for example code to get started.

The API is re-entrant, i.e. you can initialize several video intelligence objects in different threads and use them independently. However, using the same instance from different threads at the same time is not supported.

Video Intelligence¶

create¶

CPlumeraiVideoIntelligence
plumerai_video_intelligence_create(int height, int width);

Initializes a new Video Intelligence object.

This version of the constructor uses dynamic memory allocation. This is the default and recommended way to use the CPlumeraiVideoIntelligence object.

This needs to be called only once at the start of the application.

Arguments:

height: The height of the input image in pixels.
width: The width of the input image in pixels.

destroy¶

void plumerai_video_intelligence_destroy(
    CPlumeraiVideoIntelligence pvi);

Destroy the CPlumeraiVideoIntelligence object.

This needs to be called once at the very end to clean up.

Arguments:

pvi: An initialized CPlumeraiVideoIntelligence object.

create_static_mem¶

CPlumeraiVideoIntelligence
plumerai_video_intelligence_create_static_mem(int height, int width,
                                              uint8_t* buffer,
                                              size_t buffer_size);

Initializes a new Video Intelligence object.

This version of the constructor uses a user-provided memory buffer and is meant for advanced use-cases where dynamic memory allocation is not supported or not desired.

This needs to be called only once at the start of the application.

The CPlumeraiVideoIntelligence object will not take ownership of the memory buffer. The memory buffer must be at least as big as the size returned by plumerai_video_intelligence_get_required_memory_size and must stay valid for the lifetime of the CPlumeraiVideoIntelligence object.

If the size of the memory buffer is not big enough, the resulting object will be invalid, and all functions will return NOT_ENOUGH_MEMORY.

Arguments:

height: The height of the input image in pixels.
width: The width of the input image in pixels.
buffer: A pointer to the memory buffer, must stay valid for the lifetime of the CPlumeraiVideoIntelligence object.
buffer_size: The size of the memory buffer. The required size can be obtained through plumerai_video_intelligence_get_required_memory_size.

process_frame¶

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_packed_rgb888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_packed_rgba8888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_packed_bgra8888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_packed_yuyv(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_planar_rgb888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_planar_yuv420(
    CPlumeraiVideoIntelligence pvi, const uint8_t* plane0,
    const uint8_t* plane1, const uint8_t* plane2, float delta_t);

PlumeraiErrorCode
plumerai_video_intelligence_process_frame_planar_nv12(
    CPlumeraiVideoIntelligence pvi, const uint8_t* plane0,
    const uint8_t* plane1, float delta_t);

Process a single frame from a video sequence.

Make sure the image is right side up. When it is upside down it can still work but accuracy is significantly degraded.

Supported input formats:

Packed/interleaved data-types:
- PACKED_RGB888: 8-bit red, green, and blue. Red first.
  - Example in memory: R0 G0 B0 R1 G1 B1 ...
- PACKED_RGBA8888: 8-bit red, green, blue, and alpha. Red first. Also known as ABGR32.
  - Example in memory: R0 G0 B0 A0 R1 G1 B1 A1 ...
- PACKED_BGRA8888: 8-bit blue, green, red, and alpha. Blue first. Also known as ARGB32.
  - Example in memory: B0 G0 R0 A0 B1 G1 R1 A1 ...
- PACKED_YUYV: ==YUY2, Luma (Y) and chroma (U, V) with 4:2:0 subsampling.
  - Example in memory: Y0 U01 Y1 V01 Y2 U23 Y3 V23 ...
Planar formats:
- PLANAR_RGB888: 8-bit red, green, and blue. Red first. We assume the R, G, and B data is consecutive in memory without padding in between the three channels.
  - Example in memory: R0 R1 R2 ... G0 G1 G2 ... B0 B1 B2 ...
- PLANAR_YUV420: 8-bit luma (Y) and chroma (U, V) with 4:2:0 subsampling. The Y, U, and V planes can be in 3 different memory locations.
  - Example in memory: Y0 Y1 Y2 Y3 ..., U01 U23 ..., V01 V23 ...
- PLANAR_NV12: 8-bit luma (Y) and interleaved chroma (UV) with 4:2:0 subsampling. The Y and UV planes can be in 2 different memory locations.
  - Example in memory: Y0 Y1 Y2 Y3 ..., U01 V01 U23 V23 ...

Note that not all formats are available on every platform.

Arguments:

pvi: An initialized CPlumeraiVideoIntelligence object
image_data: A pointer to the image data (planeX for planar image formats).
delta_t: The time in seconds it took between this and the previous video frame (1/fps). If set to 0, then the system clock will be used to compute this value.

Returns:

An error code of type PlumeraiErrorCode. See that enum for more details.

single_image¶

PlumeraiErrorCode
plumerai_video_intelligence_single_image_packed_rgb888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_packed_rgba8888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_packed_bgra8888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_packed_yuyv(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_planar_rgb888(
    CPlumeraiVideoIntelligence pvi, const uint8_t* image_data, int height,
    int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_planar_yuv420(
    CPlumeraiVideoIntelligence pvi, const uint8_t* plane0,
    const uint8_t* plane1, const uint8_t* plane2, int height, int width);

PlumeraiErrorCode
plumerai_video_intelligence_single_image_planar_nv12(
    CPlumeraiVideoIntelligence pvi, const uint8_t* plane0,
    const uint8_t* plane1, int height, int width);

Process a single image not part of a video sequence.

This should not be used for video data. It can be used for face enrollments from a set of images. The object detection box id values obtained after calling plumerai_video_intelligence_single_image are not related to those generated through plumerai_video_intelligence_process_frame or through other calls to plumerai_video_intelligence_single_image.

See the documentation under plumerai_video_intelligence_process_frame for details about the formats.

Arguments:

pvi: An initialized CPlumeraiVideoIntelligence object
image_data: A pointer to the image data (planeX for planar image formats).
height: The height of the input image in pixels. If height = 0 the height set in the constructor will be used.
width: The width of the input image in pixels. If width = 0 the width set in the constructor will be used.

Returns:

An error code of type PlumeraiErrorCode. See that enum for more details.

set_night_mode¶

PlumeraiErrorCode plumerai_video_intelligence_set_night_mode(
    CPlumeraiVideoIntelligence pvi, bool night_mode);

Configure the video intelligence algorithm for either day mode color videos (default) or night mode IR videos.

This configures the video intelligence algorithm for optimal performance on day mode color data (default) or on night mode IR data.

After switching from day to night mode or back, the motion detection algorithm will need a couple of video frames to stabilize, so the motion-grid will not be immediately available.

This function does not have to be called before every frame, only when switching from RGB to IR video data or back.

Arguments:

pvi: An initialized CPlumeraiVideoIntelligence object.
night_mode: Set to true for night mode or false for day mode.

Returns:

Returns PlumeraiErrorCode::SUCCESS on success.

debug_next_frame¶

PlumeraiErrorCode plumerai_video_intelligence_debug_next_frame(
    CPlumeraiVideoIntelligence pvi, const uint8_t** debug_data_buffer,
    size_t* debug_data_size);

Enable debug mode for the next frame.

The next time plumerai_video_intelligence_process_frame is called, this will dump the input image as well as internal data and final results to binary blob, meant to be stored to a file. This file can then be shared with Plumerai support for further analysis. These files contain uncompressed image data and can become several megabytes large.

The user will receive a pointer to the data, but this will only be available after the next call to plumerai_video_intelligence_process_frame, see the example code. The data is only valid for one frame, and will be invalidated after the second call to plumerai_video_intelligence_process_frame, or another call to plumerai_video_intelligence_debug_next_frame.

This function is only available on platforms that support dynamic memory allocation.

Arguments:

pvi: An initialized CPlumeraiVideoIntelligence object.
debug_data_buffer: An output parameter that receives a pointer to the debug data, after the next call to plumerai_video_intelligence_process_frame.
debug_data_size: An output parameter that receives the size of the debug data, after the next call to plumerai_video_intelligence_process_frame.

Returns:

Returns PlumeraiErrorCode::SUCCESS if all went well. Returns PlumeraiErrorCode::NOT_AVAILABLE on platforms where this functionality is not available. It can return PlumeraiErrorCode::INTERNAL_ERROR if this method is called twice without calling plumerai_video_intelligence_process_frame.

get_required_memory_size¶

size_t
plumerai_video_intelligence_get_required_memory_size(int height, int width);

Get the required memory size for the CPlumeraiVideoIntelligence object.

This optional function can be used to avoid any dynamic memory allocation within the CPlumeraiVideoIntelligence object. See the advanced plumerai_video_intelligence_create_static_mem function for details.

Arguments:

height: The height of the input image in pixels.
width: The width of the input image in pixels.

code_version¶

const char* plumerai_video_intelligence_code_version(void);

Returns the version of the video intelligence code as a date and time.

For other version numbers, see also plumerai_object_detection_detector_version for the object detector, and plumerai_face_identification_embedding_version for the face embedder.

Returns:

The version of the code as YYYY.MM.DD.HH.MM date and time string.

Upgrade guide¶

From version 2.0 to 2.1¶

Apart from adding new functionality, two minor changes were made to the existing API in 2.1:

All API calls providing constants (such as the version number or the embedding size) now no longer require a CPlumeraiVideoIntelligence pvi argument.
Because there is now also a static memory version of the API, the plumerai_face_enrollment_automatic_configure_face_snapshots now take additional optional arguments. Setting the buffer argument to NULL and the buffer_size to 0 will result in the same behaviour as in 2.0.

From version 1.x to 2.0¶

The name 'People Detection' was changed to 'Video Intelligence' to reflect support for other detection classes and features such as advanced motion detection.
- The library filename changed from libplumerai{peopledetection,faceidentification} to libplumeraivideointelligence. This should be updated in the relevant build scripts.
- The main header file is now plumerai/video_intelligence_c.h instead of plumerai/people_detection_c.h, plumerai/face_identification_c.h or plumerai/face_identification_automatic_c.h.
Most function have been renamed. They now follow the scheme plumerai_<feature>_<function-name> where <feature> is one of detection_zones, face_enrollment_automatic, face_enrollment_manual, face_identification, motion_detection, object_detection or video_intelligence.
The function plumerai_video_intelligence_process_frame no longer returns bounding boxes as output. Instead, the boxes are now accessible through plumerai_object_detection_get_detections after the frame has been processed.

Code that looked like this before:

#include "plumerai/people_detection_c.h"

CPeopleDetection ppd = PeopleDetectionInit(height, width);

BoxPrediction predictions[10];
int num_results = 0;
PlumeraiErrorCode error_code = PeopleDetectionProcessFrame(
    ppd, image, predictions, 10, &num_results, delta_t);

Should be updated to this:

#include "plumerai/video_intelligence_c.h" // (1)

// Initialize the video intelligence algorithm (2)
CPlumeraiVideoIntelligence pvi =
    plumerai_video_intelligence_create(height, width);

// Process a video frame (3)
PlumeraiErrorCode error_code =
    plumerai_video_intelligence_process_frame_packed_rgb888(pvi, image, delta_t);

// Get bounding box results (4)
const BoxPrediction *predictions = NULL;
size_t num_results = 0;
error_code = plumerai_object_detection_get_detections(pvi, &predictions,
                                                      &num_results);

Replace plumerai/people_detection_c.h, plumerai/face_identification_c.h or plumerai/face_identification_automatic_c.h by plumerai/video_intelligence_c.h.
Replace CPeopleDetection by CPlumeraiVideoIntelligence and PeopleDetectionInit by plumerai_video_intelligence_create.
Replace PeopleDetectionProcessFrame by plumerai_video_intelligence_process_frame_<format>. Remove the predictions and num_results output arguments.
Get bounding box results through the new plumerai_object_detection_get_detections function.

From version 1.14 to 1.15¶

The confidence_threshold argument has been removed from the single_image function. The type PlumeraiErrorCode was introduced. Most functions now return this error code. The process-frame and single-image functions now return the number of results through an output argument instead of the return value.

If your code looked like this before:

int num_results =
    PeopleDetectionProcessFrame(ppd, image, predictions, 10, delta_t);

Then it should be updated as follows:

int num_results = 0;
PlumeraiErrorCode error_code = PeopleDetectionProcessFrame(
    ppd, image, predictions, 10, &num_results, delta_t);
if (error_code != PLUMERAI_SUCCESS) {
  printf("Error: %s\n", PeopleDetectionErrorCodeString(error_code));
}