Skip to content

Plumerai Video Intelligence Python API

This document describes the Python API for the Plumerai Video Intelligence library for videos.

The Python files are self-documented. The main entrypoint is the VideoIntelligence class which provides access to all functionality of the Plumerai software. It has a process_frame function that needs to be executed on each input frame. The various intelligence features such as object detection and familiar face identification are available through different interfaces, listed here.

Please refer to the minimal examples for example code to get started.

VideoIntelligence

VideoIntelligence

VideoIntelligence(height: int, width: int)

Initializes a new Video Intelligence object.

This needs to be called only once at the start of the application.

Arguments:

  • height: The height of the input image in pixels.
  • width: The width of the input image in pixels.

Returns:

  • Nothing.

process_frame

VideoIntelligence.process_frame(image, delta_t: float = 0.0) -> ErrorCode

Process a single frame from a video sequence with RGB input.

The image must have the height and width that was specified when the VideoIntelligence object was created.

Make sure the image is right side up. When it is upside down it can still work but accuracy is significantly degraded.

Arguments:

  • image: A tensor of shape (video_height, video_width, 3) with RGB image data. It can be a Numpy array or TensorFlow, PyTorch or Jax tensor.
  • delta_t: The time in seconds it took between this and the previous video frame (1/fps). If left to the default of 0, then the system clock will be used to compute this value.

Returns:

single_image

VideoIntelligence.single_image(image) -> ErrorCode

Process a single image not part of a video sequence.

This should not be used for video data. It can be used for face enrollments from a set of images. The object detection box id values obtained after calling single_image are not related to those generated through process_frame or through other calls to single_image.

Arguments:

  • image: A tensor of shape (*, *, 3) with RGB image data. It can be a Numpy array or TensorFlow, PyTorch or Jax tensor.

Returns:

set_night_mode

VideoIntelligence.set_night_mode(night_mode: bool) -> ErrorCode

Configure the video intelligence algorithm for either day mode color videos (default) or night mode IR videos.

This configures the video intelligence algorithm for optimal performance on day mode color data (default) or on night mode IR data.

After switching from day to night mode or back, the motion detection algorithm will need a couple of video frames to stabilize, so the motion-grid will not be immediately available.

This function does not have to be called before every frame, only when switching from RGB to IR video data or back.

Arguments:

  • night_mode: Set to true for night mode or false for day mode.

Returns:

store_state

VideoIntelligence.store_state() -> tuple[ErrorCode, bytes]

Store the current state of the algorithm to a byte array.

This function can be used when processing a video in chunks, doing different chunks at different times or on different machines. The state can be restored by calling restore_state with the data returned by store_state. When the library is built with support for familiar face identification, the state includes the face library. Constraints:

  • The delta_t parameter of process_frame can not be left to zero after restoring a previous state.
  • If familiar face identification is enabled, the state can only be stored and restored when not enrolling.

Arguments:

  • None.

Returns:

  • A tuple with an error code of type ErrorCode and a bytes object to store the serialized state in

restore_state

VideoIntelligence.restore_state(state: bytes) -> ErrorCode

Restore the state of the algorithm from a byte array.

See store_state for more information. The user must ensure that the height and width of the current object match the height and width of the state that is being restored.

Arguments:

  • state: A vector containing the serialized state.

Returns:

  • Returns SUCCESS on success. Returns STATE_CORRUPT or STATE_SETTINGS_MISMATCH on error.

debug_mode_start

VideoIntelligence.debug_mode_start(exclude_images: bool = False) -> ErrorCode

Enable debug mode.

When process_frame is called while debug mode is active, this will store debug information. This data is meant to be stored to a file to be shared with Plumerai support for further analysis. These files contain uncompressed input image data and can become large (several megabytes per frame). If exclude_images is set to true, no image data will be included.

The resulting debug data can be retrieved using debug_mode_end.

Arguments:

  • exclude_images: Set to True to exclude input images in the debug data.

Returns:

  • Returns SUCCESS if all went well. It can return ErrorCode::INTERNAL_ERROR if this method is called twice without calling debug_mode_end.

debug_mode_end

VideoIntelligence.debug_mode_end() -> tuple[ErrorCode, bytes]

Stop debug mode and retrieve the debug data.

The caller will receive the gathered debug data.

Arguments:

  • None.

Returns:

  • Returns SUCCESS if all went well. It can return ErrorCode::INTERNAL_ERROR if this method is called twice without calling debug_mode_start.

code_version

VideoIntelligence.code_version(cls) -> str

Returns the version of the video intelligence code as a date and time.

For other version numbers, see also ObjectDetection::detector_version for the object detector, and FaceIdentification::embedding_version for the face embedder.

Arguments:

  • None.

Returns:

  • The version of the code as YYYY.MM.DD.HH.MM date and time string.

object_detection

@property
VideoIntelligence.object_detection -> ObjectDetection

Get the interface for the ObjectDetection video intelligence features.

motion_detection

@property
VideoIntelligence.motion_detection -> MotionDetection

Get the interface for the MotionDetection video intelligence features.

detection_zones

@property
VideoIntelligence.detection_zones -> DetectionZones

Get the interface for the DetectionZones video intelligence features.

face_identification

@property
VideoIntelligence.face_identification -> FaceIdentification

Get the interface for the FaceIdentification video intelligence features.

face_enrollment_automatic

@property
VideoIntelligence.face_enrollment_automatic -> FaceEnrollmentAutomatic

Get the interface for the FaceEnrollmentAutomatic video intelligence features.

face_enrollment_manual

@property
VideoIntelligence.face_enrollment_manual -> FaceEnrollmentManual

Get the interface for the FaceEnrollmentManual video intelligence features.