Skip to content

Plumerai VLM Video API

Important: There are two possible scenarios:

  1. The VLM Video API is not available in your plumerai_vision_language package, i.e. the compute_video_scene_data function is not available. In this case we refer to the VLM Video Collection and VLM Video Embedder API documentation, visit this page for C++ and this page for Python.
  2. Otherwise, you can use the documentation below.

There are three stages involved in processing a video, which are covered by three functions:

For convenience there is an encode_video utility function that combines all of these in one go.

Data types

VideoEncoding

VideoEncoding: TypeAlias = bytes

Raw bytes obtained from running compute_video_encoding or encode_video.

VideoMetadata

class VideoMetadata:
    unix_timestamp: int
    ffid_ids: list[str]
    device: str
    camera_id: str = ""

Metadata input for the compute_video_encoding, compute_captioning_prompt, and encode_video functions.

When Face Identification results are available for a video, ffid_ids can be set to a list of names, i.e. ffid_ids = ["Alice", "Robert Smith", "Charlie"]. This allows searching for specific people in videos.

The device field can be left empty.

The camera_id field identifies the camera that recorded the video. Together with the camera_location_map given to search, this allows searching for videos from a specific location, see the location search example. It can be left empty when the camera is unknown.

Frame

class Frame:
    image: np.ndarray
    time: float

Frames to be combined with search results for showing thumbnails.

The time is the time in seconds from the first frame of the video at which the thumbnail was taken. search will return one of these timestamps as the most relevant to the search query.

Note that the first frame does not always have timestamp 0 in the video file. To seek a video player to the right frame, add the start time of the video stream to time (with PyAV: stream.start_time * stream.time_base), or for constant framerate videos use the frame index round(time * fps).

These frames are also required for captioning.

VideoSceneData

class VideoSceneData:
    selected_frames: list[Frame]
    data: bytes
    scene_layout: SceneLayout | None
    viewpoint_status: ViewpointStatus
    active_scene_id: int | None
    arrival_departure_events: list[ArrivalDepartureEvent] | None

Output of compute_video_scene_data. The selected_frames can be used for showing thumbnails in search results.

The other fields are only set when a scene layout was passed to compute_video_scene_data; see the scene layout overview page:

  • scene_layout: The SceneLayout that was passed in, used internally for captioning.
  • viewpoint_status: Which of the scene layout's scenes the video shows, a ViewpointStatus from the plumerai_video_intelligence package:
    • MATCH_ACTIVE: the video shows the active scene of the scene layout.
    • MATCH_INACTIVE: the video shows another scene of the scene layout. SceneLayout.set_active_scene can make it the active one.
    • MISMATCH: the video shows a scene that the scene layout does not hold. The video should be passed to compute_scene_layout.
    • UNKNOWN: no scene layout was passed, or the comparison could not be made, for example because the video is too dark.
  • active_scene_id: The ID of the scene the video shows, to be passed to SceneLayout.set_active_scene. It is -1 when no scene matched, and None without a scene layout.
  • arrival_departure_events: The persons and vehicles that arrived in or departed from the indoor space during the video, see ArrivalDepartureEvent. None without a scene layout.

ArrivalDepartureEvent

class ArrivalDepartureStatus(Enum):
    ARRIVED_INDOORS = 1
    DEPARTED_FROM_INDOORS = 2

class ArrivalDepartureConfidence(Enum):
    LOW = 0
    MEDIUM = 1
    HIGH = 2

class ArrivalDepartureEvent:
    track_ids: list[int]
    class_id: DetectionClass
    status: ArrivalDepartureStatus
    confidence: ArrivalDepartureConfidence

A person or vehicle that arrived in or departed from the indoor space during a video, as part of VideoSceneData.

Examples include a car exiting the garage onto the driveway (departure), or a person entering a house through the front door (arrival). This can include doors that are predicted to be next to the camera but not visible to it. A door that is predicted to connect one indoor space to another indoor space does not trigger arrivals or departures.

Whether an object arrived in or departed from the indoor space is the net result over the whole video: a person that walks out and comes back in again does not trigger an event. When person re-identification is disabled, track_ids holds a single track ID. When person re-identification is enabled, the tracks that were recognized as the same person produce a single event, and track_ids lists all of them. class_id is a DetectionClass from the plumerai_video_intelligence package.

When viewpoint_status is MATCH_INACTIVE or MISMATCH, no events are produced: the boundaries of the active scene are not where they are in this video.

EncodeVideoResult

class EncodeVideoResult:
    video_encoding: VideoEncoding
    selected_frames: list[Frame]
    captioning_prompt: CaptioningPrompt | None
    viewpoint_status: ViewpointStatus
    active_scene_id: int | None
    arrival_departure_events: list[ArrivalDepartureEvent] | None

Output of encode_video.

  • video_encoding: bytes of Plumerai's VideoEncoding object.
  • selected_frames: A list of Frame objects, which can be used for showing thumbnails in search results.
  • captioning_prompt: bytes of CaptioningPrompt if compute_prompt was True, otherwise None.
  • viewpoint_status, active_scene_id, arrival_departure_events: The scene layout results, which are only set when a scene_layout was passed to encode_video. See VideoSceneData for what they mean.

Functions

compute_video_scene_data

def compute_video_scene_data(
    video_data: bytes | Iterable[tuple[np.ndarray, float]],
    *,
    include_thumbnails: bool = False,
    include_captioning_data: bool = False,
    include_alerts_data: bool = False,
    night_mode: bool = False,
    scene_layout: SceneLayout | bytes | None = None,
) -> VideoSceneData:

Extract scene data from a video, required for VLM Video Search and captioning.

Processes the video through the VideoIntelligence pipeline to extract scene information, object tracks, and other metadata required for computing the video embeddings. Optionally, it can also include data required for captioning.

The resulting data can be passed to compute_video_encoding to compute the video embeddings. If include_thumbnails is True, the result will also include thumbnails. If include_captioning_data is True, the result can be passed to compute_captioning_prompt to compute the captioning prompt. If include_alerts_data is True, the result can be used for Plumerai Alerts. Disabling include_thumbnails, include_captioning_data and include_alerts_data all together will slightly lower memory usage. If a scene_layout is passed, the result also says which scene the camera shows and which persons and vehicles arrived or departed, see VideoSceneData and the scene layout overview page.

The argument video_data can either be the raw bytes of a video file (e.g. an mp4 file) or an iterable that yields raw frames as tuples (frame: np.ndarray, delta_t: float). Here frame is a numpy array of shape (H, W, 3) with dtype uint8 with RGB image data (Note: red channel first, this is not the default OpenCV format). delta_t is the time in seconds since the previous frame.

# Example 1: The bytes of any video file supported by FFMPEG
video_data = Path("cat.mp4").read_bytes()

# Example 2: A frame generator
def frame_generator():
    prev_frame_time = 0
    for frame in decode_video_file():
        delta_t = frame.time - prev_frame_time
        prev_frame_time = frame.time
        yield frame.to_ndarray(), delta_t
video_data = frame_generator()

Arguments:

  • video_data: bytes of the video file or a generator of np.ndarray RGB frames with delta-t values.
  • include_thumbnails: If True, include data required for computing thumbnails.
  • include_captioning_data: If True, include data required for AI Captions.
  • include_alerts_data: If True, include data required for Plumerai Alerts.
  • night_mode: Set to True when the video was recorded by a camera in night mode (IR, greyscale).
  • scene_layout: The SceneLayout for this camera, or the bytes from its to_bytes. This requires the plumerai_scene_layout package to be installed.

Returns:

  • VideoSceneData.

compute_video_encoding

def compute_video_encoding(
    scene_data: VideoSceneData,
    video_metadata: VideoMetadata,
) -> VideoEncoding:

Compute VLM Video Search embeddings from scene data.

The resulting VideoEncoding objects can be searched over using search.

Arguments:

Returns:

  • bytes of Plumerai's VideoEncoding object.

compute_captioning_prompt

def compute_captioning_prompt(
    scene_data: VideoSceneData,
    video_encoding: VideoEncoding,
    video_metadata: VideoMetadata,
    long_caption: bool = False,
) -> CaptioningPrompt:

Compute a captioning prompt from scene data and video embeddings.

Arguments:

Returns:

  • bytes of Plumerai's CaptioningPrompt object.

encode_video

def encode_video(
    video_data: bytes | Iterable[tuple[np.ndarray, float]],
    video_metadata: VideoMetadata,
    compute_prompt: bool = False,
    long_caption: bool = False,
    night_mode: bool = False,
    *,
    scene_layout: SceneLayout | bytes | None = None,
) -> EncodeVideoResult:

Encode a video and its metadata into Plumerai's search encoding format.

This combines compute_video_scene_data, compute_video_encoding, and optionally compute_captioning_prompt into a single function for convenience.

The argument video_data can either be the raw bytes of a video file (e.g. an mp4 file) or an iterable that yields raw frames as tuples (frame: np.ndarray, delta_t: float). Here frame is a numpy array of shape (H, W, 3) with dtype uint8 with RGB image data (Note: red channel first, this is not the default OpenCV format). delta_t is the time in seconds since the previous frame.

Arguments:

  • video_data: bytes of the video file you want to encode or a generator of np.ndarray RGB frames with delta-t values.
  • video_metadata: A populated VideoMetadata object.
  • compute_prompt: If True, will compute the prompt data for AI Captions. This is only available if Plumerai AI Captions is available for you.
  • long_caption: If True, requests a longer caption (experimental).
  • night_mode: Set to True when the video was recorded by a camera in night mode (IR, greyscale).
  • scene_layout: The SceneLayout for this camera, or the bytes from its to_bytes, see compute_video_scene_data.

Returns: