Plumerai VLM Video API¶
Important: There are two possible scenarios:
- The VLM Video API is not available in your
plumerai_vision_languagepackage, i.e. thecompute_video_scene_datafunction is not available. In this case we refer to the VLM Video Collection and VLM Video Embedder API documentation, visit this page for C++ and this page for Python. - Otherwise, you can use the documentation below.
There are three stages involved in processing a video, which are covered by three functions:
compute_video_scene_data: Extract scene data from a video: important frames and object data.compute_video_encoding: Compute VLM Video Search embeddings from scene data.- Optional:
compute_captioning_prompt: Compute a prompt for generating video captions.
For convenience there is an encode_video utility function that combines all of these in one go.
Data types¶
VideoEncoding¶
Raw bytes obtained from running compute_video_encoding or encode_video.
VideoMetadata¶
Metadata input for the compute_video_encoding, compute_captioning_prompt, and encode_video functions.
When Face Identification results are available for a video, ffid_ids can be set to a list of names, i.e. ffid_ids = ["Alice", "Robert Smith", "Charlie"]. This allows searching for specific people in videos.
The device field can be left empty.
The camera_id field identifies the camera that recorded the video. Together with the camera_location_map given to search, this allows searching for videos from a specific location, see the location search example. It can be left empty when the camera is unknown.
Frame¶
Frames to be combined with search results for showing thumbnails.
The time is the time in seconds from the first frame of the video at which the thumbnail was taken. search will return one of these timestamps as the most relevant to the search query.
Note that the first frame does not always have timestamp 0 in the video file. To seek a video player to the right frame, add the start time of the video stream to time (with PyAV: stream.start_time * stream.time_base), or for constant framerate videos use the frame index round(time * fps).
These frames are also required for captioning.
VideoSceneData¶
class VideoSceneData:
selected_frames: list[Frame]
data: bytes
scene_layout: SceneLayout | None
viewpoint_status: ViewpointStatus
active_scene_id: int | None
arrival_departure_events: list[ArrivalDepartureEvent] | None
Output of compute_video_scene_data. The selected_frames can be used for showing thumbnails in search results.
The other fields are only set when a scene layout was passed to compute_video_scene_data; see the scene layout overview page:
- scene_layout: The
SceneLayoutthat was passed in, used internally for captioning. - viewpoint_status: Which of the scene layout's scenes the video shows, a
ViewpointStatusfrom theplumerai_video_intelligencepackage:MATCH_ACTIVE: the video shows the active scene of the scene layout.MATCH_INACTIVE: the video shows another scene of the scene layout.SceneLayout.set_active_scenecan make it the active one.MISMATCH: the video shows a scene that the scene layout does not hold. The video should be passed tocompute_scene_layout.UNKNOWN: no scene layout was passed, or the comparison could not be made, for example because the video is too dark.
- active_scene_id: The ID of the scene the video shows, to be passed to
SceneLayout.set_active_scene. It is-1when no scene matched, andNonewithout a scene layout. - arrival_departure_events: The persons and vehicles that arrived in or departed from the indoor space during the video, see
ArrivalDepartureEvent.Nonewithout a scene layout.
ArrivalDepartureEvent¶
class ArrivalDepartureStatus(Enum):
ARRIVED_INDOORS = 1
DEPARTED_FROM_INDOORS = 2
class ArrivalDepartureConfidence(Enum):
LOW = 0
MEDIUM = 1
HIGH = 2
class ArrivalDepartureEvent:
track_ids: list[int]
class_id: DetectionClass
status: ArrivalDepartureStatus
confidence: ArrivalDepartureConfidence
A person or vehicle that arrived in or departed from the indoor space during a video, as part of VideoSceneData.
Examples include a car exiting the garage onto the driveway (departure), or a person entering a house through the front door (arrival). This can include doors that are predicted to be next to the camera but not visible to it. A door that is predicted to connect one indoor space to another indoor space does not trigger arrivals or departures.
Whether an object arrived in or departed from the indoor space is the net result over the whole video: a person that walks out and comes back in again does not trigger an event. When person re-identification is disabled, track_ids holds a single track ID. When person re-identification is enabled, the tracks that were recognized as the same person produce a single event, and track_ids lists all of them. class_id is a DetectionClass from the plumerai_video_intelligence package.
When viewpoint_status is MATCH_INACTIVE or MISMATCH, no events are produced: the boundaries of the active scene are not where they are in this video.
EncodeVideoResult¶
class EncodeVideoResult:
video_encoding: VideoEncoding
selected_frames: list[Frame]
captioning_prompt: CaptioningPrompt | None
viewpoint_status: ViewpointStatus
active_scene_id: int | None
arrival_departure_events: list[ArrivalDepartureEvent] | None
Output of encode_video.
- video_encoding:
bytesof Plumerai'sVideoEncodingobject. - selected_frames: A list of
Frameobjects, which can be used for showing thumbnails in search results. - captioning_prompt:
bytesofCaptioningPromptifcompute_promptwasTrue, otherwiseNone. - viewpoint_status, active_scene_id, arrival_departure_events: The scene layout results, which are only set when a
scene_layoutwas passed toencode_video. SeeVideoSceneDatafor what they mean.
Functions¶
compute_video_scene_data¶
def compute_video_scene_data(
video_data: bytes | Iterable[tuple[np.ndarray, float]],
*,
include_thumbnails: bool = False,
include_captioning_data: bool = False,
include_alerts_data: bool = False,
night_mode: bool = False,
scene_layout: SceneLayout | bytes | None = None,
) -> VideoSceneData:
Extract scene data from a video, required for VLM Video Search and captioning.
Processes the video through the VideoIntelligence pipeline to extract scene information, object tracks, and other metadata required for computing the video embeddings. Optionally, it can also include data required for captioning.
The resulting data can be passed to compute_video_encoding to compute the video embeddings. If include_thumbnails is True, the result will also include thumbnails. If include_captioning_data is True, the result can be passed to compute_captioning_prompt to compute the captioning prompt. If include_alerts_data is True, the result can be used for Plumerai Alerts. Disabling include_thumbnails, include_captioning_data and include_alerts_data all together will slightly lower memory usage. If a scene_layout is passed, the result also says which scene the camera shows and which persons and vehicles arrived or departed, see VideoSceneData and the scene layout overview page.
The argument video_data can either be the raw bytes of a video file (e.g. an mp4 file) or an iterable that yields raw frames as tuples (frame: np.ndarray, delta_t: float). Here frame is a numpy array of shape (H, W, 3) with dtype uint8 with RGB image data (Note: red channel first, this is not the default OpenCV format). delta_t is the time in seconds since the previous frame.
# Example 1: The bytes of any video file supported by FFMPEG
video_data = Path("cat.mp4").read_bytes()
# Example 2: A frame generator
def frame_generator():
prev_frame_time = 0
for frame in decode_video_file():
delta_t = frame.time - prev_frame_time
prev_frame_time = frame.time
yield frame.to_ndarray(), delta_t
video_data = frame_generator()
Arguments:
- video_data:
bytesof the video file or a generator ofnp.ndarrayRGB frames with delta-t values. - include_thumbnails: If
True, include data required for computing thumbnails. - include_captioning_data: If
True, include data required for AI Captions. - include_alerts_data: If
True, include data required for Plumerai Alerts. - night_mode: Set to
Truewhen the video was recorded by a camera in night mode (IR, greyscale). - scene_layout: The
SceneLayoutfor this camera, or thebytesfrom itsto_bytes. This requires theplumerai_scene_layoutpackage to be installed.
Returns:
VideoSceneData.
compute_video_encoding¶
def compute_video_encoding(
scene_data: VideoSceneData,
video_metadata: VideoMetadata,
) -> VideoEncoding:
Compute VLM Video Search embeddings from scene data.
The resulting VideoEncoding objects can be searched over using search.
Arguments:
- scene_data: The
VideoSceneDatabytes returned bycompute_video_scene_data. - video_metadata: A populated
VideoMetadataobject.
Returns:
bytesof Plumerai'sVideoEncodingobject.
compute_captioning_prompt¶
def compute_captioning_prompt(
scene_data: VideoSceneData,
video_encoding: VideoEncoding,
video_metadata: VideoMetadata,
long_caption: bool = False,
) -> CaptioningPrompt:
Compute a captioning prompt from scene data and video embeddings.
Arguments:
- scene_data: The
VideoSceneDatabytes returned bycompute_video_scene_data. - video_encoding: The
VideoEncodingbytes returned bycompute_video_encoding. - video_metadata: A populated
VideoMetadataobject. - long_caption: If
True, requests a longer caption (experimental).
Returns:
bytesof Plumerai'sCaptioningPromptobject.
encode_video¶
def encode_video(
video_data: bytes | Iterable[tuple[np.ndarray, float]],
video_metadata: VideoMetadata,
compute_prompt: bool = False,
long_caption: bool = False,
night_mode: bool = False,
*,
scene_layout: SceneLayout | bytes | None = None,
) -> EncodeVideoResult:
Encode a video and its metadata into Plumerai's search encoding format.
This combines compute_video_scene_data, compute_video_encoding, and optionally compute_captioning_prompt into a single function for convenience.
The argument video_data can either be the raw bytes of a video file (e.g. an mp4 file) or an iterable that yields raw frames as tuples (frame: np.ndarray, delta_t: float). Here frame is a numpy array of shape (H, W, 3) with dtype uint8 with RGB image data (Note: red channel first, this is not the default OpenCV format). delta_t is the time in seconds since the previous frame.
Arguments:
- video_data:
bytesof the video file you want to encode or a generator ofnp.ndarrayRGB frames with delta-t values. - video_metadata: A populated
VideoMetadataobject. - compute_prompt: If
True, will compute the prompt data for AI Captions. This is only available if Plumerai AI Captions is available for you. - long_caption: If
True, requests a longer caption (experimental). - night_mode: Set to
Truewhen the video was recorded by a camera in night mode (IR, greyscale). - scene_layout: The
SceneLayoutfor this camera, or thebytesfrom itsto_bytes, seecompute_video_scene_data.
Returns:
- An
EncodeVideoResultobject.