Plumerai VLM Video API¶
This page documents the API for the VLM Video Collection and VLM Video Embedder components.
These components are used in Plumerai Video Search and Plumerai Custom AI Notifications. See those product pages for an overview of how these components are used in practice.
For context, the architecture diagram from the Video Search product page is included below:
To use the text embedder and matcher components, refer to the VLM Text Search API documentation.
See the minimal examples for example useage of the VLM Video Collection and VLM Video Embedder APIs.
VLMVideoCollection¶
start_clip¶
Start a new clip to collect data for the VLM Video Embedder.
The user needs to decide when data collection should start, e.g. when an object is detected or when the camera awakes through a motion event. The VLMVideoCollection then will start collecting the necessary data for the VLM Video Embedder from this point onwards for every call to process_frame
, until end_clip
is called.
The results will be available when calling end_clip
. See end_clip
for example usage.
Returns:
- Returns
SUCCESS
on success, orCLIP_ALREADY_STARTED
when a clip was already started.
end_clip¶
Ends the data collection for the clip started with start_clip
.
The results will be moved into the clip_data
buffer.
Arguments:
- clip_data: A buffer to store the collected data in. Any existing data is overwritten.
Returns:
- Returns
SUCCESS
on success,CLIP_NOT_YET_STARTED
when a clip was not started previously or was already ended before.
Example:
auto pvi = plumerai::VideoIntelligence(height, width);
bool clip_in_progress = false;
auto clip_data = std::vector<std::uint8_t>(0);
while(true) { // video frames loop
pvi.process_frame(...);
if (...) { // e.g. predictions found, user input, fixed time
auto error_code = pvi.vlm_video_collection().start_clip();
if (error_code != plumerai::ErrorCode::SUCCESS) return;
clip_in_progress = true;
}
if (clip_in_progress && ...) { // e.g. no more predictions found
auto error_code = pvi.vlm_video_collection().end_clip(clip_data);
if (error_code != plumerai::ErrorCode::SUCCESS) return;
clip_in_progress = false;
// `clip_data` can be stored or processed with `VLMVideoEmbedder`
}
VLMVideoEmbedder¶
compute_embeddings¶
ErrorCode compute_embeddings(const std::uint8_t* clip_data,
size_t clip_data_size,
std::vector<std::uint8_t>& clip_embeddings,
bool compute_single_unit_only = false) const;
Compute video embeddings on data collected using VLMVideoCollection
.
Depending on the size of the collected data, this can be compute-heavy. The user can optionally set compute_single_unit_only
to compute only a single part of the video embeddings. When compute_single_unit_only
is set, this needs to be called in a loop with the same arguments. The error code return value will inform the user whether all units were computed, and whether the results are valid.
Arguments:
- clip_data: A pointer to the data collected using
VLMVideoCollection::start_clip
andVLMVideoCollection::end_clip
. - clip_data_size: The size of the data collected using
VLMVideoCollection::start_clip
andVLMVideoCollection::end_clip
. - clip_embeddings: The resulting embeddings. These are only valid when the function returns
SUCCESS
. - compute_single_unit_only: Can be set to do a partial computation of the embeddings. If set, this needs to be called in a loop, see above.
Returns:
- Returns
SUCCESS
when all embeddings have been computed,EMBEDDING_PART_COMPUTED
when a single unit was successfully computed (but not everything), orINVALID_CLIP_DATA
when the clip data is invalid.
Example:
auto clip_data = ...; // from `VLMVideoCollection::end_clip`.
auto pvllme = plumerai::VLMVideoEmbedder();
auto clip_embeddings = std::vector<std::uint8_t>(0);
error_code = pvllme.compute_embeddings(clip_data.data(),
clip_data.size(),
clip_embeddings);
if (error_code != plumerai::ErrorCode::SUCCESS) return;
// `clip_embeddings` can be stored for e.g. video search.