Skip to content

Plumerai Real-time Alerts API

This document describes the API for the Alerts functionality. This is available through the plumerai_alerts and plumerai_vision_language Python packages.

This component is part of Plumerai Real-time Alerts. See that product page for architectural context and real-world usage examples.

For the VLM Video Collection and VLM Video Embedder API documentation, visit this page.

Data types

AlertObject

AlertObject: TypeAlias = bytes

Raw bytes obtained from running setup_alert.

Functions

setup_alert

def setup_alert(
    query: str,
    *,
    ffid_library_names: Sequence[str] = (),
    camera_location_map: Mapping[str, Sequence[str]] | None = None,
) -> AlertObject

Create an AlertObject for a query.

The query is analyzed for names of enrolled people and camera location names. A check then only matches clips in which Familiar Face Identification recognized the named person, and clips from cameras at the named location. The names and locations given here are stored in the AlertObject: enrolling or renaming someone later does not change an existing AlertObject. Both arguments are optional. Without them the alert matches on the query text alone, as before.

The calls to Gemini never contain an enrolled name: a recognized name is replaced with 'a person' before the query is analyzed or a match is judged.

The AlertObject will be updated by every call to check_for_alert. For optimal accuracy, it is recommended to create a separate AlertObject per camera.

This function will make a call to Google's Gemini to analyse the query. This is limited to a single call with a fixed token budget. Access to Gemini must have been configured before calling this function.

On Vertex AI (default) the call uses Google Application Default Credentials, so it is enough to have one of these in place:

  • GOOGLE_APPLICATION_CREDENTIALS pointing at a service account key file or at a workload identity federation config.
  • A gcloud auth application-default login login.
  • An attached service account, when running on Google Cloud.

Two optional environment variables override the defaults:

  • GEMINI_PROJECT_ID (defaults to the project of the credentials)
  • GEMINI_LOCATION (defaults to 'global')

Alternatively, on the Gemini Developer API, which needs no Google Cloud project, set:

  • GEMINI_API_KEY

which takes precedence over the Vertex AI settings above.

Arguments:

  • query: What to match on, e.g. 'a person smoking'.
  • ffid_library_names: The names enrolled for Familiar Face Identification.
  • camera_location_map: Camera IDs mapped to that camera's location names, as in search.

Returns:

check_for_alert

def check_for_alert(
    alert_object: AlertObject,
    scene_data: VideoSceneData,
    video_encoding: VideoEncoding,
) -> tuple[bool, SearchResultMetadata | None, AlertObject | None]

Check if a video matches the query in an AlertObject.

An AlertObject must have been setup using setup_alert.

This function reports where in the video the match was found, as a SearchResultMetadata object. It is None when the video does not match the query.

This function can return an updated AlertObject. When it does, it must be stored and passed to the next check. Accuracy and cost both improve as more videos are checked. The function returns None instead of an AlertObject when this check did not modify the AlertObject.

Access to Gemini must have been configured before calling this function, see setup_alert for more information. This function can make at most a single call to Gemini, with a fixed token budget.

The scene_data must have been computed with the option include_alerts_data=True, see compute_video_scene_data.

If two videos are processed in parallel, either of the resulting AlertObjects can be stored and the other ignored.

Arguments:

Returns:

  • A tuple of a boolean indicating whether the video matches the query, the SearchResultMetadata of the match or None if the video does not match, and an updated AlertObject, or None if the AlertObject is unchanged.

Example

To check whether a video triggers matches a query, it needs to be processed using compute_video_scene_data and compute_video_encoding from the plumerai_vision_language package.

from pathlib import Path
from plumerai_vision_language import (
    compute_video_encoding,
    compute_video_scene_data,
    VideoMetadata,
)
from plumerai_alerts import check_for_alert, setup_alert

# One-time setup
alert_objects = {
    "person smoking": setup_alert("person smoking"),
    "ambulance": setup_alert("ambulance"),
    "Chris at the front door": setup_alert(
        "Chris at the front door",
        ffid_library_names=["Chris"],
        camera_location_map={"cam-1": ["front door"]},
    ),
}

# Process a video
video_bytes = Path("video.mp4").read_bytes()
scene_data = compute_video_scene_data(video_bytes, include_alerts_data=True)
metadata = VideoMetadata.default()
video_encoding = compute_video_encoding(scene_data, metadata)

# Check if the video triggers any of the alerts
for alert_string, alert_object in alert_objects.items():
    is_alert_triggered, match_metadata, new_alert_object = check_for_alert(
        alert_object, scene_data, video_encoding
    )
    # Store the new `AlertObject` if it was updated
    if new_alert_object is not None:
        alert_objects[alert_string] = new_alert_object