Plumerai Scene Layout API¶
This document describes the API for the Scene Layout functionality. This is available through the plumerai_scene_layout and plumerai_vision_language Python packages.
See the scene layout overview page for a high-level overview of the functionality.
Data types¶
SceneLayout¶
class SceneLayout:
def __init__(self) -> None: ...
def is_active_scene_complete(self) -> bool: ...
def active_scene_id(self) -> int: ...
def active_scene_type(self) -> str | None: ...
def set_active_scene(self, scene_id: int) -> None: ...
def get_inference_data(self) -> bytes: ...
def get_viewpoint_data(self) -> bytes: ...
def to_bytes(self) -> bytes: ...
@classmethod
def from_bytes(cls, data: bytes) -> SceneLayout: ...
A description of the static parts of a camera's scene.
The scene layout is a description of some of the static parts of a scene from the viewpoint of a camera, for example where the doors, driveways and other boundaries are. This information can then be used to give more meaning to the location of detected objects, for example to see whether a person arrived at or departed from the indoor space.
One SceneLayout should be stored per camera. A new camera starts with an empty scene layout, SceneLayout(), which holds no scenes yet. There is then an initial setup phase where the scene is learned from multiple videos using compute_scene_layout. This uses AI models that are more computationally expensive than those used for regular video processing. Once the scene layout is sufficiently accurate, it remains valid until the camera's viewpoint changes, for example because the camera was moved. In that case, the new scene has to be computed in the same way, using compute_scene_layout.
The SceneLayout object contains information to verify whether the camera has moved. This information is called the viewpoint data. Viewpoint verification can produce false positives. For example, if a garage door occupies most of the image, opening the door may incorrectly appear to be a new camera viewpoint. To avoid the expensive recomputation of the scene layout for such false positive movements, the SceneLayout object can store viewpoint data for multiple scenes.
The SceneLayout object has one active scene and zero or more inactive scenes. Each scene has its own layout data and viewpoint data. When it is detected that an inactive scene is now in view again, the currently active scene can be switched. In that case, the SceneLayout object can be updated without requiring an expensive recomputation.
Constructor¶
Create an empty layout for a camera that nothing has been learned about yet.
An empty layout holds no scenes: active_scene_id is -1 and is_active_scene_complete is False. Its inference data and viewpoint data can already be passed to Plumerai Video Intelligence, which then knows that the scene layout feature is in use but has no data yet. Pass it to compute_scene_layout to learn the first scene.
is_active_scene_complete¶
Whether the currently active scene is accurate enough and needs no further videos.
When extra videos of the active scene could further improve the accuracy of classifying arrivals and departures, this function will return False.
When this is False, more videos should be processed using compute_scene_layout.
This does not indicate whether the camera is currently showing the active scene.
It is possible that the active scene is complete, but the camera has since moved to a scene that this layout does not hold yet, which then has to be computed using compute_scene_layout. When processing a video using plumerai_vision_language or plumerai_video_intelligence, the viewpoint_status result will indicate whether this is the case.
Returns:
Truewhen no further videos are needed andFalsewhen additional videos may improve the active scene.
active_scene_id¶
The Scene ID of the active scene.
The active scene is the scene of the video most recently passed to compute_scene_layout, or the one most recently set with set_active_scene.
A Scene ID stays the same for as long as this SceneLayout holds that scene, and is never given to another scene. It can be compared against the Scene ID that the plumerai_video_intelligence Python package returns for a video.
Returns:
- The Scene ID, or
-1when nothing has been learned about this camera yet.
active_scene_type¶
The scene type of the active scene.
Returns:
"outdoor","garage"or"indoor_room", orNonewhen the active scene has not been learned well enough to tell yet.
set_active_scene¶
Make the scene with this ID the active one.
Subsequent calls to get_inference_data will return the data for this scene.
Use this function when Plumerai Video Intelligence reports that a video shows a scene of this layout other than the active one: the camera has gone back to a scene that is already known, and the inference data for that scene can be shipped to the device without analyzing another video.
The scene becomes the most recently recognized one, so it is also the last to be dropped when this layout holds too many scenes.
Arguments:
- scene_id A Scene ID from
active_scene_id, or from the status that theplumerai_video_intelligencePython package returns for a video.
Raises:
ValueErrorif this layout holds no scene with that ID.
get_inference_data¶
Get the data required for inference with Plumerai Video Intelligence.
This data is used when processing videos with the plumerai_video_intelligence Python package or when doing on-device video processing.
This data is specific to the currently active scene. It changes when the active scene is switched, and when compute_scene_layout updates the layout.
Returns:
bytesto be passed to Plumerai Video Intelligence.
get_viewpoint_data¶
Get the data required to determine which stored scene the camera currently shows.
This is used with the plumerai_video_intelligence Python package, alongside get_inference_data. It reports whether a video shows the active scene, another scene of this SceneLayout, or a scene this SceneLayout does not hold. When processing videos using plumerai_vision_language, this is called automatically.
This should be passed together with the data from get_inference_data of the same SceneLayout.
Returns:
bytesto be passed to the Plumerai Video Intelligence Python package.
to_bytes, from_bytes¶
to_bytes serializes this layout for storage. It returns bytes to be stored for this camera, and passed back to from_bytes later.
from_bytes reads back a layout stored with to_bytes. It takes the bytes previously returned by to_bytes and returns the SceneLayout those bytes hold.
The serialized data can also be passed to compute_video_scene_data.
Functions¶
compute_scene_layout¶
Compute the scene layout for a camera using a video.
The SceneLayout stored for this camera is passed to this function, which returns a new SceneLayout with more accurate information. The input SceneLayout object is not modified in-place. For a new camera, pass an empty SceneLayout().
When videos are processed with Plumerai Video Intelligence, this scene layout is used to classify events such as arrivals and departures. This function should be called with a video when SceneLayout.is_active_scene_complete is False, to improve the accuracy of the active scene, or when Plumerai Video Intelligence reports that the video shows a scene this SceneLayout does not hold, to add that scene. Typically multiple video clips are needed to get accurate scene layout information; SceneLayout.is_active_scene_complete says when no further video is needed.
This function will make calls to Google's Gemini to analyze parts of the video. This is limited to a fixed token budget. Access to Gemini must have been configured before calling this function.
On Vertex AI (default) the call uses Google Application Default Credentials, so it is enough to have one of these in place:
GOOGLE_APPLICATION_CREDENTIALSpointing at a service account key file or at a workload identity federation config.- A
gcloud auth application-default loginlogin. - An attached service account, when running on Google Cloud.
Two optional environment variables override the defaults:
GEMINI_PROJECT_ID(defaults to the project of the credentials)GEMINI_LOCATION(defaults to 'global')
Alternatively, on the Gemini Developer API, which needs no Google Cloud project, set:
GEMINI_API_KEY
which takes precedence over the Vertex AI settings above.
Arguments:
- video_data
bytesof a video file. - scene_layout
SceneLayoutfor this camera, orSceneLayout()if the camera is new.
Returns:
- Updated
SceneLayoutdata.
Example¶
The example below shows how to use the scene layout data when processing a video.
The highlighted lines show the additions needed to use the scene layout functionality compared to normal usage of plumerai_vision_language without it.
from pathlib import Path
from plumerai_vision_language import (
compute_captioning_prompt,
compute_video_encoding,
compute_video_scene_data,
VideoMetadata,
)
from plumerai_scene_layout import compute_scene_layout, SceneLayout
from plumerai_video_intelligence import ViewpointStatus
async def process_video(
video_path: Path,
scene_layout_path: Path,
video_metadata: VideoMetadata,
video_encoding_path: Path,
):
if scene_layout_path.exists():
scene_layout = SceneLayout.from_bytes(scene_layout_path.read_bytes())
else:
scene_layout = SceneLayout()
# Process the video as usual. This additionally computes whether the camera
# has moved with respect to the active scene (see below).
scene_data = compute_video_scene_data(
video_path.read_bytes(),
scene_layout=scene_layout,
)
# Handle the arrival and departure events, fill in your own logic here.
print(scene_data.arrival_departure_events)
# Compute the video-search encoding
plumerai_video_encoding = compute_video_encoding(scene_data, video_metadata)
# Compute the captioning prompt.
# This internally uses the arrival and departure information.
captioning_prompt = compute_captioning_prompt(
scene_data, plumerai_video_encoding, video_metadata
)
# Request the caption asynchronously
caption = await caption_request(captioning_prompt)
plumerai_video_encoding = store_caption(plumerai_video_encoding, caption)
video_encoding_path.write_bytes(plumerai_video_encoding)
# Check if the scene layout data needs to be updated with this video.
# This can happen for different reasons:
# - The layout can be made more accurate (`is_active_scene_complete`).
# This includes a new camera without scene layout data.
# - The camera has moved (`viewpoint_status`).
# If the video was not usable (`ViewpointStatus.UNKNOWN`) because it was
# too dark for example, then there is no need to do anything here.
if (
not scene_layout.is_active_scene_complete()
or scene_data.viewpoint_status != ViewpointStatus.MATCH_ACTIVE
) and scene_data.viewpoint_status != ViewpointStatus.UNKNOWN:
# It is possible the camera was not looking at the active scene but
# a previously stored scene matches.
if scene_data.viewpoint_status == ViewpointStatus.MATCH_INACTIVE:
# Make the earlier scene active without expensive computation.
scene_layout.set_active_scene(scene_data.active_scene_id)
else:
# Update the scene layout with information from this video.
# This uses AI models that are more computationally expensive.
# This part is not latency-critical so it could be offloaded to a
# cheaper node to let the current one process new videos instead.
scene_layout = compute_scene_layout(video_path.read_bytes(), scene_layout)
scene_layout_path.write_bytes(scene_layout.to_bytes())
# The scene layout data has changed and is now valid for this camera.
# Send the on-device portion to edge devices.
# Optionally wait until it is complete before sending:
# if scene_layout.is_active_scene_complete():
inference_bytes = scene_layout.get_inference_data()
send_scene_layout_to_device(inference_bytes, ...) # Your own logic here
After regular video processing, inspect viewpoint_status and scene_layout.is_active_scene_complete. If the video matches a previously stored inactive scene, make that scene active. Otherwise, call compute_scene_layout when the active scene needs more data or when the video shows a new scene. This has to happen after regular video processing, since that determines whether the camera moved compared to the active scene.
Instead of using compute_video_scene_data, compute_video_encoding and compute_captioning_prompt it is possible to use the encode_video utility function: