Skip to content

Plumerai Scene Layout

This document gives a high-level overview of the Scene Layout functionality for Video-to-Text and arrival/departure information.

The scene layout is a description of some of the static parts of a scene from the viewpoint of a camera. It describes where doors, driveways and other boundaries are. This information can then be used to give more meaning to the location of detected objects, for example to see whether a person arrived at or departed from a house.

Currently this supports indoor garage cameras and outdoor cameras that are mounted near a door leading inside (for example video doorbells).

For a newly placed camera there is an initial setup phase where this scene layout data is computed from multiple videos. This uses AI models that are more computationally expensive than those used for regular video processing. Once the scene layout is accurate enough, it stays valid until the camera's viewpoint changes.

There are two scene layout components:

  • Regular video processing: process every video to get results such as search embeddings, captions, and arrival/departure information. This uses scene layout data as input.
  • Generating new scene layout data: process specific videos when the scene layout data needs updating.

The two components work together as follows. Store one SceneLayout object per camera, starting with an empty SceneLayout() for a new camera. For every new video from that camera:

  • Run regular video processing with the scene layout as input.
  • Use the outputs such as arrival and departure events.
  • Check the result flags to see if the SceneLayout object needs updating.

For more details about the API, and for example code, visit the API page.