Examples¶
VLM Video Processing¶
There are multiple stages involved in processing a video, covered by the functions compute_video_scene_data, compute_video_encoding, and optionally compute_captioning_prompt. For convenience there is an encode_video utility function that combines all of these in one go.
Example: Processing a video¶
from pathlib import Path
from plumerai_vision_language import compute_video_encoding, compute_video_scene_data, VideoMetadata
# The bytes of any video file supported by FFMPEG, e.g.:
video_bytes = Path("cat.mp4").read_bytes()
scene_data = compute_video_scene_data(video_bytes, include_thumbnails=True)
metadata = VideoMetadata(unix_timestamp=0, ffid_ids=["a","b"],device="CamV1")
plumerai_video_encoding = compute_video_encoding(scene_data, video_metadata)
thumbnails = scene_data.selected_frames
The function compute_video_scene_data returns the Plumerai VideoSceneData object (in bytes) to be used as input to the next step. The function compute_video_encoding returns the Plumerai VideoEncoding object (in bytes) to be used for search. These VideoEncoding objects should be stored for each video but separately per user. The VideoSceneData object also contains a list of thumbnails, these each have an image attribute and a time attribute, these should be combined with the time from the search result to show a thumbnail that matches a search query.
In case the video has already been decoded, compute_video_scene_data accepts a generator function that yields RGB frames as numpy arrays together with a delta_t value that specifies the time between frames in seconds. An example of this is shown below:
from plumerai_vision_language import compute_video_scene_data
def frame_generator():
prev_frame_time = 0
for frame in decode_video_file(): # decode_video_file to be implemented
delta_t = frame.time - prev_frame_time
prev_frame_time = frame.time
yield frame.to_ndarray(), delta_t
scene_data = compute_video_scene_data(frame_generator())
Example: Encode video¶
The above example can be shortened with the encode_video utility function.
from pathlib import Path
from plumerai_vision_language import encode_video, VideoMetadata
video_bytes = Path("cat.mp4").read_bytes()
metadata = VideoMetadata(unix_timestamp=0, ffid_ids=["a","b"],device="CamV1")
result = encode_video(video_bytes, metadata)
plumerai_video_encoding = result.video_encoding
thumbnails = result.selected_frames
VLM Text Search¶
Example: Regular video search¶
See the VLM Text Search API documentation.
Example: Individual video scoring¶
See the VLM Text Search API documentation.
Example: FFID names search¶
When encoding a video, you can provide a list of the names of users that were identified by Plumerai's FFID (Familiar Face Identification) in the ffid_ids field of VideoMetadata:
from pathlib import Path
from plumerai_vision_language import compute_video_encoding, compute_video_scene_data, VideoMetadata
# The bytes of any video file supported by FFMPEG, e.g.:
video_bytes = Path("cat.mp4").read_bytes()
metadata = VideoMetadata(
unix_timestamp=0,
ffid_ids=["Alice", "Robert Smith", "Charlie"],
device="camera1"
)
scene_data = compute_video_scene_data(video_bytes)
plumeari_video_encoding = compute_video_encoding(scene_data, metadata)
With this, search can now filter by the people identified in a video, e.g.:
from plumerai_vision_language import search
# Video 1 has a video of cat with no ffid_ids
video_encoding_1 = ...
# Video 2 has Robert Smith delivering a Fedex package
video_encoding_2 = ...
user_history = [video_encoding_1, video_encoding_2]
# This query won't match either video because "Alice" wasn't included in
# the ffid_ids for video 1
search(user_history, "a cat with Alice")
# Example output: SearchResult(..., is_match=[IsMatch.NO, IsMatch.NO])
# This query matches video 2 because "Robert Smith" is in ffid_ids we match
# "Robert" to "Bob" (we keep a lookup table to match names to nicknames if they
# are used in search queries) in the query AND a Fedex delivery is occurring.
search(user_history, "Bob making a Fedex delivery")
If the FFID names are not yet known when the video is encoded, they can be stored on the VideoEncoding afterwards with store_ffid_ids. This replaces any names that were passed in the VideoMetadata:
from plumerai_vision_language import store_ffid_ids
video_encoding_1 = store_ffid_ids(video_encoding_1, ["Alice"])
The set of possible names that can be matched is the union of the set of all of the names from each VideoEncoding in the user_history. If a name isn't present in the user_history it won't be parsed from the search query (if it appears in a search query) and therefore won't affect search results.
Example: location search¶
When encoding a video, you can identify the camera that recorded it in the camera_id field of VideoMetadata:
from plumerai_vision_language import compute_video_encoding, compute_video_scene_data, VideoMetadata
video_bytes = Path("cat.mp4").read_bytes()
metadata = VideoMetadata(
unix_timestamp=0,
ffid_ids=[],
device="SmartCameraV1",
camera_id="cam-1",
)
scene_data = compute_video_scene_data(video_bytes)
plumerai_video_encoding = compute_video_encoding(scene_data, metadata)
With this, search can filter by location when it is given the locations of each camera. When a query names one of the locations, only videos from a camera at that location can match:
from plumerai_vision_language import search
# Video 1 is a cat recorded by "cam-1"
video_encoding_1 = ...
# Video 2 is a delivery recorded by "cam-2"
video_encoding_2 = ...
user_history = [video_encoding_1, video_encoding_2]
# Only video 1 is at the front door, so video 2 is IsMatch.NO regardless of
# its content. Video 1 is searched for "a cat".
camera_location_map = {"cam-1": ["front door", "porch"], "cam-2": ["driveway"]}
search(user_history, "a cat at the front door", camera_location_map=camera_location_map)
# Example output: SearchResult(..., is_match=[IsMatch.YES, IsMatch.NO])
Locations are matched as whole words, ignoring case and punctuation, together with a preceding preposition and determiner (e.g. "at the front door"). A camera can have several locations, and a query that names any of them matches the camera. A location that is not in camera_location_map is left in the query and does not affect the results. Videos without a stored camera_id, or from a camera that is not at a location in the query, never match a query that names a location.
Video-to-Text (optional)¶
After starting the captioning server (see the installation instructions), you can start using it by following the examples below.
Example: Making a request¶
Example code for making a request to your running vLLM server. The compute_prompt argument must be set to True to compute the necessary captioning_prompt, else captioning_prompt is None.
from pathlib import Path
import requests
from plumerai_vision_language import encode_video, VideoMetadata
def make_request(prompt_data: bytes) -> str:
headers = {
"content-type": "application/plumerai",
}
url = "http://0.0.0.0:8000/v1/chat/completions"
response = requests.post(url, data=prompt_data, headers=headers)
out = response.json()
return out["caption"]
# The bytes of any video file supported by FFMPEG, e.g.:
video_bytes = Path("cat.mp4").read_bytes()
metadata = VideoMetadata(unix_timestamp=0, ffid_ids=["a","b"], device="CamV1")
result = encode_video(video_bytes, metadata, compute_prompt=True)
plumerai_video_encoding = result.video_encoding
caption = make_request(result.captioning_prompt)
assert caption == "A cat was walking around in your back yard."
Example: Adding captions to the VideoEncoding object¶
Captions can be stored on video encodings to improve search performance, which is particularly useful for allowing users to search for notifications they have just received. When the informative words from a search query are all found in a caption, the video will receive an IsMatch.YES result.
from plumerai_vision_language import store_caption, search
# After generating a caption, add it to the video encoding
video_encoding_with_caption = store_caption(plumerai_video_encoding, caption)
# Improves search accuracy and consistency with captions
search_result = search([video_encoding_with_caption], "cat in yard")
Example: Long captions¶
We now provide long captions as an experimental feature. To get the corresponding prompt to generate long captions set long_caption=True in encode_video, e.g.:
from plumerai_vision_language import encode_video
long_caption_prompt_data = encode_video(
video_bytes, video_metadata, compute_prompt=True, long_caption=True
).captioning_prompt
Example: FFID names in captions¶
We support adding names to captions - currently only if a video has a single person. We get Plumerai's FFID (Familiar Face Identification) information from the same VideoMetadata as is used in search. For example:
from pathlib import Path
from plumerai_vision_language import encode_video, VideoMetadata
video_bytes = Path("person.mp4").read_bytes()
# Without FFID names - generic caption
metadata_without_names = VideoMetadata(
unix_timestamp=0,
ffid_ids=[],
device="SmartCameraV1"
)
prompt_data = encode_video(
video_bytes, metadata_without_names, compute_prompt=True
).captioning_prompt
caption = make_request(prompt_data)
assert caption == "A person was delivering a package to your front door."
# With FFID names - personalized caption
metadata_with_names = VideoMetadata(
unix_timestamp=0,
ffid_ids=["Alice"],
device="SmartCameraV1"
)
prompt_data = encode_video(
video_bytes, metadata_with_names, compute_prompt=True
).captioning_prompt
caption = make_request(prompt_data)
assert caption == "Alice was delivering a package to your front door."
Example: Providing device metadata to captioning prompt¶
Providing the device name for known devices improves captioning performance as this can be used to provide additional context to the LLM e.g.
from plumerai_vision_language import encode_video, VideoMetadata
metadata = VideoMetadata(
unix_timestamp=0,
ffid_ids=[],
device="VideoKeypadCam"
)
prompt_data = encode_video(video_bytes, metadata, compute_prompt=True).captioning_prompt
caption = make_request(prompt_data)
assert caption == "A person opened the front door and entered."
We support device names containing specific substrings only, ask Plumerai for an exact list. Other device names will be ignored.
Optimization: model loading¶
The respective models are loaded on the fly for each API function to save memory. If you need to preload the model then you can call the respective API function with some valid inputs. For example: