Installation¶
This document describes how to install the Plumerai Vision Language library (provided as plumerai_vision_language-<version>.tar), as well as the optional software needed for Plumerai Video-to-Text (provided as separate files).
Plumerai Vision Language library¶
Extract the plumerai_vision_language-<version>.tar file into a directory of your choice. For the purpose of this documentation, we assume vision_language/: You might have already done this, e.g.:
Then install the library using the following command:
pip install 'plumerai_vision_language[video-processing]' --no-cache-dir --find-links vision_language/
If you did not license cloud-based video processing (only on-device video-processing), then omit the video-processing extra:
Optimization: processing multiple videos in parallel¶
The video processing functions are thread-safe, but they do not release the Python GIL, so running them in multiple Python threads gives no parallel speedup. Use one process per concurrently processed video instead.
Each process can still use multiple internal threads to speed up the processing of a single video. The environment variable PLUMERAI_NUM_THREADS (default 1) controls how many are used per video. When running multiple processes, make sure PLUMERAI_NUM_THREADS multiplied by the number of processes is not higher than the number of CPU cores available. Increasing PLUMERAI_NUM_THREADS while decreasing the number of parallel processes will improve per-video latency at the cost of a lower total throughput.
By default each process loads its own copy of the model weights. To load them only once and share them between processes, set the environment variable PLUMERAI_USE_CACHE=1 for every process. This weight memory sharing then happens through memory-mapped files in /tmp, which must be a tmpfs.
If this runs in Kubernetes, make sure to either run these processes within the same container, or to configure a tmpfs volume mounted at /tmp in each container, shared across the containers of a pod.
If /tmp is not usable, set PLUMERAI_CACHE_PATH=/some/other/path to choose a different tmpfs directory, and make sure all processes use the same path.
Optional: Plumerai Real-time Alerts¶
If you licensed the Plumerai Real-time Alerts feature, you should also install the plumerai_alerts package:
Optional: Plumerai Scene Layout¶
If you also licensed the Plumerai Scene Layout feature, you should also install the plumerai_scene_layout package:
Optional: Captioning Server¶
This step is only needed for the Video-to-Text functionality and only if you have received the plumerai_captioning-<version>.tar and plumerai-llm-v*.tar files from Plumerai.
Note: It is highly recommended to do the following installation in a separate environment from the Plumerai Vision Language library as they have conflicting dependencies.
Download and extract the plumerai_captioning-<version>.tar file into a directory of your choice. For the purpose of this documentation, we assume captioning_serve_wheels/, e.g.:
mkdir captioning_serve_wheels
tar -xf plumerai_captioning-<version>.tar --directory captioning_serve_wheels
Then install the library using the following command (will install CUDA 13.0 automatically). We recommend using uv for faster installation:
uv pip install plumerai_captioning_cli --no-cache-dir --find-links captioning_serve_wheels/ --torch-backend=cu130 --extra-index-url https://flashinfer.ai/whl/cu130
Alternatively it can be installed with pip using:
pip install plumerai_captioning_cli --no-cache-dir --find-links captioning_serve_wheels/ --extra-index-url https://download.pytorch.org/whl/cu130 --extra-index-url https://flashinfer.ai/whl/cu130
Note: When upgrading, install into a new environment as described above. Installing over the environment of an earlier version keeps the old packages and can leave the server unable to start.
This is for running a vLLM server with Plumerai's middleware and commands optimised for video caption generation with the Plumerai prompt and LLM.
Next, extract the plumerai-llm-v*.tar file into a directory of your choice (e.g. captioning_llm/) you can start the server from the environment where you installed plumerai_captioning_cli:
See help for a full list of arguments.
Optimize Deployment¶
The captioning server should be run on a GPU instance. For optimal cost efficiency this should be a g7e.2xlarge or g6e.xlarge GPU instance when using the AWS cloud, with the plumerai_vision_language library running on a separate c8a CPU instance.
The model weights alone take 35 GB of GPU memory, so a 32 GB GPU cannot hold them. On an instance with several such GPUs, such as a g7.12xlarge with two 32 GB GPUs, split the model over the GPUs with --tensor-parallel-size:
This is a fallback for when the single-GPU instance types are unavailable: it costs more per caption than a g7e.2xlarge and is slower.
Improve Memory Usage¶
We recommend using Jemalloc for lower memory fragmentation and slightly improved performance. This can be enabled via the LD_PRELOAD environment variable. On Ubuntu this can be done as follows:
Improve Startup Time¶
On its first run, the captioning server compiles and caches model artifacts in ~/.cache/vllm/ and ~/.triton/. You can share this cache across multiple machines to speed up subsequent cold starts.
Warning: Do not share the cache between different GPU types, as these artifacts are strictly tied to the GPU architecture and plumerai_vision_language version.
Health Checks, Logs and Traces¶
Health Checks¶
After starting up, the captioning server sends itself two captioning requests to compile the remaining GPU kernels, which takes a few seconds. Until that is done, GET /health returns status 503, afterwards it returns 200. Use /health as the readiness check of the server, so that it only receives requests once these are fast.
JSON Logs¶
Pass --json-logs (or set the environment variable PLUMERAI_JSON_LOGS=1) to log to stdout with one JSON object per line. Each line has the fields timestamp, level, logger, process and message. Some lines have extra fields, such as exc_info with the traceback of an exception. Each request is logged by the logger uvicorn.access, with the method, path and status code in the message.
Startup progress bars and some warnings from third-party libraries are not JSON and are written to stderr.
Traces¶
The captioning server can send a trace of each request to an OpenTelemetry collector, such as a Datadog Agent with OTLP ingestion enabled. Pass the collector's address with --otlp-traces-endpoint (or the environment variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT), e.g. --otlp-traces-endpoint http://localhost:4317. By default the traces are sent with gRPC; set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf to send them over HTTP instead. With HTTP, the endpoint must include the path, e.g. http://localhost:4318/v1/traces. Set the service name with the environment variable OTEL_SERVICE_NAME, and add tags with OTEL_RESOURCE_ATTRIBUTES, e.g. OTEL_RESOURCE_ATTRIBUTES=env=prod.
Each request produces a span named llm_request. Its attributes contain, among others, the time the request waited in the queue (gen_ai.latency.time_in_queue), the time to the first generated token (gen_ai.latency.time_to_first_token), the total time (gen_ai.latency.e2e), and the number of input and output tokens (gen_ai.usage.prompt_tokens and gen_ai.usage.completion_tokens). The server also sends spans about its startup, such as loading the model. If a request has an x-request-id header, the span's gen_ai.request.id is chatcmpl- followed by its value. If a request carries a W3C traceparent header, its span becomes part of that trace. Requests that are stopped because the client disconnected do not produce a span.