Skip to content

Plumerai Video Intelligence examples

This page contains some examples of how to use the Plumerai Video Intelligence library to perform simple tasks. If you are looking for a minimal example, please also check out the examples under the API pages:

  1. The C API example.
  2. The C++ API example.
  3. The Python API example.

Example programs on this page:

Common code and usage instructions

All the examples on this page use OpenCV as a camera input source, however that is just a choice: the library can work with any video input source. Since OpenCV has both a C++ and a Python API, we show examples for both on this page. However, we do not show an example of the Plumerai Video Intelligence C API, since OpenCV has no proper C API anymore since version 2.4. If you wish to see more examples of our C API, please contact Plumerai.

To be able to compile and run the examples on this page, please install OpenCV first:

apt install libopencv-dev libopencv-highgui-dev libopencv-videoio-dev libv4l-dev

Download from the OpenCV website or follow the official instructions.

In case you want to run the Python examples, you'll additionally need Python 3.7 or newer, pip, the opencv-python pip package, and the Plumerai Video Intelligence Python wheel.

Since all examples below share the same camera initialization code, that part can be stored in a common header file init_camera.h (C++) or init_camera.py (Python):

init_camera.h
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#pragma once

#include "opencv2/core.hpp"
#include "opencv2/videoio.hpp"

cv::VideoCapture init_camera(int height, int width) {
  constexpr auto camera_capture_api = cv::CAP_V4L;  // (1)
  auto capture_format = cv::VideoWriter::fourcc('M', 'J', 'P', 'G');  // (2)
  constexpr int camera_id = 0;  // (3)

  cv::VideoCapture camera(camera_id, camera_capture_api);
  camera.set(cv::CAP_PROP_FRAME_HEIGHT, height);
  camera.set(cv::CAP_PROP_FRAME_WIDTH, width);
  camera.set(cv::CAP_PROP_FOURCC, capture_format);

  if (!camera.isOpened()) {
    throw std::runtime_error("Unable to open video source");
  }
  return camera;
}

  1. This line sets the capture API to Video4Linux. Change this as needed for your system. See the OpenCV website for a list of all options.

  2. This line sets the camera format to Motion-JPEG. If your camera does not support this, then it could be changed to another format.

  3. Here the camera to use is selected. In case of multiple cameras, 0 is the first camera, 1 is the second, etc.

init_camera.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import cv2 as cv

def init_camera(height: int, width: int) -> cv.VideoCapture:
    camera_capture_api = cv.CAP_V4L  # (1)
    capture_format = cv.VideoWriter.fourcc("M", "J", "P", "G")  # (2)
    camera_id = 0  # (3)

    camera = cv.VideoCapture(camera_id, camera_capture_api)
    camera.set(cv.CAP_PROP_FRAME_HEIGHT, height)
    camera.set(cv.CAP_PROP_FRAME_WIDTH, width)
    camera.set(cv.CAP_PROP_FOURCC, capture_format)

    if camera is None or not camera.isOpened():
        raise RuntimeError("Unable to open video source")
    return camera

  1. This line sets the capture API to Video4Linux. Change this as needed for your system. See the OpenCV website for a list of all options.

  2. This line sets the camera format to Motion-JPEG. If your camera does not support this, then it could be changed to another format.

  3. Here the camera to use is selected. In case of multiple cameras, 0 is the first camera, 1 is the second, etc.

Running one of the examples below is as simple as copy-pasting the code into a file example.cpp (C++) or example.py (Python) in the same folder as the above init_camera.{h,py} and followed by:

g++ -std=c++11 -O3 example.cpp -I/path/to/plumeraivideointelligence/include/ /path/to/plumeraivideointelligence/lib/<PLATFORM>/libplumeraivideointelligence.a `pkg-config --cflags --libs opencv4` -ldl -pthread -o example
./example

python example.py

People Counting

People counting using the Plumerai People Detection library is as simple as storing the highest found person ID value. This works since the library assigns the ID 1 to the first person it detects, ID 2 to the second, and so on.

6 people in a hallway 3 people in a hallway

Example of the people counting application.

Below is a simple people counting example that prints out the number of unique people it has encountered so far everytime this value changes.

The highlighted lines indicate the use of the Plumerai Video Intelligence API.

people_counting.cc
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
#include <algorithm>
#include <iostream>

#include "init_camera.h"
#include "plumerai/video_intelligence.h"

using namespace plumerai;

int main() {
  constexpr int height = 720;  // example camera image height in pixels
  constexpr int width = 1280;  // example camera image width in pixels
  cv::VideoCapture camera = init_camera(height, width);

  auto pvi = plumerai::VideoIntelligence(height, width);

  std::cout << "Starting to count people" << std::endl;
  size_t num_unique_people = 0;
  size_t highest_id_found = 0;

  cv::Mat camera_image;
  while (true) {
    const auto camera_read_success = camera.read(camera_image);
    if (!camera_read_success) {
      throw std::runtime_error("Could not read frame from camera");
    }

    auto error_code = pvi.process_frame(
        ImagePointer<ImageFormat::PACKED_RGB888>(camera_image.data));
    if (error_code != ErrorCode::SUCCESS) {
      printf("Error code: %s\n", error_code_string(error_code));
      return 1;
    }

    std::vector<BoxPrediction> predictions;
    error_code = pvi.object_detection().get_detections(predictions);
    if (error_code != ErrorCode::SUCCESS) {
      printf("Error code: %s\n", error_code_string(error_code));
      return 1;
    }

    std::sort(  // (1)
        predictions.begin(), predictions.end(),
        [](BoxPrediction &a, BoxPrediction &b) -> bool { return a.id < b.id; });
    for (auto &p : predictions) {  // (2)
      if (p.class_id != CLASS_PERSON) continue;
      if (p.id > highest_id_found) {  // (3)
        num_unique_people++;
        highest_id_found = p.id;
        std::cout << "People count: " << num_unique_people << std::endl;
      }
    }
  }
}

  1. If multiple people are found, we sort them by ID to make the people counting logic below work.

  2. Here we loop over all the predictions returned by process_frame. If no people are detected, then this loop will not execute.

  3. Here p.id is the tracker ID returned by the Plumerai Video Intelligence library. We can use that to count the number of people detected so far.

people_counting.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
from init_camera import init_camera
import plumerai_video_intelligence as pvi_api

height = 720  # example camera image height in pixels
width = 1280  # example camera image width in pixels
camera = init_camera(height, width)

pvi = pvi_api.VideoIntelligence(height, width)

print("Starting to count people")
num_unique_people, highest_id_found = 0, 0

while True:
    camera_read_success, camera_image = camera.read()
    if not camera_read_success:
        raise RuntimeError("Could not read frame from camera")

    pvi.process_frame(camera_image.data)
    _, predictions = pvi.object_detection.get_detections()

    for p in sorted(predictions, key=lambda x: x.id):  # (1)
        if p.class_id != pvi_api.DetectionClass.CLASS_PERSON:
            continue
        if p.id > highest_id_found:  # (2)
            num_unique_people += 1
            highest_id_found = p.id
            print(f"People count: {num_unique_people}")

  1. Here we loop over all the predictions returned by process_frame. If no people are detected, then this loop will not execute. If multiple people are found, we sort them by ID to make the people counting logic below work.

  2. Here p.id is the tracker ID returned by the Plumerai Video Intelligence library. We can use that to count the number of people detected so far.

Privacy mask

The Plumerai Video Intelligence library can also be easily used to automatically apply a person privacy mask on a video stream.

A person masked for privacy Two people masked for privacy

Example of the privacy mask application.

Below is a simple privacy masking example that pixelates all areas with people and displays the results to screen.

The highlighted lines indicate the use of the Plumerai Video Intelligence API.

privacy_mask.cc
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
#include <algorithm>

#include "init_camera.h"
#include "opencv2/highgui/highgui.hpp"
#include "plumerai/video_intelligence.h"

using namespace plumerai;

int main() {
  constexpr int height = 720;  // example camera image height in pixels
  constexpr int width = 1280;  // example camera image width in pixels
  cv::VideoCapture camera = init_camera(height, width);

  auto pvi = plumerai::VideoIntelligence(height, width);

  cv::Mat camera_image;
  while (true) {
    const auto camera_read_success = camera.read(camera_image);
    if (!camera_read_success) {
      throw std::runtime_error("Could not read frame from camera");
    }

    auto error_code = pvi.process_frame(
        ImagePointer<ImageFormat::PACKED_RGB888>(camera_image.data));
    if (error_code != ErrorCode::SUCCESS) {
      printf("Error code: %s\n", error_code_string(error_code));
      return 1;
    }

    std::vector<BoxPrediction> predictions;
    error_code = pvi.object_detection().get_detections(predictions);
    if (error_code != ErrorCode::SUCCESS) {
      printf("Error code: %s\n", error_code_string(error_code));
      return 1;
    }

    for (auto &p : predictions) {  // (1)
      if (p.class_id != CLASS_PERSON) continue;

      // Transform relative coordinates into pixel coordinates (2)
      int x_min = std::max(0, std::min((int)(p.x_min * width), width - 1));
      int y_min = std::max(0, std::min((int)(p.y_min * height), height - 1));
      int x_max = std::max(0, std::min((int)(p.x_max * width), width - 1));
      int y_max = std::max(0, std::min((int)(p.y_max * height), height - 1));

      // Pixelate the boxes to preserve privacy
      constexpr auto pixelate_size = 24;  // (3)
      for (int y = y_min; y < y_max; y += pixelate_size) {
        for (int x = x_min; x < x_max; x += pixelate_size) {
          // Reference coordinates for the large pixel (4)
          const auto y_ref = std::min(y + pixelate_size / 2, height - 1);
          const auto x_ref = std::min(x + pixelate_size / 2, width - 1);
          for (int c = 0; c < 3; ++c) {  // (5)
            auto value = camera_image.data[(y_ref * width + x_ref) * 3 + c];
            for (int y2 = y; y2 < y + pixelate_size && y2 < y_max; y2++) {
              for (int x2 = x; x2 < x + pixelate_size && x2 < x_max; x2++) {
                camera_image.data[(y2 * width + x2) * 3 + c] = value;
              }
            }
          }
        }
      }
    }

    // Display the results on screen
    cv::imshow("Plumerai demo", camera_image);
    cv::waitKey(30);  // in ms; can be changed to force a certain framerate
    if (cv::getWindowProperty("Plumerai demo", 0) == -1) {
      return 0;  // Exit when the X button is clicked
    }
  }
}

  1. Here we loop over all the predictions returned by process_frame. If no people are detected, then this loop will not execute.

  2. The coordinates returned by process_frame are in floating-point between 0 and 1. In these four lines they are transformed to integer pixel coordinates to lie in the range of (0, width - 1) and (0, height - 1).

  3. The size of the pixels used for the privacy mask can be changed here: the larger, the less recognizable people are.

  4. In these two lines we compute the center reference coordinates of the large pixel, see the description of the intRoundTo function above.

  5. This is a loop over 3 colours in an RGB image.

privacy_mask.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
import cv2 as cv
import numpy as np
from init_camera import init_camera
import plumerai_video_intelligence as pvi_api

height = 720  # example camera image height in pixels
width = 1280  # example camera image width in pixels
camera = init_camera(height, width)

pvi = pvi_api.VideoIntelligence(height, width)

while True:
    camera_read_success, camera_image = camera.read()
    if not camera_read_success:
        raise RuntimeError("Could not read frame from camera")
    camera_data = np.asarray(camera_image.data)

    pvi.process_frame(camera_data)
    _, predictions = pvi.object_detection.get_detections()

    for p in predictions:
        if p.class_id != pvi_api.DetectionClass.CLASS_PERSON:
            continue

        # Transform relative coordinates into pixel coordinates
        x_min = max(0, min(int(p.x_min * width), width - 1))
        y_min = max(0, min(int(p.y_min * height), height - 1))
        x_max = max(0, min(int(p.x_max * width), width - 1))
        y_max = max(0, min(int(p.y_max * height), height - 1))

        # Pixelate the boxes to preserve privacy
        pixelate_size = 24
        for y in range(y_min, y_max, pixelate_size):
            for x in range(x_min, x_max, pixelate_size):
                y_ref = min(y + pixelate_size // 2, height - 1)
                x_ref = min(x + pixelate_size // 2, width - 1)
                rgb_value = camera_data[y_ref, x_ref, :]
                y2 = min(y + pixelate_size, y_max)
                x2 = min(x + pixelate_size, x_max)
                camera_data[y:y2, x:x2, :] = rgb_value

    # Display the results on screen
    cv.imshow("Plumerai demo", camera_data)
    cv.waitKey(30)  # in ms; can be changed to force a certain framerate
    if cv.getWindowProperty("Plumerai demo", cv.WND_PROP_VISIBLE) <= 0:
        break  # Exit when the X button is clicked

Perimeter detection

The Plumerai Video Intelligence library can also be easily used to detect if people enter a certain perimeter / area-of-interest, e.g. a front lawn or a forbidden security area.

A restricted garden as perimeter A restricted staircase as perimeter

Example of the perimeter detection application:
a restricted garden (left) and a restricted staircase (right).

Below is a simple perimeter detection example that sets a perimeter to the lower left quadrant of the screen, and then triggers an alarm by printing a message to screen if a person entered the area. In the example we assume that 'entering an area' is defined as the central lower point of a bounding-box, i.e. where a person's feet are if they are standing up-right. However, this logic can be adjusted of course to match your application.

The highlighted lines indicate the use of the Plumerai Video Intelligence API.

perimeter_detection.cc
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
#include <iostream>

#include "init_camera.h"
#include "plumerai/video_intelligence.h"

using namespace plumerai;

int main() {
  constexpr int height = 720;  // example camera image height in pixels
  constexpr int width = 1280;  // example camera image width in pixels
  cv::VideoCapture camera = init_camera(height, width);

  auto pvi = plumerai::VideoIntelligence(height, width);

  // Set an example perimeter / area of interest in relative coordinates (1)
  auto perimeter_x_min = 0.0f;  // left half
  auto perimeter_x_max = 0.5f;  // (2)
  auto perimeter_y_min = 0.5f;  // bottom half
  auto perimeter_y_max = 1.0f;
  std::cout << "Starting the perimeter detection alarm, perimeter: ";
  std::cout << "x: " << perimeter_x_min << " - " << perimeter_x_max << ", ";
  std::cout << "y: " << perimeter_y_min << " - " << perimeter_y_max;
  std::cout << std::endl;

  cv::Mat camera_image;
  while (true) {
    const auto camera_read_success = camera.read(camera_image);
    if (!camera_read_success) {
      throw std::runtime_error("Could not read frame from camera");
    }

    auto error_code = pvi.process_frame(
        ImagePointer<ImageFormat::PACKED_RGB888>(camera_image.data));
    if (error_code != ErrorCode::SUCCESS) {
      printf("Error code: %s\n", error_code_string(error_code));
      return 1;
    }

    std::vector<BoxPrediction> predictions;
    error_code = pvi.object_detection().get_detections(predictions);
    if (error_code != ErrorCode::SUCCESS) {
      printf("Error code: %s\n", error_code_string(error_code));
      return 1;
    }

    for (auto &p : predictions) {
      if (p.class_id != CLASS_PERSON) continue;

      // We take the person coordinates of roughly where the feet are (3)
      auto person_x = p.x_min + (p.x_max - p.x_min) / 2;  // (4)
      auto person_y = p.y_max;                            // (5)
      if ((perimeter_x_min < person_x && person_x < perimeter_x_max) and
          (perimeter_y_min < person_y && person_y < perimeter_y_max)) {
        std::cout << "Alarm! Person detected in perimeter!" << std::endl;
      }
    }
  }
}

  1. The coordinates are between 0 and 1 such that they are independent of the configured width and height, and can be easily matched with the boxes returned by the Plumerai Video Intelligence API, which are also between 0 and 1.

  2. Note that when using a webcam the view is mirrored so the left half on the screen means you actually need to go to the right to move inside the example perimeter.

  3. This is a choice and should be changed depending on the exact use-case.

  4. The middle point from the x/width dimension of the person box.

  5. The bottom point from the y/height dimension of the person box.

perimeter_detection.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
from init_camera import init_camera
import plumerai_video_intelligence as pvi_api

height = 720  # example camera image height in pixels
width = 1280  # example camera image width in pixels
camera = init_camera(height, width)

pvi = pvi_api.VideoIntelligence(height, width)

# Set an example perimeter / area of interest in relative coordinates (1)
perimeter_x_min, perimeter_x_max = 0.0, 0.5  # left half (2)
perimeter_y_min, perimeter_y_max = 0.5, 1.0  # bottom half
print("Starting the perimeter detection alarm, perimeter at:")
print(f"  x: {perimeter_x_min} - {perimeter_x_max}")
print(f"  y: {perimeter_y_min} - {perimeter_y_max}")

while True:
    camera_read_success, camera_image = camera.read()
    if not camera_read_success:
        raise RuntimeError("Could not read frame from camera")

    pvi.process_frame(camera_image.data)
    _, predictions = pvi.object_detection.get_detections()

    for p in predictions:
        if p.class_id != pvi_api.DetectionClass.CLASS_PERSON:
            continue

        # We take the person coordinates of roughly where the feet are (3)
        person_x = p.x_min + (p.x_max - p.x_min) / 2  # (4)
        person_y = p.y_max  # (5)
        if (perimeter_x_min < person_x < perimeter_x_max) and (
            perimeter_y_min < person_y < perimeter_y_max
        ):
            print("Alarm! Person detected in perimeter!")

  1. The coordinates are between 0 and 1 such that they are independent of the configured width and height, and can be easily matched with the boxes returned by the Plumerai Video Intelligence API, which are also between 0 and 1.

  2. Note that when using a webcam the view is mirrored so the left half on the screen means you actually need to go to the right to move inside the example perimeter.

  3. This is a choice and should be changed depending on the exact use-case.

  4. The middle point from the x/width dimension of the person box.

  5. The bottom point from the y/height dimension of the person box.