Plumerai Multi-Camera Re-Identification API¶

This page documents the API for the Multi-Camera Re-Identification (ReID) component.

The API documentation can be found at the end of this page.

Overview¶

This section describes the main components that need to be implemented for Re-Identification.

State¶

Every camera, or set of cameras, keeps track of a ReID state:

A binary blob.
Approximately 5 KiB per unique identity.

When a camera boots up (or a video clip starts processing in the cloud), the most recent ReID state (if available) can be loaded from on-device storage or the cloud, and then loaded into the VideoIntelligence instance. When a camera shuts down (or a video clip finishes processing in the cloud), the ReID state must be stored again.

In multi-camera scenarios, the ReID states can be periodically shared between cameras, or they can be shared with a central server, which can combine the information from different cameras. This makes it possible to re-identify objects across cameras.

Matching¶

The Re-Identification results are obtained through the function get_matching_track_ids. This function can be called directly after processing each video frame (on the camera or in the cloud). Note that the identification process can take a few frames, so the output of this function can change with every video frame. For multi-camera scenarios, this function can also be called on a server that has all the shared information from all cameras but does not process videos itself.

The function takes as input a track ID, and outputs a list of track IDs that are identified to be the same identity.

Track IDs are 64-bit integers and in multi-camera scenarios they are automatically unique across cameras (the probability of a collision is negligible).

When storing object detection results in a database, it is recommended to store their track IDs. The list of matching track IDs can then be queried to see which entries should be merged.

For example, for a smart-home camera, the following logic could be used when a new person detection is encountered:

If there was no other recent person detection, then send out a push notification, and store the track ID as metadata.
If there were one or more recent person detections, get their track IDs from the metadata. For the next 5 seconds:
- On every frame, call get_matching_track_ids to check for a match between the new track ID and one of the previous ones. If there is a match, update the existing notification but do not send out a new one.
- After 5 seconds, if no match was found yet, send out a new notification for a new person detection.

Example pseudo code¶

The code below is pseudo-code, where error checks have been emitted for brevity.

Example code with error checks can be found on the minimal examples page.

Single camera ReID¶

re_id_single_camera.cc
const auto re_id_state = read_from_flash_or_database("re_id_cam1.bin");

auto pvi = plumerai::VideoIntelligence(height, width);

// Time is needed to know when tracks are too old
pvi.set_clock_time(time(0));

// Unique identifier is needed to distinguish states from other cameras
pvi.set_camera_name("user_1_camera_A", "Front yard");

// Restore the ReIdentification state
pvi.re_identification().merge_re_id_state(re_id_state);

// Process video frames
while(true) {
  pvi.process_frame(...);

  std::vector<BoxPrediction> predictions;
  pvi.object_detection().get_detections(predictions);
  for (const auto &p : predictions) {
    if (p.class_id != CLASS_PERSON) continue;

    // Get matching track IDs for this person
    std::vector<std::int64_t> matching_track_ids;
    pvi.re_identification().get_matching_track_ids(
        p.track_id, matching_track_ids);

    printf("Prediction for track ID %ld has %zu matching track IDs.\n",
        p.track_id, matching_track_ids.size());
    // Potentially update notifications based on matches.
  }
}

// Store ReID state back to flash or database
std::vector<std::uint8_t> new_re_id_state;
pvi.re_identification().get_re_id_state(new_re_id_state);
write_to_flash_or_database("re_id_cam1.bin", new_re_id_state);

Multi-Camera ReID¶

Instead of checking for matching track IDs, cameras periodically send their ReID state to a central location (for example the cloud or a local hub).

Optionally, every camera can keep track of a local ReID state (see 'Extended' below). This can be useful for on-device processing to get quicker results while a clip is still on-going.

SimpleExtended

re_id_multi_camera_simple.cc
auto pvi = plumerai::VideoIntelligence(height, width);

// Time is needed to know when tracks are too old
pvi.set_clock_time(time(0));

// Unique identifier is needed to distinguish states from other cameras
pvi.set_camera_name("user_1_camera_A", "Front yard");

while(true) {
  pvi.process_frame(...);

  // Send state to central node every 5 seconds.
  time_since_server_sync += delta_t;
  if (time_since_server_sync > 5) {
    time_since_server_sync = 0;

    std::vector<std::uint8_t> new_re_id_state;
    pvi.re_identification().get_re_id_state(new_re_id_state);
    send_to_server(new_re_id_state);
  }
}

re_id_multi_camera_extended.cc
const auto re_id_state = read_from_flash_or_database("re_id_cam1.bin");

auto pvi = plumerai::VideoIntelligence(height, width);

// Time is needed to know when tracks are too old
pvi.set_clock_time(time(0));

// Unique identifier is needed to distinguish states from other cameras
pvi.set_camera_name("user_1_camera_A", "Front yard");

// Restore the ReIdentification state
pvi.re_identification().merge_re_id_state(re_id_state);

while(true) {
  pvi.process_frame(...);

  // Send state to central node every 5 seconds
  time_since_server_sync += delta_t;
  if (time_since_server_sync > 5) {
    time_since_server_sync = 0;

    std::vector<std::uint8_t> new_re_id_state;
    pvi.re_identification().get_re_id_state(new_re_id_state);
    send_to_server(new_re_id_state);
  }

  // Also check for ReID matches locally
  std::vector<BoxPrediction> predictions;
  pvi.object_detection().get_detections(predictions);
  for (const auto &p : predictions) {
      if (p.class_id != CLASS_PERSON) continue;

      // Get matching track IDs for this person
      const std::int64_t *matching_track_ids = nullptr;
      std::size_t num_matching_track_ids = -1;
      pvi.re_identification().get_matching_track_ids(
          p.track_id, &matching_track_ids, &num_matching_track_ids);

      printf("Prediction for track ID %ld has %zu matching track IDs.\n",
          p.track_id, num_matching_track_ids);
      // Potentially update notifications based on matches.
  }

}

// Store ReID state back to flash or database
std::vector<std::uint8_t> new_re_id_state;
pvi.re_identification().get_re_id_state(new_re_id_state);
write_to_flash_or_database("re_id_cam1.bin", new_re_id_state);

Whenever the cloud receives a ReID state, it can combine that information with information from other cameras.

re_id_central.cc
void on_re_id_state_event(const std::vector<uint8_t>& new_re_id_state) {
  auto pvi = plumerai::VideoIntelligence(dummy_height, dummy_width);
  pvi.set_clock_time(time(0));

  // Load the accumulated state and the new state
  const auto accumulated_state = read_from_database("re_id.bin");
  pvi.re_identification().merge_re_id_state(accumulated_state);
  pvi.re_identification().merge_re_id_state(new_re_id_state);

  // Get matching track IDs for a person
  const std::int64_t person_track_id = ...;
  std::vector<std::int64_t> matching_track_ids;
  pvi.re_identification().get_matching_track_ids(
      person_track_id, matching_track_ids);

  // Potentially update notifications based on matches.

  // Store the new accumulated state
  std::vector<std::uint8_t> new_accumulated_state;
  pvi.re_identification().get_re_id_state(new_accumulated_state);
  store_to_database("re_id.bin", new_accumulated_state);
}

ReIdentification¶

get_matching_track_ids¶

ErrorCode get_matching_track_ids(std::int64_t track_id,
                                 const std::int64_t** matching_track_ids,
                                 std::size_t* num_matching_track_ids) const;
ErrorCode get_matching_track_ids(
    std::int64_t track_id,
    std::vector<std::int64_t>& matching_track_ids) const;

Retrieve the matching track IDs of a given track ID.

If the track ID was not found or the class of the track does not have support for ReIdentification, INVALID_TRACK_ID is returned.

When the track is not (yet) matched to other tracks, a list with one track ID is returned (its own ID), with an error code SUCCESS.

When the track ID was matched to other track IDs, all known matching track IDs are returned in the list, with an error code SUCCESS.

This function only returns matching track IDs that are known locally to this instance of the ReIdentification algorithm.

In night mode this function will only return the track ID itself because there is not enough appearance information in IR video data to uniquely identify people.

The data pointed to by matching_track_ids is valid only until the next call to process_frame, get_matching_track_ids, or merge_re_id_state, whichever comes first. Copy the returned list before calling get_matching_track_ids again if querying multiple track IDs in succession.

On some platforms, a version that uses std::vector is available, where the result is not invalidated by other calls to get_matching_track_ids.

Arguments:

track_id: A track ID.
matching_track_ids: An output parameter that receives a pointer to a list of track IDs that are known to match the given track ID.
num_matching_track_ids: An output parameter that receives the number of returned IDs.

Returns:

Returns SUCCESS when there was no error, or INVALID_TRACK_ID if the track ID was not found.

Example:

// Loop over all predictions made this frame,
// obtained from `object_detection().get_detections()`.

for (const auto &p : predictions) {
  if (p.class_id != CLASS_PERSON) continue;

  const std::int64_t *matching_track_ids = nullptr;
  std::size_t num_matching_track_ids = -1;
  error_code = pvi.re_identification().get_matching_track_ids(
      p.track_id, &matching_track_ids, &num_matching_track_ids);
  if (error_code != plumerai::ErrorCode::SUCCESS) {
    printf("Error: %s\n", plumerai::error_code_string(error_code));
  }
  printf("Prediction for track ID %ld has %zu matching track IDs: ",
         p.track_id, num_matching_track_ids);
  for (std::size_t i = 0; i < num_matching_track_ids; ++i) {
    printf("%ld ", matching_track_ids[i]);
  }
  printf("\n");
}

get_re_id_state¶

ErrorCode get_re_id_state(std::vector<std::uint8_t>& state) const;

Get the current state of the ReIdentification algorithm.

This can be used to store the state to persistent storage when a device shuts down and restore it later when it reboots.

This state data can change during any VideoIntelligence::process_frame call.

This function can only be called when the clock time has been set through VideoIntelligence::set_clock_time, and a unique camera name has been set through VideoIntelligence::set_camera_name.

Arguments:

state: A output argument that receives the data.

Returns:

Returns SUCCESS on success, TIME_NOT_SET when the clock time has not been set, or CAMERA_NAME_NOT_SET when the unique camera name has not been set.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);
// After processing some frames...
std::vector<std::uint8_t> re_id_state;
auto error_code = pvi.re_identification().get_re_id_state(re_id_state);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: get_re_id_state returned %s\n",
         plumerai::error_code_string(error_code));
}

merge_re_id_state¶

ErrorCode merge_re_id_state(const std::vector<std::uint8_t>& state);

Merge ReIdentification data into the current VideoIntelligence instance.

This can be used to restore a previous state from persistent storage, for example when a device reboots. When called with data obtained from a different instance of VideoIntelligence, for example from a different camera, this will merge the ReIdentification data from that instance into the current instance.

This function can only be called when the clock time has been set through VideoIntelligence::set_clock_time.

Arguments:

state: The data to merge.

Returns:

Returns SUCCESS on success, TIME_NOT_SET when the clock time has not been set, STATE_CORRUPT when the state was corrupted, or STATE_SETTINGS_MISMATCH when the state was created with an incompatible version of the software.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);
// The state as obtained by `get_re_id_state`, e.g. loaded from memory
std::vector<std::uint8_t> state = ...;
auto error_code = pvi.re_identification().merge_re_id_state(state);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: merge_re_id_state returned %s\n",
         plumerai::error_code_string(error_code));
}

is_available¶

static bool is_available();

Check if the ReIdentification algorithm functionality is available.

This function can be used to check if the library was built with support for ReIdentification. Note that ReIdentification is an optional product feature and requires a separate license. The license determines which object classes are supported for ReIdentification.

Returns:

Returns true if the functionality of ReIdentification is available.