Skip to content

Plumerai Familiar Face Identification C++ API

This document describes the C++ API for the Plumerai Familiar Face Identification software with automatic enrollment for videos on Arm Cortex-A and x86.

The C++ API consists of a class plumerai::FaceIdentificationAutomatic which is a subclass of plumerai::PeopleDetection. Processing video input is described in the documentation of the base class. The FaceIdentificationAutomatic class adds functionality for controlling the face library, which is documented here. The API is similar to the standard Plumerai People Detection API, so please refer to the People Detection C++ API example for example code to get started.

The API is re-entrant, i.e. you can instantiate several FaceIdentificationAutomatic objects in different threads and use them independently. However, using the same instance from different threads at the same time is not supported.

API

FaceIdentificationAutomatic

FaceIdentificationAutomatic

FaceIdentificationAutomatic(int height, int width);

Initializes a new face identification object.

This needs to be called only once at the start of the application.

Arguments:

  • height: The height of the input image in pixels.
  • width: The width of the input image in pixels.

Returns:

  • Nothing.

configure_face_snapshots

ErrorCodeType configure_face_snapshots(const FaceSnapshotSettings &settings);

Configure the face snapshots for automatic enrollment.

By default, face snapshots are disabled.

Calling this function will invalidate any stored face snapshots of faces that are already enrolled. Trying to retrieve those face snapshots will return uninitialized data.

Arguments:

Returns:

  • Returns SUCCESS on success, or INVALID_ARGUMENT when the provided settings are not valid.

get_face_snapshot

ErrorCodeType get_face_snapshot(const int face_id,
                                const uint8_t **face_snapshot_data) const;

Return a snapshot of an automatically enrolled face.

Note that UNKNOWN_FACE_ID can be returned when a face id that was previously valid is overwritten with a new entry.

Arguments:

  • face_id: A non-negative integer face-id.
  • face_snapshot_data: The pointer to the stored snapshot data is returned via this output argument.

Returns:

  • Returns SUCCESS on success, UNKNOWN_FACE_ID when the specified face id could not be found, or FACE_SNAPSHOTS_DISABLED when face snapshots were not configured using configure_face_snapshots.

get_face_ids

std::vector<int> get_face_ids() const;

Return the face ids of all automatically enrolled faces.

Arguments:

  • None.

Returns:

  • A vector of integers, each representing a face id.

tag_face_id

ErrorCodeType tag_face_id(const int face_id, bool important = true);

Tag a face id as 'important' to prevent it from being overwritten.

When the face library is full and a new face is encountered, the algorithm will overwrite the oldest face in the library that is not tagged as important. A typical use-case of this is to tag a face as important when the user has named them in a front-end application.

When all entries in the face library are tagged as important and a new face is encountered, then no new faces will be enrolled.

By default, an automatically enrolled face is not tagged as important. However, library entries created by restore_face_embedding_data are automatically tagged as important.

Note that UNKNOWN_FACE_ID can be returned when a face id that was previously valid is overwritten with a new entry.

Arguments:

  • face_id: A non-negative integer face-id.
  • important: Whether to tag the face as important or to remove a previously applied tag.

Returns:

  • Returns SUCCESS on success, UNKNOWN_FACE_ID when the specified face id could not be found.

remove_face_embedding

ErrorCodeType remove_face_embedding(const int face_id);

Remove a face embedding from the face library for any process_frame calls that follow.

Note that UNKNOWN_FACE_ID can be returned when a face id that was previously valid is overwritten with a new entry.

Arguments:

  • face_id: A non-negative integer face-id.

Returns:

  • Returns SUCCESS on success, UNKNOWN_FACE_ID on failure.

face_embedding_version_checksum

static std::uint32_t face_embedding_version_checksum();

Returns a version checksum for the face embedding data obtained from get_face_embedding_data and required by load_face_embedding_data.

This checksum is used to check if the face embedding data generated with one library is are compatible with another library.

Arguments:

  • None.

Returns:

  • The version checksum of the face embedding data.

max_face_library_size

static std::size_t max_face_library_size();

Returns the maximum size of the face library.

When the library is full and new faces are detected, they will overwrite older faces, except for faces that are tagged as important using tag_face_id. When all faces are tagged as important, new faces will not be added to the library beyond this maximum size.

Arguments:

  • None.

Returns:

  • The maximum size of the face library.

get_face_embedding_data

ErrorCodeType get_face_embedding_data(
    const int face_id, std::vector<std::uint8_t> &embedding_data,
    bool include_face_snapshot) const;

Get the face embedding data for a face-ID, for persistent storage.

This can for example be used to store the face library when a device shuts down and restore it later when it reboots.

This embedding data can change during any process_frame call.

Note that UNKNOWN_FACE_ID can be returned when a face id that was previously valid is overwritten with a new entry.

Arguments:

  • face_id: A non-negative integer face-id.
  • embedding_data: A vector to store the embedding data in.
  • include_face_snapshot: Whether to include the face snapshot in the embedding data. This snapshot is stored as raw RGB data.

Returns:

  • Returns SUCCESS on success, UNKNOWN_FACE_ID when the specified face id could not be found, or INTERNAL_ERROR in case of an unexpected error.

restore_face_embedding_data

ErrorCodeType restore_face_embedding_data(
    const int face_id, const std::vector<std::uint8_t> &embedding_data);

Restore the face embedding data for a face-ID, from persistent storage.

When the specified face_id does not yet exist then it will be created. When the specified face_id already exists, the internal embedding data will be overwritten with new embedding data. If the new embedding data is corrupt, this will result in removal of the existing face embedding data.

If a new face_id was created, this function will automatically tag it as 'important', see also tag_face_id.

If the embedding data does not include a face snapshot, then the face snapshot data for this face id will contain uninitialized data. If the embedding data includes a face snapshot, then configure_face_snapshots must have been called first, with enabled set to true and the same height and width as the stored data. If the height and width do not match, then STATE_SETTINGS_MISMATCH is returned.

Arguments:

  • face_id: A non-negative integer face-id.
  • embedding_data: The embedding data to restore.

Returns:

  • Returns SUCCESS on success, or FACE_LIBRARY_FULL when the face library is full, or STATE_CORRUPT when the embedding data was corrupted. If the height and width of an included face snapshot do not match, then STATE_SETTINGS_MISMATCH is returned.

merge_face_embeddings

ErrorCodeType merge_face_embeddings(const int face_id_dst,
                                    const int face_id_src);

Merge two entries of the face library.

If a single person is enrolled twice in the face library, this function can be used to merge their enrollments back to a single one. After calling this function face_id_src is no longer valid and face_id_dst will be updated with the additional information from face_id_src.

Arguments:

  • face_id_dst: A non-negative integer face-id.
  • face_id_src: A non-negative integer face-id, must be different from face_id_dst.

Returns:

  • Returns SUCCESS on success, UNKNOWN_FACE_ID when either of the face ids could not be found, or INTERNAL_ERROR in case of an unexpected error.

get_face_id

int get_face_id(const BoxPrediction &person_box) const;

Retrieve the face-ID that belongs to a person box.

Retrieves the face-ID given a person box returned by process_frame. This function has three possible return values:

  • FACE_ID_UNIDENTIFIED when the provided box is not of CLASS_PERSON or when a face is not yet identified, e.g. the face is not fully visible.
  • FACE_ID_NOT_IN_LIBRARY when the face is visible but not found in the library, e.g. when a stranger is in view.
  • Any non-negative integer face-ID provided when a familiar face is identified.

This function should not be called directly after restoring from a previous state.

Arguments:

  • person_box: A box returned by process_frame with class_id equal to DetectionClass::CLASS_PERSON.

Returns:

  • A non-negative integer face-ID when a face is identified, FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY when it is not identified successfully, or FACE_ID_UNIDENTIFIED when the provided box is not a valid person box.

get_person_box_from_face_box

bool get_person_box_from_face_box(const BoxPrediction &face_box,
                                  BoxPrediction *result_person_box) const;

Retrieve a person box corresponding to a face box.

Arguments:

  • face_box: The target face to find a match for. This has to be a box from the most recent call to process_frame.
  • result_person_box: The resulting box, only valid if the return value is true (see below). The caller needs to allocate memory for this box prior to calling this function.

Returns:

  • False in case of invalid arguments or when it was not possible to uniquely match a face box to a person box. Returns true in case a match was found.

ErrorCodeFamiliarFaceIDAutomatic

typedef enum {
  // Returned by `remove_face_embedding` when the provided face-ID is invalid.
  UNKNOWN_FACE_ID = -2001,

  // Returned by `load_face_embedding_data` if the face library is full.
  FACE_LIBRARY_FULL = -2002,
} ErrorCodeFamiliarFaceIDAutomatic;

Possible error codes for the FFID API. Furthermore, the regular ErrorCode values might be returned.

FaceSnapshotSettings

struct FaceSnapshotSettings {
  bool enabled;
  int height;
  int width;
};

Constants

static const int FACE_ID_UNIDENTIFIED = -1;
static const int FACE_ID_NOT_IN_LIBRARY = -2;

If the face-identification model is not confident about a face, it will output FACE_ID_UNIDENTIFIED. If the model is confident that a face is not in the face library, it will output FACE_ID_NOT_IN_LIBRARY.

Example usage

Below is an example of using the C++ API shown above.

#include <cstdint>
#include <vector>

#include "plumerai/face_identification_automatic.h"

int main() {
  // Settings, to be changed as needed
  constexpr int width = 1600;   // camera image width in pixels
  constexpr int height = 1200;  // camera image height in pixels
  constexpr auto image_format = plumerai::ImageFormat::PACKED_RGB888;

  // Initialize the `FaceIdentificationAutomatic` object
  auto ffid = plumerai::FaceIdentificationAutomatic(height, width);

  // Loop over frames in a video stream (example: 20 frames)
  for (int t = 0; t < 20; ++t) {
    // Some example input here, normally this is where camera data is acquired
    auto image = std::vector<std::uint8_t>(height * width * 3);  // 3 for RGB

    // The time between two video frames in seconds
    // In this example we assume a constant frame rate of 30 fps, but variable
    // rates are supported.
    const float delta_t = 1.f / 30.f;

    // Process the frame
    std::vector<BoxPrediction> predictions(0);
    const auto error_code =
        ffid.process_frame<image_format>(image.data(), predictions, delta_t);
    if (error_code != plumerai::ErrorCode::SUCCESS) {
      printf("Error: %s\n", plumerai::error_code_string(error_code));
      return 1;
    }

    // Report the number of faces in the library so far. At first the library
    // will be empty, but as soon as a face is well visible for a while, it
    // will be added to the library with a new unique face-ID. The library
    // will grow over time, unless `remove_face_embedding` is called.
    const auto face_ids = ffid.get_face_ids();
    printf("Total of %zu people in the familiar face-ID library\n",
           face_ids.size());

    // Display the results to stdout
    for (auto &p : predictions) {
      if (p.confidence < 0.8f) { continue; }  // only high-confidence boxes
      if (p.class_id != CLASS_PERSON) { continue; } // only people
      const auto face_id = ffid.get_face_id(p);  // one from 'face_ids'
      printf(
          "Box #%d with face-ID %d with confidence %.2f "
          "(x,y)->(%.2f,%.2f) till (%.2f,%.2f)\n",
          p.id, face_id, p.confidence, p.x_min, p.y_min, p.x_max, p.y_max);
    }
    if (predictions.size() == 0) {
      printf("No bounding boxes found in this frame\n");
    }
  }
  return 0;
}