Plumerai Familiar Face Identification C++ API¶

This document describes the C++ API for the Plumerai Familiar Face Identification software for videos on Arm Cortex-A and x86.

The API¶

The C++ API consists of a class plumerai::FaceIdentification which is a subclass of plumerai::PeopleDetection. Processing video input is described in the documentation of the base class. The FaceIdentification class adds functionality for controlling the face library, which is documented here.

The API is re-entrant, i.e. you can instantiate several FaceIdentification objects in different threads and use them independently. However, using the same instance from different threads at the same time is not supported.

FaceIdentification::FaceIdentification¶

FaceIdentification::FaceIdentification(int height, int width)

Initializes a new FaceIdentification object.

This needs to be called only once at the start of the application.

Arguments

height int: The height of the input image in pixels.
width int: The width of the input image in pixels.

Returns

Nothing.

FaceIdentification::face_embedding_size¶

static std::size_t face_embedding_size();

Returns the size of the face embedding vectors returned by finish_face_enrollment and required by add_face_embedding.

Returns

std::size_t: The size of the face embedding vectors.

FaceIdentification::face_embedding_version_checksum¶

static std::uint32_t face_embedding_version_checksum();

Returns a version checksum for the face embedding vectors returned by finish_face_enrollment and required by add_face_embedding.

This checksum is used to check if the face embedding vectors generated with one library build are compatible with another library.

Returns

std::uint32_t: The version checksum of the face embedding vectors.

FaceIdentification::start_face_enrollment¶

EnrollmentStatus start_face_enrollment(
    const std::vector<std::int8_t> &previous_enrollment =
        std::vector<std::int8_t>(),
    const Region &region_of_interest = Region());

This starts the enrollment procedure for a new face.

During subsequent calls to process_frame or single_image, face embeddings will be computed as well as metrics that quantify the quality of the embeddings. After every frame get_face_enrollment_status can be called to check the current status and see if enough frames have been processed for a high quality embedding. During enrollment there should be a single face in the image, or in the optional Region specified by region_of_interest, ideally clearly and completely visible. The enrollment procedure can be finalized by calling finish_face_enrollment.

Arguments

previous_enrollment std::vector<std::int8_t>: This can be set to the result of a previous enrollment procedure to supplement it with new data. It can be left empty if no previous enrollment data is available. The default is empty.
region_of_interest Region: This can be set to a region of interest in the image. Only faces that have overlap with this region will be used for enrollment. The default region is the entire image.

Returns

EnrollmentStatus: Returns ENROLLMENT_IN_PROGRESS in the error_code field if the enrollment procedure was started successfully, otherwise returns ERROR_ALREADY_ENROLLING or ERROR_INVALID_PREVIOUS_ENROLLMENT.

FaceIdentification::get_face_enrollment_status¶

EnrollmentStatus get_face_enrollment_status();

Get the status of the current face enrollment.

This status will update after every call to process_frame. This can be used to inform the user of the progress (the score value in EnrollmentStatus) and about what to do (the error_code in EnrollmentStatus), such as making sure only a single face is visible.

Returns

EnrollmentStatus: The status of the current face enrollment.

FaceIdentification::finish_face_enrollment¶

std::vector<std::int8_t> finish_face_enrollment();

Finalize the face enrollment procedure started by start_face_enrollment and obtain the face embedding.

The application is responsible for storing this embedding on persistent storage and passing it to add_face_embedding every time an instance of this class is created. The face enrollment procedure does not automatically call add_face_embedding.

Returns

std::vector<std::int8_t>: The resulting face embedding vector, or an empty vector in case of failure.

FaceIdentification::add_face_embedding¶

bool add_face_embedding(const std::vector<std::int8_t> &face_embedding,
                        const int face_id);

Add a face embedding to the face library for any process_frame calls that follow.

A second embedding for the same face id will overwrite the previous one. When process_frame detects a person that matches a face, it will use the face_id provided here in the returned BoxPrediction struct.

Arguments

face_embedding const std::vector<std::int8_t> &: A face embedding as returned by finish_face_enrollment.
face_id int: An integer face-id, is not allowed to be equal to FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY.

Returns

bool: Returns true on success.

FaceIdentification::remove_face_embedding¶

bool remove_face_embedding(const int face_id);

Remove a face embedding to the face library for any process_frame calls that follow.

Arguments

face_id int: An integer face-id, previously used in add_face_embedding

Returns

bool: Returns true on success.

Region¶

struct Region {
  float y_min = 0.0f;  // top coordinate between 0 and 1 in height dimension
  float x_min = 0.0f;  // left coordinate between 0 and 1 in width dimension
  float y_max = 1.0f;  // bottom coordinate between 0 and 1 in height dimension
  float x_max = 1.0f;  // right coordinate between 0 and 1 in width dimension
};

A structure representing a region of the input image. Coordinates are between 0 and 1, the origin is at the top-left.

EnrollmentErrorCode¶

typedef enum {
    // No error, enrollment is in progress, last frame was successfully used.
    ENROLLMENT_IN_PROGRESS = 0,

    // This library is not built with support for face identification.
    ERROR_FACE_IDENTIFICATION_DISABLED = 1,

    // Called `plumerai_start_face_enrollment` but enrollment is already started.
    ERROR_ALREADY_ENROLLING = 2,

    // Called `plumerai_start_face_enrollment` but the last enrollment is invalid.
    ERROR_INVALID_PREVIOUS_ENROLLMENT = 3,

    // Called `plumerai_get_face_enrollment_status` but enrollment is not started.
    ERROR_NOT_ENROLLING = 4,

    // Last frame was ignored because multiple faces were detected within the
    // specified region of interest. Try again with one face visible.
    ERROR_MULTIPLE_FACES_IN_VIEW = 5,

    // Last frame was ignored because no faces were detected within the specified
    // region of interest. Try again with one face visible.
    ERROR_NO_FACE_IN_VIEW = 6,

    // Last frame was ignored because the face detection was too close to the
    // edge. Too close to the edge means that the border of the face bounding-box
    // is within 2% of the edge of the image. In theory, it is OK if a face is
    // close to edge, however, it can also mean that a face is only partially
    // visible. Thus, this check is added to prevent partial faces from being
    // enrolled.
    ERROR_FACE_CLOSE_TO_EDGE = 7,
} EnrollmentErrorCode;

Possible error codes for the enrollment, as used in EnrollmentStatus.

EnrollmentStatus¶

struct EnrollmentStatus {
  EnrollmentErrorCode error_code;
  float score;  // Value between 0 and 1, higher means better quality
};

The return value of start_face_enrollment and get_face_enrollment_status., combining a score and a status code.

It is important to realize that:

The error_code value refers to the last frame that was processed only.
The score is the total score until now, not only of the last frame.

Regarding the score:

The value lies between 0.0 and 1.0.
The higher, the better.
The score is some way to measure diversity of enrolled images. And thus, the first successfully enrolled frame will always have a score 0.0.
If the next frame also results in a score of 0.0, then it was either identical (from the neural network perspective) or there was an error.
The score can also go down as new images are added.
It is possible to end enrollment with a score of 0 and the identifications may still be correct. We advise to perform the enrollment procedure again though.

Constants¶

static const int FACE_ID_UNIDENTIFIED = -1;
static const int FACE_ID_NOT_IN_LIBRARY = -2;

If the face-identification model is not confident about a face, it will output FACE_ID_UNIDENTIFIED. If the model is confident that a face is not in the face library, it will output FACE_ID_NOT_IN_LIBRARY.

Example usage¶

Below is an example of using the C++ API shown above.

#include <cstdint>
#include <vector>

#include "plumerai/face_identification.h"

int main() {
  // Settings, to be changed as needed
  constexpr int width = 1600;   // camera image width in pixels
  constexpr int height = 1200;  // camera image height in pixels
  constexpr auto image_format = plumerai::ImageFormat::PACKED_RGB888;

  // Initialize the `FaceIdentification` object
  auto ffid = plumerai::FaceIdentification(height, width);

  // Start the enrollment process
  ffid.start_face_enrollment();

  // Enroll for 10 frames (just an example, more frames is better)
  for (int t = 0; t < 10; ++t) {
    // Some example input here, normally this is where camera data is acquired
    auto image = std::vector<std::uint8_t>(height * width * 3);  // 3 for RGB

    // Process the frame
    ffid.process_frame<image_format>(image.data());

    auto enrollment_status = ffid.get_face_enrollment_status();

    printf("Enrollment status: %d\n", enrollment_status.error_code);

    // Stop enrollment when the quality is good enough
    if (enrollment_status.score > 0.8) break;
  }

  // Finish enrollment
  auto embedding = ffid.finish_face_enrollment();

  // Add the embedding to the library with face id 1.
  ffid.add_face_embedding(embedding, 1);

  // Loop over frames in a video stream
  while (true) {
    // Some example input here, normally this is where camera data is acquired
    auto image = std::vector<std::uint8_t>(height * width * 3);  // 3 for RGB

    // Process the frame
    auto predictions = ffid.process_frame<image_format>(image.data());

    // Display the results to stdout
    for (auto &p : predictions) {
      if (p.confidence < 0.8f) { continue; }  // only high-confidence boxes
      if (p.class_id != plumerai::CLASS_ID_PERSON) { continue; }
      printf(
          "Box #%d and face-ID %d with confidence %.2f @ (x,y) -> "
          "(%.2f,%.2f) till (%.2f,%.2f)\n",
          p.id, p.face_id, p.confidence, p.x_min, p.y_min, p.x_max,
          p.y_max);
    }
  }
  return 0;
}