Plumerai Familiar Face Identification C++ API¶

This document describes the C++ API for the Plumerai Familiar Face Identification software for videos on Arm Cortex-A and x86. For an example, see here.

The C++ API consists of a class plumerai::FaceIdentification which is a subclass of plumerai::PeopleDetection. Processing video input is described in the documentation of the base class. The FaceIdentification class adds functionality for controlling the face library, which is documented here.

The API is re-entrant, i.e. you can instantiate several FaceIdentification objects in different threads and use them independently. However, using the same instance from different threads at the same time is not supported.

API¶

FaceIdentification¶

FaceIdentification::FaceIdentification(int height, int width);

Initializes a new FaceIdentification object.

This needs to be called only once at the start of the application.

Arguments

height int: The height of the input image in pixels.
width int: The width of the input image in pixels.

Returns

Nothing.

face_embedding_size¶

static std::size_t FaceIdentification::face_embedding_size();

Returns the size of the face embedding vectors returned by finish_face_enrollment and required by add_face_embedding.

Returns

std::size_t: The size of the face embedding vectors.

face_embedding_version_checksum¶

static std::uint32_t FaceIdentification::face_embedding_version_checksum();

Returns a version checksum for the face embedding vectors returned by finish_face_enrollment and required by add_face_embedding.

This checksum is used to check if the face embedding vectors generated with one library build are compatible with another library.

Returns

std::uint32_t: The version checksum of the face embedding vectors.

start_face_enrollment¶

ErrorCodeFamiliarFaceID FaceIdentification::start_face_enrollment(
    const std::vector<std::int8_t> &previous_enrollment =
        std::vector<std::int8_t>(),
    const Region &region_of_interest = Region());

This starts the enrollment procedure for a new face.

During subsequent calls to process_frame or single_image, face embeddings will be computed as well as metrics that quantify the quality of the embeddings. After every frame get_cumulative_enrollment_score can be called to check if enough frames have been processed for a high quality embedding. During enrollment there should be a single face in the image, or in the optional Region specified by region_of_interest, ideally clearly and completely visible. The enrollment procedure can be finalized by calling finish_face_enrollment.

Arguments

previous_enrollment std::vector<std::int8_t>: This can be set to the result of a previous enrollment procedure to supplement it with new data. It can be left empty if no previous enrollment data is available. The default is empty.
region_of_interest Region: This can be set to a region of interest in the image. Only faces that have overlap with this region will be used for enrollment. The default region is the entire image.

Returns

ErrorCodeFamiliarFaceID: Returns ENROLLMENT_IN_PROGRESS if the enrollment procedure was started successfully, otherwise returns ALREADY_ENROLLING or INVALID_PREVIOUS_ENROLLMENT.

Example

auto ffid = plumerai::FaceIdentification(height, width);

auto error_code = ffid.start_face_enrollment();
if (error_code != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
  printf("ERROR: start_face_enrollment returned %d\n", error_code);
}

// Process the face images
std::vector<BoxPrediction> predictions(0);
for (size_t i = 0; i < num_images; ++i) {
  error_code = ffid.process_frame<image_format>(image_data[i], predictions);
  if (error_code != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
    // The face image might be rejected if the face is not clearly visible
    printf("Warning, face image %d rejected: %d\n", i, error_code);
  }
}

auto embedding = ffid.finish_face_enrollment();

get_cumulative_enrollment_score¶

float FaceIdentification::get_cumulative_enrollment_score();

Return the cumulative enrollment score.

It is important to realize that the score is the total score until now, not only of the last frame. Properties of the score:

The value lies between 0.0 and 1.0.
The higher, the better.
The score is some way to measure diversity of enrolled images. And thus, the first successfully enrolled frame will always have a score 0.0.
If the next frame also results in a score of 0.0, then it was either identical (from the neural network perspective) or there was an error.
The score can also go down as new images are added.
It is possible to end enrollment with a score of 0, however it is not recommended. Still, familiar face identification might still work.

Returns

float: The cumulative enrollment score.

finish_face_enrollment¶

std::vector<std::int8_t> FaceIdentification::finish_face_enrollment();

Finalize the face enrollment procedure started by start_face_enrollment and obtain the face embedding.

The application is responsible for storing this embedding on persistent storage and passing it to add_face_embedding every time an instance of this class is created. The face enrollment procedure does not automatically call add_face_embedding.

Returns

std::vector<std::int8_t>: The resulting face embedding vector, or an empty vector in case of failure.

Example

auto ffid = plumerai::FaceIdentification(height, width);

auto error_code = ffid.start_face_enrollment();
if (error_code != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
  printf("ERROR: start_face_enrollment returned %d\n", error_code);
}

// Process the face images
std::vector<BoxPrediction> predictions(0);
for (size_t i = 0; i < num_images; ++i) {
  error_code = ffid.process_frame<image_format>(image_data[i], predictions);
  if (error_code != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
    // The face image might be rejected if the face is not clearly visible
    printf("Warning, face image %d rejected: %d\n", i, error_code);
  }
}

auto embedding = ffid.finish_face_enrollment();

add_face_embedding¶

bool FaceIdentification::add_face_embedding(
    const std::vector<std::int8_t> &face_embedding,
    const int face_id);

Add a face embedding to the face library for any process_frame calls that follow.

A second embedding for the same face id will overwrite the previous one. When process_frame detects a person that matches a face, it will use the face_id provided here in the returned BoxPrediction struct.

Arguments

face_embedding const std::vector<std::int8_t> &: A face embedding as returned by finish_face_enrollment.
face_id int: An integer face-id, is not allowed to be equal to FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY.

Returns

bool: Returns true on success.

Example

auto ffid = plumerai::FaceIdentification(height, width);

std::vector<std::int8_t> face_embedding = ...;

// Add with face-id 3
auto result = ffid.add_face_embedding(face_embedding, 3);
if (!result) {
  printf("ERROR: Couldn't add face embedding\n");
}

remove_face_embedding¶

bool FaceIdentification::remove_face_embedding(const int face_id);

Remove a face embedding to the face library for any process_frame calls that follow.

Arguments

face_id int: An integer face-id, previously used in add_face_embedding

Returns

bool: Returns true on success.

get_face_id¶

int FaceIdentification::get_face_id(const BoxPrediction &person_box) const;

Retrieve the face ID that belongs to a person box.

Retrieves the face ID given a person box returned by process_frame. This function has three possible return values:

FACE_ID_UNIDENTIFIED when the provided box is not of CLASS_PERSON or when a face is not yet identified, e.g. the face is not fully visible.
FACE_ID_NOT_IN_LIBRARY when the face is visible but not found in the library, e.g. when a stranger is in view.
Any integer face-ID provided previously to add_face_embedding when a familiar face is identified.

Arguments

person_box BoxPrediction: A box returned by process_frame with class_id equal to DetectionClass::CLASS_PERSON.

Returns

int: An integer face-ID as provided when calling add_face_embedding when a face is identified, FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY when it is not identified successfully, or FACE_ID_UNIDENTIFIED when the provided box is not a valid person box.

get_person_box_from_face_box¶

See get_face_box_from_person_box below.

get_face_box_from_person_box¶

bool FaceIdentification::get_person_box_from_face_box(
    const BoxPrediction &face_box,
    BoxPrediction *result_person_box) const;
bool FaceIdentification::get_face_box_from_person_box(
    const BoxPrediction &person_box,
    BoxPrediction *result_face_box) const;

Retrieve a person box corresponding to a face box or vice-versa.

Arguments

{face,person}_box const BoxPrediction: The target face or person to find a match for. This has to be a box from the most recent call to process_frame.
result_{person,face}_box BoxPrediction*: The resulting box, only valid if the return value is true (see below). The caller needs to allocate memory for this box prior to calling this function.

Returns

bool: False in case of invalid arguments or when it was not possible to uniquely match a face box to a person box. Returns true in case a match was found.

Region¶

struct Region {
  float y_min = 0.0f;  // top coordinate between 0 and 1 in height dimension
  float x_min = 0.0f;  // left coordinate between 0 and 1 in width dimension
  float y_max = 1.0f;  // bottom coordinate between 0 and 1 in height dimension
  float x_max = 1.0f;  // right coordinate between 0 and 1 in width dimension
};

A structure representing a region of the input image. Coordinates are between 0 and 1, the origin is at the top-left.

ErrorCodeFamiliarFaceID¶

typedef enum {
  // Everything OK, enrollment is in progress.
  ENROLLMENT_IN_PROGRESS = -1000,

  // This library is not built with support for face identification.
  FACE_IDENTIFICATION_DISABLED = -1001,

  // Called `plumerai_start_face_enrollment` but enrollment is already started.
  ALREADY_ENROLLING = -1002,

  // Called `plumerai_start_face_enrollment` but the last enrollment is invalid.
  INVALID_PREVIOUS_ENROLLMENT = -1003,

  // Last frame was ignored because multiple faces were detected within the
  // specified region of interest. Try again with one face visible.
  MULTIPLE_FACES_IN_VIEW = -1004,

  // Last frame was ignored because no faces were detected within the specified
  // region of interest. Try again with one face visible.
  NO_FACE_IN_VIEW = -1005,

  // Last frame was ignored because the face detection was too close to the
  // edge. Too close to the edge means that the border of the face bounding-box
  // is within 2% of the edge of the image. In theory, it is OK if a face is
  // close to edge, however, it can also mean that a face is only partially
  // visible. Thus, this check is added to prevent partial faces from being
  // enrolled.
  FACE_CLOSE_TO_EDGE = -1006,
} ErrorCodeFamiliarFaceID;

Possible error codes for the enrollment procedure. Furthermore, the regular ErrorCode values might be returned.

Constants¶

static const int FACE_ID_UNIDENTIFIED = -1;
static const int FACE_ID_NOT_IN_LIBRARY = -2;

If the face-identification model is not confident about a face, it will output FACE_ID_UNIDENTIFIED. If the model is confident that a face is not in the face library, it will output FACE_ID_NOT_IN_LIBRARY.