Plumerai Familiar Face Identification C++ API¶

This document describes the C++ API for the Plumerai Familiar Face Identification software for videos on Arm Cortex-A and x86. For an example, see here.

The C++ API consists of a class plumerai::FaceIdentification which is a subclass of plumerai::PeopleDetection. Processing video input is described in the documentation of the base class. The FaceIdentification class adds functionality for controlling the face library, which is documented here.

The API is re-entrant, i.e. you can instantiate several FaceIdentification objects in different threads and use them independently. However, using the same instance from different threads at the same time is not supported.

API¶

FaceIdentification¶

FaceIdentification::FaceIdentification(int height, int width);

Initializes a new FaceIdentification object.

This needs to be called only once at the start of the application.

Arguments

height int: The height of the input image in pixels.
width int: The width of the input image in pixels.

Returns

Nothing.

face_embedding_size¶

static std::size_t FaceIdentification::face_embedding_size();

Returns the size of the face embedding vectors returned by finish_face_enrollment and required by add_face_embedding.

Returns

std::size_t: The size of the face embedding vectors.

face_embedding_version_checksum¶

static std::uint32_t FaceIdentification::face_embedding_version_checksum();

Returns a version checksum for the face embedding vectors returned by finish_face_enrollment and required by add_face_embedding.

This checksum is used to check if the face embedding vectors generated with one library build are compatible with another library.

Returns

std::uint32_t: The version checksum of the face embedding vectors.

start_face_enrollment¶

ErrorCodeType FaceIdentification::start_face_enrollment(
    const Region &region_of_interest = Region());

This starts the enrollment procedure for a new face.

During subsequent calls to process_frame or single_image, face embeddings will be computed. During enrollment there should be a single face in the image, or in the optional Region specified by region_of_interest, ideally clearly and completely visible. The enrollment procedure can be finalized by calling finish_face_enrollment.

Arguments

region_of_interest Region: This can be set to a region of interest in the image. Only faces that have overlap with this region will be used for enrollment. The default region is the entire image.

Returns

ErrorCodeType: Returns ENROLLMENT_IN_PROGRESS if the enrollment procedure was started successfully, otherwise returns ALREADY_ENROLLING.

Example

auto ffid = plumerai::FaceIdentification(height, width);

auto error_code = ffid.start_face_enrollment();
if (error_code != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
  printf("ERROR: start_face_enrollment returned %d\n", error_code);
}

// Process the face images
std::vector<BoxPrediction> predictions(0);
for (size_t i = 0; i < num_images; ++i) {
  error_code = ffid.process_frame<image_format>(image_data[i], predictions);
  if (error_code != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
    // The face image might be rejected if the face is not clearly visible
    printf("Warning, face image %d rejected: %d\n", i, error_code);
  }
}

std::vector<int8_t> embedding;
error_code = ffid.finish_face_enrollment(embedding);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: finish_face_enrollment returned %d\n", error_code);
}

finish_face_enrollment¶

ErrorCodeType FaceIdentification::finish_face_enrollment(
    std::vector<std::int8_t> &embedding);

Finalize the face enrollment procedure started by start_face_enrollment and obtain the face embedding.

The application is responsible for storing this embedding on persistent storage and passing it to add_face_embedding every time an instance of this class is created. The face enrollment procedure does not automatically call add_face_embedding.

Arguments

face_embedding std::vector<std::int8_t> &: The resulting face embedding vector if the result is SUCCESS or LOW_QUALITY_ENROLLMENT, or an empty vector otherwise.

Returns

ErrorCodeType: Returns SUCCESS when enrollment was succesful. Otherwise returns one of NOT_ENROLLING, ENROLLMENT_FAILED, LOW_QUALITY_ENROLLMENT. See the enum for more information.

Example

auto ffid = plumerai::FaceIdentification(height, width);

auto error_code = ffid.start_face_enrollment();
if (error_code != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
  printf("ERROR: start_face_enrollment returned %d\n", error_code);
}

// Process the face images
std::vector<BoxPrediction> predictions(0);
for (size_t i = 0; i < num_images; ++i) {
  error_code = ffid.process_frame<image_format>(image_data[i], predictions);
  if (error_code != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
    // The face image might be rejected if the face is not clearly visible
    printf("Warning, face image %d rejected: %d\n", i, error_code);
  }
}

std::vector<int8_t> embedding;
error_code = ffid.finish_face_enrollment(embedding);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: finish_face_enrollment returned %d\n", error_code);
}

add_face_embedding¶

ErrorCodeType FaceIdentification::add_face_embedding(
    const std::vector<std::int8_t> &face_embedding, const int face_id);

Add a face embedding to the face library for any process_frame calls that follow.

A second embedding for the same face id will overwrite the previous one. When process_frame detects a person that matches a face, it will use the face_id provided here in the returned BoxPrediction struct.

Arguments

face_embedding const std::vector<std::int8_t> &: A face embedding as obtained from finish_face_enrollment.
face_id int: An integer face-id, is not allowed to be equal to FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY.

Returns

ErrorCodeType: Returns SUCCESS on success, INVALID_FACE_ID or INVALID_EMBEDDING on failure.

Example

auto ffid = plumerai::FaceIdentification(height, width);

std::vector<std::int8_t> face_embedding = ...;

// Add with face-id 3
auto error_code = ffid.add_face_embedding(face_embedding, 3);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: add_face_embedding returned %d\n", error_code);
}

remove_face_embedding¶

ErrorCodeType FaceIdentification::remove_face_embedding(const int face_id);

Remove a face embedding to the face library for any process_frame calls that follow.

Arguments

face_id int: An integer face-id, previously used in add_face_embedding

Returns

ErrorCodeType: Returns SUCCESS on success, INVALID_FACE_ID on failure.

get_face_id¶

int FaceIdentification::get_face_id(const BoxPrediction &person_box) const;

Retrieve the face ID that belongs to a person box.

Retrieves the face ID given a person box returned by process_frame. This function has three possible return values:

FACE_ID_UNIDENTIFIED when the provided box is not of CLASS_PERSON or when a face is not yet identified, e.g. the face is not fully visible.
FACE_ID_NOT_IN_LIBRARY when the face is visible but not found in the library, e.g. when a stranger is in view.
Any integer face-ID provided previously to add_face_embedding when a familiar face is identified.

Arguments

person_box BoxPrediction: A box returned by process_frame with class_id equal to DetectionClass::CLASS_PERSON.

Returns

int: An integer face-ID as provided when calling add_face_embedding when a face is identified, FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY when it is not identified successfully, or FACE_ID_UNIDENTIFIED when the provided box is not a valid person box.

get_person_box_from_face_box¶

See get_face_box_from_person_box below.

get_face_box_from_person_box¶

bool FaceIdentification::get_person_box_from_face_box(
    const BoxPrediction &face_box,
    BoxPrediction *result_person_box) const;
bool FaceIdentification::get_face_box_from_person_box(
    const BoxPrediction &person_box,
    BoxPrediction *result_face_box) const;

Retrieve a person box corresponding to a face box or vice-versa.

Arguments

{face,person}_box const BoxPrediction: The target face or person to find a match for. This has to be a box from the most recent call to process_frame.
result_{person,face}_box BoxPrediction*: The resulting box, only valid if the return value is true (see below). The caller needs to allocate memory for this box prior to calling this function.

Returns

bool: False in case of invalid arguments or when it was not possible to uniquely match a face box to a person box. Returns true in case a match was found.

Region¶

struct Region {
  float y_min = 0.0f;  // top coordinate between 0 and 1 in height dimension
  float x_min = 0.0f;  // left coordinate between 0 and 1 in width dimension
  float y_max = 1.0f;  // bottom coordinate between 0 and 1 in height dimension
  float x_max = 1.0f;  // right coordinate between 0 and 1 in width dimension
};

A structure representing a region of the input image. Coordinates are between 0 and 1, the origin is at the top-left.

ErrorCodeFamiliarFaceID¶

typedef enum {
  // Everything OK, enrollment is in progress.
  ENROLLMENT_IN_PROGRESS = -1000,

  // This library is not built with support for face identification.
  FACE_IDENTIFICATION_DISABLED = -1001,

  // Called `start_face_enrollment` but enrollment is already started.
  ALREADY_ENROLLING = -1002,

  // Last frame was ignored because multiple faces were detected within the
  // specified region of interest. Try again with one face visible.
  MULTIPLE_FACES_IN_VIEW = -1003,

  // Last frame was ignored because no face was detected, or it was not clearly
  // visible, or the lighting was not good enough for enrollment. Try again with
  // one face clearly visible.
  NO_FACE_IN_VIEW = -1004,

  // Last frame was ignored because the face detection was too close to the
  // edge. Too close to the edge means that the border of the face bounding-box
  // is within 2% of the edge of the image. In theory, it is OK if a face is
  // close to edge, however, it can also mean that a face is only partially
  // visible. Thus, this check is added to prevent partial faces from being
  // enrolled.
  FACE_CLOSE_TO_EDGE = -1005,

  // Called `finish_face_enrollment` but enrollment was not started.
  NOT_ENROLLING = -1006,

  // This is returned by `finish_face_enrollment` when the enrollment failed
  // because no faces could be detected in any of the images.
  ENROLLMENT_FAILED = -1007,

  // This is returned by `finish_face_enrollment` when a face embedding vector
  // is returned but the quality of the enrollment is very low. In this case
  // re-enrolllment is recommended.
  LOW_QUALITY_ENROLLMENT = -1008,

  // Returned by `add_face_embedding` or `remove_face_embedding` when the
  // provided face ID is invalid.
  INVALID_FACE_ID = -1009,

  // Returned by `add_face_embedding` when the provided face embedding is
  // invalid.
  INVALID_EMBEDDING = -1010,
} ErrorCodeFamiliarFaceID;

Possible error codes for the enrollment procedure. Furthermore, the regular ErrorCode values might be returned.

Constants¶

static const int FACE_ID_UNIDENTIFIED = -1;
static const int FACE_ID_NOT_IN_LIBRARY = -2;

If the face-identification model is not confident about a face, it will output FACE_ID_UNIDENTIFIED. If the model is confident that a face is not in the face library, it will output FACE_ID_NOT_IN_LIBRARY.

Upgrade guide¶

From version 1.14 to 1.15¶

In version 1.15 the API of start_face_enrollment and finish_face_enrollment changed compared to earlier versions:

The function get_cumulative_enrollment_score was removed.
The previous_embedding argument of start_face_enrollment was removed.
The resulting embedding of finish_face_enrollment is now returned via a reference argument.
The return type of both functions is now an error code, and finish_face_enrollment can indicate a low quality enrollment through this error code. There is no more need to check the enrollment score for the quality of the embedding.

If your code looked like this before:

const auto embedding = ffid.finish_face_enrollment();

Then it should be updated as follows:

std::vector<int8_t> embedding;
const auto error_code = ffid.finish_face_enrollment(embedding);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("Error code: %d\n", error_code);
}