Plumerai Familiar Face Identification C++ API¶

This document describes the C++ API for the Plumerai Familiar Face Identification software with manual enrollment for videos on Arm Cortex-A and x86. For an example, see here.

The C++ API consists of a class plumerai::FaceIdentification which is a subclass of plumerai::PeopleDetection. Processing video input is described in the documentation of the base class. The FaceIdentification class adds functionality for controlling the face library, which is documented here.

The API is re-entrant, i.e. you can instantiate several FaceIdentification objects in different threads and use them independently. However, using the same instance from different threads at the same time is not supported.

API¶

FaceIdentification¶

FaceIdentification(int height, int width);

Initializes a new face identification object.

This needs to be called only once at the start of the application.

Arguments:

height: The height of the input image in pixels.
width: The width of the input image in pixels.

Returns:

Nothing.

face_embedding_size¶

static std::size_t face_embedding_size();

Returns the size of the face embedding vectors obtained from finish_face_enrollment and required by add_face_embedding.

Arguments:

None.

Returns:

The size of the face embedding vectors.

face_embedding_version_checksum¶

static std::uint32_t face_embedding_version_checksum();

Returns a version checksum for the face embedding vectors obtained from finish_face_enrollment and required by add_face_embedding.

This checksum is used to check if the face embedding vectors generated with one library build are compatible with another library.

Arguments:

None.

Returns:

The version checksum of the face embedding vectors.

start_face_enrollment¶

ErrorCodeType start_face_enrollment(
    const Region &region_of_interest = Region());

This starts the enrollment procedure for a new face.

During subsequent calls to process_frame or single_image, face embeddings will be computed. During enrollment there should be a single face in the image, or in the optional Region specified by region_of_interest, ideally clearly and completely visible. The enrollment procedure can be finalized by calling finish_face_enrollment.

Arguments:

region_of_interest: This can be set to a region of interest in the image. Only faces that have overlap with this region will be used for enrollment. The default Region is the entire image.

Returns:

Returns ENROLLMENT_IN_PROGRESS if the enrollment procedure was started successfully, otherwise returns ALREADY_ENROLLING.

Example:

auto ffid = plumerai::FaceIdentification(height, width);

auto ec = ffid.start_face_enrollment();
if (ec != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
  printf("ERROR: start_face_enrollment returned %d\n", ec);
}

// Process the face images
std::vector<BoxPrediction> predictions(0);
for (size_t i = 0; i < num_images; ++i) {
  ec = ffid.process_frame<image_format>(image_data[i], predictions);
  if (ec != plumerai::ErrorCodeFamiliarFaceID::ENROLLMENT_IN_PROGRESS) {
    // The image might be rejected if the face is not clearly visible
    printf("Warning, face image %d rejected: %d\n", i, ec);
  }
}

std::vector<int8_t> embedding;
ec = ffid.finish_face_enrollment(embedding);
if (ec != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: finish_face_enrollment returned %d\n", ec);
}

finish_face_enrollment¶

ErrorCodeType finish_face_enrollment(
    std::vector<std::int8_t> &face_embedding);

Finalize the face enrollment procedure started by start_face_enrollment and obtain the face embedding.

The application is responsible for storing this embedding on persistent storage and passing it to add_face_embedding every time an instance of this class is created. The face enrollment procedure does not automatically call add_face_embedding.

For an example, see the example under start_face_enrollment above.

Arguments:

face_embedding: The resulting face embedding vector if the result is SUCCESS or LOW_QUALITY_ENROLLMENT, or an empty vector otherwise.

Returns:

Returns SUCCESS when enrollment was successful. Otherwise, returns one of NOT_ENROLLING, ENROLLMENT_FAILED, LOW_QUALITY_ENROLLMENT. See the enum for more information.

add_face_embedding¶

ErrorCodeType add_face_embedding(
    const std::vector<std::int8_t> &face_embedding, const int face_id);

Add a face embedding to the face library for any process_frame calls that follow.

A second embedding for the same face id will overwrite the previous one. When calling get_face_id on a detected person box, it will use the face_id provided here as return value.

Arguments:

face_embedding: A face embedding as obtained from finish_face_enrollment.
face_id: An integer face-id, is not allowed to be equal to FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY.

Returns:

Returns SUCCESS on success, INVALID_FACE_ID or INVALID_EMBEDDING on failure.

Example:

auto ffid = plumerai::FaceIdentification(height, width);

std::vector<std::int8_t> face_embedding = ...;

// Add with face-id 3
auto error_code = ffid.add_face_embedding(face_embedding, 3);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("ERROR: add_face_embedding returned %d\n", error_code);
}

remove_face_embedding¶

ErrorCodeType remove_face_embedding(const int face_id);

Remove a face embedding from the face library for any process_frame calls that follow.

Arguments:

face_id: An integer face-id, previously used in add_face_embedding.

Returns:

Returns SUCCESS on success, INVALID_FACE_ID on failure.

get_face_id¶

int get_face_id(const BoxPrediction &person_box) const;

Retrieve the face ID that belongs to a person box.

Retrieves the face ID given a person box returned by process_frame. This function has three possible return values:

FACE_ID_UNIDENTIFIED when the provided box is not of CLASS_PERSON or when a face is not yet identified, e.g. the face is not fully visible.
FACE_ID_NOT_IN_LIBRARY when the face is visible but not found in the library, e.g. when a stranger is in view.
Any integer face-ID provided previously to add_face_embedding when a familiar face is identified.

This function should not be called directly after restoring from a previous state.

This function should not be called with results of single_image calls.

Arguments:

person_box: A box returned by process_frame with class_id equal to DetectionClass::CLASS_PERSON.

Returns:

An integer face-ID as provided when calling add_face_embedding when a face is identified, FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY when it is not identified successfully, or FACE_ID_UNIDENTIFIED when the provided box is not a valid person box.

get_person_box_from_face_box¶

bool get_person_box_from_face_box(const BoxPrediction &face_box,
                                  BoxPrediction *result_person_box) const;

Retrieve a person box corresponding to a face box.

This function should not be called with results of single_image calls.

Arguments:

face_box: The target face to find a match for. This has to be a box from the most recent call to process_frame.
result_person_box: The resulting box, only valid if the return value is true (see below). The caller needs to allocate memory for this box prior to calling this function.

Returns:

False in case of invalid arguments or when it was not possible to uniquely match a face box to a person box. Returns true in case a match was found.

Region¶

struct Region {
  float y_min = 0.0f;  // top coordinate between 0 and 1 in height dimension
  float x_min = 0.0f;  // left coordinate between 0 and 1 in width dimension
  float y_max = 1.0f;  // bottom coordinate between 0 and 1 in height dimension
  float x_max = 1.0f;  // right coordinate between 0 and 1 in width dimension
};

A structure representing a region of the input image. Coordinates are between 0 and 1, the origin is at the top-left.

ErrorCodeFamiliarFaceID¶

typedef enum {
  // Everything OK, enrollment is in progress.
  ENROLLMENT_IN_PROGRESS = -1000,

  // This library is not built with support for face identification.
  FACE_IDENTIFICATION_DISABLED = -1001,

  // Called `start_face_enrollment` but enrollment is already started.
  ALREADY_ENROLLING = -1002,

  // Last frame was ignored because multiple faces were detected within the
  // specified region of interest. Try again with one face visible.
  MULTIPLE_FACES_IN_VIEW = -1003,

  // Last frame was ignored because no face was detected, or it was not clearly
  // visible, or the lighting was not good enough for enrollment. Try again with
  // one face clearly visible.
  NO_FACE_IN_VIEW = -1004,

  // Last frame was ignored because the face detection was too close to the
  // edge. Too close to the edge means that the border of the face bounding-box
  // is within 2% of the edge of the image. In theory, it is OK if a face is
  // close to edge, however, it can also mean that a face is only partially
  // visible. Thus, this check is added to prevent partial faces from being
  // enrolled.
  FACE_CLOSE_TO_EDGE = -1005,

  // Called `finish_face_enrollment` but enrollment was not started.
  NOT_ENROLLING = -1006,

  // This is returned by `finish_face_enrollment` when the enrollment failed
  // because no faces could be detected in any of the images.
  ENROLLMENT_FAILED = -1007,

  // This is returned by `finish_face_enrollment` when a face embedding vector
  // is returned but the quality of the enrollment is very low. In this case
  // re-enrolllment is recommended.
  LOW_QUALITY_ENROLLMENT = -1008,

  // Returned by `add_face_embedding` or `remove_face_embedding` when the
  // provided face ID is invalid.
  INVALID_FACE_ID = -1009,

  // Returned by `add_face_embedding` when the provided face embedding is
  // invalid.
  INVALID_EMBEDDING = -1010,
} ErrorCodeFamiliarFaceID;

Possible error codes for the enrollment procedure. Furthermore, the regular ErrorCode values might be returned.

Constants¶

static const int FACE_ID_UNIDENTIFIED = -1;
static const int FACE_ID_NOT_IN_LIBRARY = -2;

If the face-identification model is not confident about a face, it will output FACE_ID_UNIDENTIFIED. If the model is confident that a face is not in the face library, it will output FACE_ID_NOT_IN_LIBRARY.

Upgrade guide¶

From version 1.14 to 1.15¶

In version 1.15 the API of start_face_enrollment and finish_face_enrollment changed compared to earlier versions:

The function get_cumulative_enrollment_score was removed.
The previous_embedding argument of start_face_enrollment was removed.
The resulting embedding of finish_face_enrollment is now returned via a reference argument.
The return type of both functions is now an error code, and finish_face_enrollment can indicate a low quality enrollment through this error code. There is no more need to check the enrollment score for the quality of the embedding.

If your code looked like this before:

const auto embedding = ffid.finish_face_enrollment();

Then it should be updated as follows:

std::vector<int8_t> embedding;
const auto error_code = ffid.finish_face_enrollment(embedding);
if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("Error code: %d\n", error_code);
}