Plumerai Familiar Face Identification API¶

This document describes the API for the Familiar Face Identification functionality.

Note that the software is built with either the Automatic Enrollment option or the Manual Enrollment option enabled. This page provides documentation for both options, but at runtime only one of the two will be available.

FaceIdentification¶

get_face_id¶

int get_face_id(const BoxPrediction& person_box) const;

Retrieve the face ID that belongs to a person box.

Retrieves the face ID given a person box from the most recent video frame.

This function has three possible return values:

FACE_ID_UNIDENTIFIED when the provided box is not of CLASS_PERSON or when a face is not yet identified, e.g. the face is not fully visible.
FACE_ID_NOT_IN_LIBRARY when the face is visible but not found in the library, e.g. when a stranger is in view.
Any integer face ID provided previously to FaceEnrollmentManual::add_embedding when a familiar face is identified.

This function should not be called directly after restoring from a previous state through VideoIntelligence::restore_state.

This function should not be called with results of VideoIntelligence::single_image calls.

Arguments:

person_box: A box from the most recent video frame with class_id equal to DetectionClass::CLASS_PERSON.

Returns:

An integer face ID as provided when calling FaceEnrollmentManual::add_embedding when a face is identified, FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY when it is not identified successfully, or FACE_ID_UNIDENTIFIED when the provided box is not a valid person box.

get_person_box_from_face_box¶

ErrorCode get_person_box_from_face_box(
    const BoxPrediction& face_box, BoxPrediction* result_person_box) const;

Retrieve a person box corresponding to a face box.

This function should not be called with results of VideoIntelligence::single_image calls.

Arguments:

face_box: The target face to find a match for. This has to be a box from the most recent video frame.
result_person_box: The resulting box, only valid if the return value is true (see below). The caller needs to allocate memory for this box prior to calling this function.

Returns:

Returns SUCCESS in case a match was found. Returns NO_BOX_MATCH if it was not possible to uniquely match a face box to a person box. Returns INVALID_BOX in case of invalid arguments.

get_face_box_from_person_box¶

ErrorCode get_face_box_from_person_box(const BoxPrediction& person_box,
                                       BoxPrediction* result_face_box) const;

Retrieve a face box corresponding to a person box.

This function should not be called with results of VideoIntelligence::single_image calls.

Arguments:

person_box: The target person to find a match for. This has to be a box from the most recent video frame.
result_face_box: The resulting box, only valid if the return value is true (see below). The caller needs to allocate memory for this box prior to calling this function.

Returns:

Returns SUCCESS in case a match was found. Returns NO_BOX_MATCH if it was not possible to uniquely match a face box to a person box. Returns INVALID_BOX in case of invalid arguments.

embedding_version¶

static std::uint32_t embedding_version();

Returns a version number for the face embedding data used by the face identification logic.

This version can be used to check if the face embedding data generated with one version of the Plumerai software are compatible with those generated by another version of the Plumerai software.

Returns:

The version number of the face embedding data.

FaceEnrollmentAutomatic¶

get_face_ids¶

ErrorCode get_face_ids(int* face_ids, std::size_t* num_face_ids) const;
ErrorCode get_face_ids(std::vector<int>& face_ids) const;

Return the face IDs of all automatically enrolled faces.

This list of face IDs is not related to the faces currently in view. The face IDs returned by this function are all faces that have ever been enrolled into the face library. To retrieve information about the faces currently in view, use FaceIdentification::get_face_id.

The caller must provide a buffer of size at least max_face_library_size integers.

Arguments:

face_ids: A pointer to a user-allocated list that receives the enrolled face IDs. It must hold at least max_face_library_size integers.
num_face_ids: An output parameter that receives the number of returned values.

Returns:

Returns SUCCESS on success.

tag_face_id¶

ErrorCode tag_face_id(const int face_id, bool important = true);

Tag a face ID as 'important' to prevent it from being overwritten.

When the face library is full and a new face is encountered, the algorithm will overwrite the oldest face in the library that is not tagged as important. A typical use-case of this is to tag a face as important when the user has named them in a front-end application.

When all entries in the face library are tagged as important and a new face is encountered, then no new faces will be enrolled.

By default, an automatically enrolled face is not tagged as important. However, library entries created by restore_enrollment_data are automatically tagged as important.

Note that INVALID_FACE_ID can be returned when a face ID that was previously valid is overwritten with a new entry.

Arguments:

face_id: A non-negative integer face ID.
important: Whether to tag the face as important or to remove a previously applied tag.

Returns:

Returns SUCCESS on success, INVALID_FACE_ID when the specified face ID could not be found.

remove_face_id¶

ErrorCode remove_face_id(const int face_id);

Remove a face ID from the face library for any VideoIntelligence::process_frame calls that follow.

Note that INVALID_FACE_ID can be returned when a face ID that was previously valid is overwritten with a new entry.

Arguments:

face_id: A non-negative integer face ID.

Returns:

Returns SUCCESS on success, INVALID_FACE_ID on failure.

remove_all_face_ids¶

ErrorCode remove_all_face_ids();

Remove all face IDs from the face library.

Returns:

Returns SUCCESS on success.

get_enrollment_data¶

ErrorCode get_enrollment_data(const int face_id,
                              std::vector<std::uint8_t>& enrollment_data,
                              bool include_face_snapshots) const;

Get the face enrollment data for a face ID, for persistent storage.

This can for example be used to store the face library when a device shuts down and restore it later when it reboots.

This enrollment data can change during any VideoIntelligence::process_frame call.

Note that INVALID_FACE_ID can be returned when a face ID that was previously valid is overwritten with a new entry.

Arguments:

face_id: A non-negative integer face ID.
enrollment_data: A pointer to a user-allocated list that receives the data.
include_face_snapshots: Whether to include the face snapshots in the data. The snapshots are stored as raw RGB data.

Returns:

Returns SUCCESS on success, INVALID_FACE_ID when the specified face ID could not be found, or INTERNAL_ERROR in case of an unexpected error.

restore_enrollment_data¶

ErrorCode restore_enrollment_data(
    const int face_id, const std::vector<std::uint8_t>& enrollment_data);

Restore the face enrollment data for a face ID, from persistent storage.

When the specified face_id does not yet exist then it will be created. When the specified face_id already exists, the internal enrollment data will be overwritten with new enrollment data. If the new enrollment data is corrupt, this will result in removal of the existing face enrollment data.

If a new face_id was created, this function will automatically tag it as 'important', see also tag_face_id.

If the enrollment data does not include a face snapshot, then a face snapshot for this face ID will not be available. If the enrollment data includes a face snapshot, then configure_face_snapshots must have been called first, with enabled set to true and the same height and width as the stored data. If the height and width do not match, then STATE_SETTINGS_MISMATCH is returned.

Arguments:

face_id: A non-negative integer face ID.
enrollment_data: The data to restore.

Returns:

Returns SUCCESS on success, or FACE_LIBRARY_FULL when the face library is full, or STATE_CORRUPT when the enrollment data was corrupted. If the height and width of an included face snapshot do not match, then STATE_SETTINGS_MISMATCH is returned.

merge_face_ids¶

ErrorCode merge_face_ids(const int face_id_dst, const int face_id_src);

Merge two entries of the face library.

If a single person is enrolled twice in the face library, this function can be used to merge their enrollments back to a single one. After calling this function face_id_src is no longer valid and face_id_dst will be updated with the additional information from face_id_src.

Arguments:

face_id_dst: A non-negative integer face ID.
face_id_src: A non-negative integer face ID, must be different from face_id_dst.

Returns:

Returns SUCCESS on success, INVALID_FACE_ID when either of the face IDs could not be found, or INTERNAL_ERROR in case of an unexpected error.

get_face_enrollment_track_ids¶

ErrorCode get_face_enrollment_track_ids(const int face_id,
                                        std::int64_t* track_ids,
                                        std::size_t* num_track_ids) const;
ErrorCode get_face_enrollment_track_ids(
    const int face_id, std::vector<std::int64_t>& track_ids) const;

Return the track IDs for an automatically enrolled face.

The caller must provide a buffer of size at least max_tracks_per_face_id integers.

A face enrollment (face ID) can be made up of multiple person tracks, each of which is identified with a track ID. When a person is re-identified and their new face information is considered relevant for the algorithm, the new person track will be added to the existing face enrollment, or replace a previous one.

This list of track IDs can therefore change after processing any video frame.

Arguments:

face_id: A non-negative integer face ID.
track_ids: A pointer to a user-allocated list that receives the track IDs. It must hold at least max_tracks_per_face_id integers.
num_track_ids: An output parameter that receives the number of returned values.

Returns:

Returns SUCCESS on success.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);

std::vector<int> enrolled_face_ids;
pvi.face_enrollment_automatic().get_face_ids(enrolled_face_ids);
const int face_id = enrolled_face_ids[0];

const auto max_tracks_per_face_id =
    pvi.face_enrollment_automatic().max_tracks_per_face_id();
// Allocate space for enough track ids
std::vector<std::int64_t> track_ids(max_tracks_per_face_id);
std::size_t num_track_ids = 0;
pvi.face_enrollment_automatic().get_face_enrollment_track_ids(face_id,
                                                              track_ids.data(),
                                                              &num_track_ids);
// Use track_ids to get face snapshots or reassign tracks

remove_face_enrollment_track¶

ErrorCode remove_face_enrollment_track(const int face_id,
                                       const std::int64_t track_id);

Remove a track that is part of a face enrollment.

This can be used if the Plumerai library made a mistake with automatic enrollment, and a user wants to manually correct the face library. By inspecting the face snapshots of each track ID, the user can decide to remove a wrong track.

If this was the only track of face_id then this face ID is removed entirely.

Arguments:

face_id: A non-negative integer face ID that is the source of the track.
track_id: The track ID of the track to remove.

Returns:

One of SUCCESS, INVALID_FACE_ID or INVALID_TRACK_ID.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);

std::vector<int> enrolled_face_ids;
pvi.face_enrollment_automatic().get_face_ids(enrolled_face_ids);
// Example: remove the first track of the first face ID
const int face_id = enrolled_face_ids[0];
std::vector<std::int64_t> track_ids;
pvi.face_enrollment_automatic().get_face_enrollment_track_ids(face_id,
                                                              track_ids);
pvi.face_enrollment_automatic().remove_face_enrollment_track(face_id,
                                                             track_ids[0]);

reassign_face_enrollment_track¶

ErrorCode reassign_face_enrollment_track(const int face_id_src,
                                         const std::int64_t track_id_src,
                                         const int face_id_dst);

Reassign a face track from face ID face_id_src to a new or existing face id face_id_dst.

This can be used if the Plumerai library made a mistake with automatic enrollment, and a user wants to manually correct the face library. By inspecting the face snapshots of each track ID, the user can decide to reassign a track to the correct face ID.

If face_id_dst does not exist, it will be created. If the face library is full, this will overwrite the oldest library entry, unless all of them are tagged as important in which case this function returns FACE_LIBRARY_FULL.

Note the following special behavior:

If face_id_src has only one track and it is reassigned, then face_id_src is automatically removed entirely.

It is possible that one or more existing tracks of face_id_dst are overwritten depending on whether the algorithm think it will improve the overall quality of the face enrollment.

The algorithm may determine that the original tracks of face_id_dst provide a better enrollment without the newly reassigned track. In such cases, face_id_dst remains unchanged, but the track is still removed from face_id_src.

Arguments:

face_id_src: A non-negative integer face ID that is the source of the track.
track_id_src: The track ID of the track to reassign.
face_id_dst: A non-negative integer face ID that is the destination of the track. If this face ID does not exist, it will be created.

Returns:

One of SUCCESS, FACE_LIBRARY_FULL, INVALID_FACE_ID or INVALID_TRACK_ID.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);

std::vector<int> enrolled_face_ids;
pvi.face_enrollment_automatic().get_face_ids(enrolled_face_ids);
// Example: reassign the first track of the first face ID to the second
// face ID
const int face_id_src = enrolled_face_ids[0];
const int face_id_dst = enrolled_face_ids[1];
std::vector<std::int64_t> track_ids;
pvi.face_enrollment_automatic().get_face_enrollment_track_ids(face_id_src,
                                                              track_ids);
pvi.face_enrollment_automatic().reassign_face_enrollment_track(face_id_src,
                                                               track_ids[0],
                                                               face_id_dst);

configure_face_snapshots¶

ErrorCode configure_face_snapshots(const FaceSnapshotSettings& settings,
                                   std::uint8_t* buffer = nullptr,
                                   std::size_t buffer_size = 0);

Configure the face snapshots for automatic enrollment.

By default, face snapshots are disabled.

Calling this function will invalidate any stored face snapshots of faces that are already enrolled. Trying to retrieve those face snapshots will return uninitialized data. When buffer is nullptr and buffer_size is 0, the function will allocate heap memory to store the face snapshots (default, recommended). Otherwise, the function will use a user-provided buffer to store the face snapshots.

Arguments:

settings: See FaceSnapshotSettings for more details.
buffer: An optional buffer to store face snapshots, must stay valid for the lifetime of the VideoIntelligence object or until the next call to configure_face_snapshots. If nullptr, the function will allocate heap memory.
buffer_size: The size of the buffer. The required size can be obtained through get_face_snapshots_required_memory_size.

Returns:

Returns SUCCESS on success, INVALID_ARGUMENT when the provided settings are not valid, or NOT_ENOUGH_MEMORY if the provided buffer is not large enough.

get_face_snapshots_required_memory_size¶

std::size_t get_face_snapshots_required_memory_size(
    const FaceSnapshotSettings& settings) const;

Get the required memory size for the face snapshots.

This optional function can be used to determine the required memory size for the face snapshots on system without dynamic memory allocation. See configure_face_snapshots for details.

Arguments:

settings: See FaceSnapshotSettings for more details.

Returns:

The required memory size in bytes.

get_face_snapshot¶

ErrorCode get_face_snapshot(const int face_id, const std::int64_t track_id,
                            const std::uint8_t** face_snapshot_data) const;

Return a snapshot of an automatically enrolled face.

A face enrollment can be made up of multiple person tracks. Every snapshot originates from a single person track. Use get_face_enrollment_track_ids to get a list of track IDs to use in this function.

Note that INVALID_FACE_ID can be returned when a face ID that was previously valid is overwritten with a new entry.

After every processed frame the list of snapshots can change. New images might be used by the algorithm if they are of higher quality.

Arguments:

face_id: A non-negative integer face ID.
track_id: The track ID of the snapshot to retrieve.
face_snapshot_data: The pointer to the stored snapshot data is returned via this output argument.

Returns:

Returns SUCCESS on success, INVALID_FACE_ID when the specified face ID could not be found, INVALID_TRACK_ID when the track ID could not be found, or FACE_SNAPSHOTS_DISABLED when face snapshots were not configured using configure_face_snapshots.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);

std::vector<int> enrolled_face_ids;
pvi.face_enrollment_automatic().get_face_ids(enrolled_face_ids);
for (auto face_id : enrolled_face_ids) {
  std::vector<std::int64_t> track_ids;
  pvi.face_enrollment_automatic().get_face_enrollment_track_ids(face_id,
                                                                track_ids);
  for (auto track_id : track_ids) {
    const uint8_t *face_snapshot_data = nullptr;
    pvi.face_enrollment_automatic().get_face_snapshot(face_id, track_id,
                                                      &face_snapshot_data);
    // Render face snapshot to screen or save to disk
  }
}

max_face_library_size¶

static std::size_t max_face_library_size();

Returns the maximum size of the face library.

When the library is full and new faces are detected, they will overwrite older faces, except for faces that are tagged as important using tag_face_id. When all faces are tagged as important, new faces will not be added to the library beyond this maximum size.

Returns:

The maximum size of the face library.

max_tracks_per_face_id¶

static std::size_t max_tracks_per_face_id();

Returns the maximum number of tracks that a single face ID in the library consists of.

See get_face_enrollment_track_ids for more information.

When a face ID already has the maximum number of tracks stored, then a new person track will only update this face ID if it is of better quality than the existing tracks.

Returns:

The maximum number of tracks for a single face ID.

FaceEnrollmentManual¶

embedding_size¶

static std::size_t embedding_size();

Returns the size of the face embeddings obtained from finish_enrollment and required by add_embedding.

Returns:

The size of the face embeddings.

max_face_library_size¶

static std::size_t max_face_library_size();

Returns the maximum size of the face library.

When the library is full, add_embedding will return FACE_LIBRARY_FULL and won't be able to add new embeddings, until FaceEnrollmentManual::remove_embedding is called.

Returns:

The maximum size of the face library.

start_enrollment¶

ErrorCode start_enrollment(const Region& region_of_interest = Region());

This starts the enrollment procedure for a new face.

During subsequent calls to VideoIntelligence::process_frame or VideoIntelligence::single_image, face embeddings will be computed. During enrollment there should be a single face in the image, or in the optional Region specified by region_of_interest (not available in the C API), ideally clearly and completely visible. The enrollment procedure can be finalized by calling finish_enrollment.

Arguments:

region_of_interest: This can be set to a region of interest in the image. Only faces that have overlap with this region will be used for enrollment. The default Region is the entire image.

Returns:

Returns ENROLLMENT_IN_PROGRESS if the enrollment procedure was started successfully, otherwise returns ALREADY_ENROLLING.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);

auto ec = pvi.face_enrollment_manual().start_enrollment();
if (ec != plumerai::ErrorCode::ENROLLMENT_IN_PROGRESS) {
  printf("ERROR: start_enrollment returned %d\n", ec);
}

// Process the face images
for (size_t i = 0; i < num_images; ++i) {
  ec = pvi.single_image(image, height, width);
  if (ec != plumerai::ErrorCode::ENROLLMENT_IN_PROGRESS) {
    // The image might be rejected if the face is not clearly visible
    printf("Warning, face image %d rejected: %d\n", i, ec);
  }
}

int8_t embedding[FaceEnrollmentManual::embedding_size()];
ec = pvi.face_enrollment_manual().finish_enrollment(embedding);
if (ec != plumerai::ErrorCode::SUCCESS) {
  printf("Error: %s\n", plumerai::error_code_string(error_code));
}

finish_enrollment¶

ErrorCode finish_enrollment(std::int8_t* face_embedding);
ErrorCode finish_enrollment(std::vector<std::int8_t>& face_embedding);

Finalize the face enrollment procedure started by start_enrollment and obtain the face embedding.

The caller must provide a buffer of size embedding_size.

The application is responsible for storing this embedding on persistent storage and passing it to add_embedding every time an instance of this class is created. The face enrollment procedure does not automatically call add_embedding.

For an example, see the example under start_enrollment above.

On some platforms, a version that uses std::vector is available.

Arguments:

face_embedding: The resulting face embedding if the result is SUCCESS or LOW_QUALITY_ENROLLMENT.

Returns:

Returns SUCCESS when enrollment was successful. Otherwise, returns one of NOT_ENROLLING, ENROLLMENT_FAILED, LOW_QUALITY_ENROLLMENT. See the enum for more information.

add_embedding¶

ErrorCode add_embedding(const std::int8_t* face_embedding,
                        std::size_t face_embedding_size, int face_id);
ErrorCode add_embedding(const std::vector<std::int8_t>& face_embedding,
                        int face_id);

Add a face embedding to the face library for any VideoIntelligence::process_frame calls that follow.

A second embedding for the same face ID will overwrite the previous one. When calling FaceIdentification::get_face_id on a detected person box, it will use the face_id provided here as return value.

On some platforms, a version that uses std::vector is available.

Arguments:

face_embedding: A face embedding as obtained from finish_enrollment.
face_embedding_size: Only used in the case when face_embedding is a raw pointer: the size of the face embedding data.
face_id: An integer face ID, is not allowed to be equal to FACE_ID_UNIDENTIFIED or FACE_ID_NOT_IN_LIBRARY.

Returns:

Returns SUCCESS on success, INVALID_FACE_ID, FACE_LIBRARY_FULL, or INVALID_EMBEDDING on failure.

Example:

auto pvi = plumerai::VideoIntelligence(height, width);

std::vector<std::int8_t> face_embedding = ...;

// Add with face ID 3
auto error_code =
    pvi.face_enrollment_manual().add_embedding(face_embedding, 3);

if (error_code != plumerai::ErrorCode::SUCCESS) {
  printf("Error: %s\n", plumerai::error_code_string(error_code));
}

get_face_ids¶

ErrorCode get_face_ids(int* face_ids, std::size_t* num_face_ids) const;

Return the face IDs of all enrolled faces.

This list of face IDs is not related to the faces currently in view. The face IDs returned by this function are all faces that have ever been enrolled into the face library. To retrieve information about the faces currently in view, use FaceIdentification::get_face_id.

The caller must provide a buffer of size at least max_library_size integers.

Arguments:

face_ids: A pointer to a user-allocated list that receives the enrolled face IDs. It must hold at least max_library_size integers.
num_face_ids: An output parameter that receives the number of returned values.

Returns:

Returns SUCCESS on success.

remove_embedding¶

ErrorCode remove_embedding(int face_id);

Remove a face embedding from the face library for any VideoIntelligence::process_frame calls that follow.

Arguments:

face_id: An integer face ID, previously used in add_embedding.

Returns:

Returns SUCCESS on success, INVALID_FACE_ID on failure.

remove_all_embeddings¶

ErrorCode remove_all_embeddings();

Remove all face embeddings from the face library.

Returns:

Returns SUCCESS on success.

Region¶

struct Region {
  float y_min = 0.0f;  // top coordinate between 0 and 1 in height dimension
  float x_min = 0.0f;  // left coordinate between 0 and 1 in width dimension
  float y_max = 1.0f;  // bottom coordinate between 0 and 1 in height dimension
  float x_max = 1.0f;  // right coordinate between 0 and 1 in width dimension
};

A structure representing a region of the input image. Coordinates are between 0 and 1, the origin is at the top-left.

Constants¶

static const int FACE_ID_UNIDENTIFIED = -1;
static const int FACE_ID_NOT_IN_LIBRARY = -2;

If the face-identification model is not confident about a face, it will output FACE_ID_UNIDENTIFIED. If the model is confident that a face is not in the face library, it will output FACE_ID_NOT_IN_LIBRARY.