Skip to content

Integration checklist for the Plumerai Video Intelligence library

This page describes a series of checks to perform when integrating the API. The checks are written assuming that the C++ API is used, but the same checks hold for the other APIs.

Even if the product appears to be working as expected, it is possible that accuracy may be degraded, so we recommend to always follow this checklist.

Please check all the following items

  • Input image data: If the input image data is not entirely correct, then some object detection features may still work and might give the impression that the implementation is successful, but the accuracy can still be severely degraded.
    • Check that the image is right side up. The pixels should start at the top-left, go line-by-line, and end at the bottom-right.
    • Check the color format: If the camera (or other source) natively produces YUV data, there is no need to convert to RGB first, because the Plumerai library can accept YUV data directly. This can save the additional compute of an extra conversion step.
    • Check that the color channels are not flipped. It is a common mistake to mix-up RGB and BGR, and sometimes the naming scheme is inconsistent: the camera vendor might refer to their format as RGB when it is BGR or vice versa. If using the RGB input format, make sure that the first byte in memory is the red pixel and the third byte is the blue pixel.
  • Input resolution
    • It is important to have the highest input resolution possible. We recommend a minimum of 720p, but 1080p or higher is preferred. At lower resolutions it is harder to detect objects at a distance. In case of face identification, there is not enough detail visible in the faces to identify them at low resolutions. Note that a higher resolution has only a small impact on the execution time of the Plumerai library.

  • Timing information: The delta_t parameter of process_frame should be correct. Incorrect values for delta_t will negatively affect accuracy.

    • The delta_t parameter should be the time in seconds (as floating point value) between the capture of the current frame and the previous frame. Variable framerates are supported.

      Typical camera APIs provide a timestamp with each image frame. The delta_t parameter should be the difference between consecutive timestamps.

      If delta_t = 0 is used, the library will use the system clock to measure the time between invocations of process_frame, and this will not always be the same as the time between the originally captured frame timestamps. While this may work for a quick demo application, it will likely not be optimal.

      Common mistakes are passing a time in milliseconds or passing the total time elapsed so far, i.e. a timestamp of the current frame instead of the difference between consecutive frames.

  • ISP tuning: Tuning camera settings such as exposure for video intelligence accuracy may result in different settings compared to optimizing for human perception. Motion blur can negatively impact detection accuracy. Therefore it is recommended to have a relatively low exposure value, even if that would result in an image that looks slightly underexposed to the human eye. We recommend recording a person walking or running in front of the camera and pausing the recorded video at multiple moments to verify that the image looks sharp.

After completing the checklist, we recommend using the function debug_next_frame to generate a file with debug information. This file can then be analyzed by Plumerai to confirm everything is working as expected.