July 2024

Human Pose Estimation - Real-time and ROS Integration

Articles

• Aceptance of IROS paper
• Reviews of IROS paper: Overleaf Review details (Google Docs)
• Resubmission of calibration paper to JMS
• Resubmission of calibration paper to IEEE Transactions on Robotics

Human Pose Estimation - Adaptation to Real Time

One of the tasks during this time was to adapt the human pose estimation methodology to work in real time.

Adaptations necessary

• Create an argument that defines the number of frames to optimize at each time. Meaning that, if we are in frame i, we will also optimize the last n frames, being n an argument given by the operator.
• Check if we are in the first frames and adapt the frame list to optimize accordingly. For example, if we want to optimize the last 10 frames at each time and we are only in frame 5, we can only optimize the last exisinting 5 frames. Only when we reach frame 10 and foward we are able to optimize the last 10 frames
• Adapt the first guess to be the last known pose of the human. For the first detected frame, we optimize (as before) from (0,0,0) for all joints, but from that frame foward, we always use the pose of the optimized frame to accelerate the process.
• Read images in real time
• Adapt visualization to real-time and reduce time-consuming features
• Adapt code to perform faster by using more time-efficient functions and numba processing when possible

There were/are two main challenges in this adaption to real time. The first and more obvious is the time-efficiency of the process. When I started this process, the time per frame for 5 simultaneous frames and without the frame-to-frame and link length residuals was around 2.5s. Currently, for 5 simultaneous frames, I can reach times between 150-750ms with the link length and frame-to-frame residuals. The time still varies a lot from frame to frame, some take a lot more time to optimize than others. Also, the more frames we optimize at each time, the more time it takes.

Here are two examples when optimizing with 5 and 20 simultaneous frames, respectively.

The second challenge faced with this adaptation (as you can probably notice in the videos) is that the performance significantly decreased. There is a balance betweem the number of previous frames we can use and how time-consuming the optimization will be. But even when increasing the number of frames to 20, the performance doesn't seem to improve a lot (please take into consideration, as I explained previously, that the first 20 frames don't actually have 20 simultaneous frames being optimize). Also, the optimization takes a lot more time, around 2.5s per frame.

As you can see on the following debug table, specially the right leg is having significally high errors when compared to previous performances.

Human Pose Estimation - ROS Integration

The integration of the human pose estimation algorithm with ROS will allow to run this package alongside robotics applications and therefore enable 3D monitoring for human-robot collaboration.

Adaptations necessary:

• Create node that reads ROS images (the topics are given as an argument), runs MediaPipe and constantly publishes a customized message with the 2D keypoint information for each camera topic.
• Launch xacro with robot model.
• Read CameraInfo to find the intrinsics, distortion, w and h for each camera and store in dictionary.
• Read tf information to find camera extrinsics and store in dictionary.
• Create dictionary for 3D poses.
• Create a marker structure that designs the skeleton in RVIZ.
• Create new node that subscribes to those topics and stores them in a dictionary.
• Create universal callback that receives 2D poses and assigns them to the correct topic in the dictionary.
• Check if cameras have a valid detection.
• Assign frame indexes to arriving camera poses, knowing that the optimization can occur only if we have two or more valid camera detections. This is tricky because a frame index needs to be synchronized for all cameras in order for us to know if we have two or more 2D detections at that time and be able to run optimization. As we are constantly loosing 2D detections as people leave the field of view of the camera, I have yet to find an efficient way to assign frame indexes and determine if can activate the optimization at a given moment.
• Make the necessary verifications before optimization.
• Create the necessary residuals.
• Run optimization if the conditions are met.
• Publish 3D poses in a customized message.
• Publish markers with 3D skeleton design.

This migration to ROS is almost finalized, apart from the time-synchronized decision of when the conditions are met to activate the optimization.

Issues of the month

• Real time HPE - open
• Why does objective function runs the same thing for each iteration? - open
• Time diagnosis - open
• ROS integration - open
• Create n subscribers with the same callback - closed
• Mismatch between created residuals and assigned residuals in the objective function - open