Triangulation between pairs of images
The goal is to triangulate each skeleton keypoint in the same timestamp to obtain a 3D skeleton for each pair of cameras. The following image shows the indexes of the keypoints in the current skeleton model used (BODY25).
How does the triangulation works and what does it require?
Prior to the triangulation it is mandatory to perform two other operations: intrisic and extrinsic calibration. The intrisic calibration gives us the camera parameters for each camera and the extrinsic calibration gives us the transformations between pairs that will be used to triangulate. The latter is obtained from reading the calibrated xacro file with the information of each link and recalculating the transformation between the two links involved. Then we can obtain the projection matrix using both the intrinsic and extrinsic matrixes.
Assuming a point in space, X, and that we observe this point from two cameras, and we know the point's pixel coordinates in each camera. Using the projection matrix, we know that
The vectors are defined as followed
We know that the vector x⃗pix is parallel to the vector P⃗X⃗, and therefore
their cross product must be equal to zero. Knowing that we can develop this equation as follows:
By knowing that, in triangulation, the value o X must be the same in camera 1 and camera 2, we can solve the equation and obtain the value of X.
Check more details on the code here Here
Problems
As we can see in the following image, the keypoint indexes that are coming from the 2D openpose have vertically mirrored keypoint indexes. For example, the right shoulder has the index 5 in camera 3 and index 2 in camera 4, which naturally will cause that the results of the triangulation are incorrect.
Another problem is how to identify badly identifies keypoints in the 2D images. For example:
