June 2022

Calibration of RGB-D cameras and hand-eye system development

Calibrating different brands of RGB-D cameras

First I calibrated a simulated system with only an RGB-D camera. This system was calibrated with an induced error of 0.1m and 0.1rad. As can be seen in the image above, the calibration was successful and the 3D models of the depth and RGB sensors overlap. The reprojection error of this calibration was 2.43 pixels.

In real world, I tested this calibration in three different camera models: Asus Xtion, Obrecc Astra Pro and Intel Realsense. First, in the orbecc calibration the models of both the sensors had a significant displacement as seen in the image, which led me to think that there might be something wrong in the camera driver since the methodology was proved to be right in simulation. I did not do a reprojection error analysis on this camera.

The camera that inspired me the most trust was the Asus Xtion so I decided to calibrate it. The reprojection error obtained was 3.55 pixels, which is quite good for real world. Nevertheless, the tridimensional models still present a sligh displacement and they should be overlapped. Unlike the case of the orbecc, this time the displacement is mainly vertical.

Thirdly I tested on the Intel Realsense camera. This was the one that we had the most interest because it is installed on the hand-eye setup. After recording the data I concluded that it was not possible to calibrate these sensors because the data is highly inconsistent and inaccurate, as can be seen in the images of the chessboard detection bellow.

Here is an image of the Asus Xtion detection for comparison:

This data inconsistency means that the realsense depth sensor cannot be calibrated and therefore we must find a different solution for the RGB-D hand-eye calibration.

Calibrating a RGB-D hand-eye system combined with fixed sensors

The system to calibrate has 5 sensors: 1 LiDAR, 2 RGBs and 2 Depths. The following image shows the setup that will be used.

I calibrated this system in a simulated environment with 24 collections and an induced error of 0.1m and 0.1 rad and obtained good visual results. I still haven't proceeded to the reprojections analysis.

I also recorded real data for this system but I used the installed realsense camera, which is now proved to bu unfit for calibration so new data must be acquired with a different camera on the hand-eye system.

Other tasks

• Review and submission of JMS article - first review
• Development of workplan for the 3 months in Barcelona
• Organization of new manuscript for depth calibration
• Writing the introduction
• State-of-the-art about RGB-D and hand-eye systems

Issues

Click mouse to indicate seed point in depth labeling does not entirely work - open

Evaluate if Realsense images are not fit for depth labelling - open

Pyrdown causing error increase in depth modality - open

Images don't appear while calibration due to wrong configuration of rviz template - closed

Possible depth labelling problem with robot - open