Skip to main content

Stereo Depth Algorithm

Function Introduction​

The Digua stereo depth estimation algorithm takes stereo image data as input and outputs a disparity map and a depth map corresponding to the left view. The algorithm draws inspiration from the IGEV network and employs a GRU architecture, offering good data generalization and high inference efficiency.

Stereo algorithm repository: https://github.com/D-Robotics/hobot_stereonet

MIPI camera repository: https://github.com/D-Robotics/hobot_mipi_cam

ZED camera repository: https://github.com/D-Robotics/hobot_zed_cam

Stereo algorithm explanation: Live Recording | AI Stereo Algorithm Deployment on RDK X5

Supported Platforms​

PlatformOS SupportExample Function
RDK X5, RDK X5 ModuleUbuntu 22.04 (Humble)Start stereo camera, infer depth results, and display on the web
RDK S100, RDK S100PUbuntu 22.04 (Humble)Start stereo camera, infer depth results, and display on the web

Algorithm Version Information​

The following versions of the stereo algorithm are currently available:

Algorithm VersionFeatures
X5 V2.0int16 quantization, high precision, low frame rate, max 15FPS depth map at 640*352 resolution
X5 V2.1int16 quantization, with confidence output, max 15FPS depth map at 640*352 resolution
X5 V2.2int8 quantization, lower precision, higher frame rate, max 23FPS depth map at 640*352 resolution
X5 V2.3int8 quantization, further improved frame rate, max 27FPS depth map at 640*352 resolution
X5 V2.4 int16int16 quantization, trained with more data, max 15FPS depth map at 640*352 resolution
X5 V2.4 int8V2.4 int8 quantization version, max 23FPS depth map at 640*352 resolution
S100 V2.1int16 quantization, with confidence output, max 53FPS depth map at 640*352 resolution
S100 V2.4int16 quantization, with confidence output, trained with more data, max 53FPS depth map at 640*352 resolution

Preparation​

RDK Platform​

  1. RDK has been flashed with Ubuntu 22.04 system image.
  2. TogetheROS.Bot has been successfully installed on RDK.
  3. If real-time online inference is required, please prepare a stereo camera. Currently, the official MIPI camera and ZED mini/2i USB camera are supported.
  4. If offline inference is required, please prepare rectified stereo image data.
  5. Ensure that the PC can access RDK over the network.

System and Package Versions​

VersionQuery Method
RDK X5 System Image Version3.3.1 or highercat /etc/version
RDK S100 System Image Version4.0.2-Beta or highercat /etc/version
tros-humble-hobot-stereonet Package Version2.4.4 or higherapt list | grep tros-humble-hobot-stereonet/
tros-humble-mipi-cam Package Version2.3.11 or higherapt list | grep tros-humble-mipi-cam/
tros-humble-hobot-zed-cam Package Version2.3.3 or higherapt list | grep tros-humble-hobot-zed-cam/
  • If the system version does not meet the requirements, please refer to the corresponding documentation section for image flashing.
  • If the package version does not meet the requirements, please execute the following commands to update:
sudo apt update
sudo apt upgrade
Note

If the sudo apt update command fails or reports errors, please refer to FAQ section Q10: How to handle failures or errors with the apt update command? for solutions.

Usage Instructions​

Note

Please execute the commands in this document as the root user. Other users may encounter permission issues, leading to unnecessary errors.

os_user

The stereo depth algorithm supports multiple cameras, and the startup commands vary. The specific startup commands are as follows:

(1) Startup with RDK Official 230AI MIPI Stereo Camera​

  • The RDK official MIPI stereo camera is shown below:

RDK_Stereo_Cam_230ai

Note: Please check the silkscreen on the back of the camera for CDPxxx-V3 to confirm that the camera is version V3.

  • The installation method is shown below. Ensure the wiring is correct to avoid swapping left and right images, which can cause errors in the stereo algorithm:

RDK_X5_230ai

RDK_S100_230ai

  • Confirm that the camera is connected properly by connecting to RDK via SSH and executing the following command. If addresses such as 0x30, 0x32, and 0x50 are output, the camera is connected properly:
# RDK X5
i2cdetect -r -y 4
i2cdetect -r -y 6

# RDK S100
i2cdetect -r -y 1
i2cdetect -r -y 2

i2cdetect_230ai

  • Start the corresponding version of the stereo algorithm using different launch files. Connect to RDK X5 via SSH and execute the following command:
# Only supports RDK X5

# Configure tros.b humble environment
source /opt/tros/humble/setup.bash

# Start the stereo model launch file, which includes the startup of the algorithm and stereo camera nodes
ros2 launch hobot_stereonet stereonet_model_web_visual_v2.0.launch.py \
mipi_image_width:=640 mipi_image_height:=352 mipi_lpwm_enable:=True mipi_image_framerate:=30.0 \
need_rectify:=False height_min:=-10.0 height_max:=10.0 pc_max_depth:=5.0

Parameter descriptions:

NameValueDescription
mipi_image_widthSet to 640MIPI camera output resolution is 640*352
mipi_image_heightSet to 352MIPI camera output resolution is 640*352
mipi_lpwm_enableSet to TrueMIPI camera enables hardware synchronization
mipi_image_framerateSet to 30.0MIPI camera output frame rate is 30.0FPS
need_rectifySet to FalseSince the official camera comes with factory calibration parameters and automatically rectifies, there is no need to load custom calibration files for rectification
height_minSet to -10.0Minimum point cloud height is -10.0m
height_maxSet to 10.0Maximum point cloud height is 10.0m
pc_max_depthSet to 5.0Maximum point cloud distance is 5.0m
uncertainty_thSet to 0.1Confidence threshold, used to filter noise points. It is recommended to set it to 0.1. A lower confidence threshold results in stricter filtering.
  • The following log indicates that the stereo algorithm has started successfully. fx/fy/cx/cy/base_line are the camera intrinsic parameters. If the depth map is normal but the estimated distance is incorrect, there may be an issue with the camera intrinsic parameters:

stereonet_run_success_log

  • View the depth map via the web interface by entering http://ip:8000 in the browser (the RDK IP in the image is 192.168.1.100):

web_depth_visual

  • View the point cloud via rviz2. RDK can directly install rviz2 for viewing. Note the following configurations in rviz2:
# Install rviz2
sudo apt install ros-humble-rviz2
# Start rviz2
source /opt/tros/humble/setup.bash
rviz2

stereonet_rviz

  • If the user wants to save the depth estimation results, the following parameters can be added to enable saving. save_image_all enables saving, save_freq controls the saving frequency, save_dir specifies the save directory (automatically created if it does not exist), and save_total controls the total number of saves. The program will save camera intrinsic parameters, left and right images, disparity map, depth map, and visualization map:
# Configure tros.b humble environment
source /opt/tros/humble/setup.bash

# Here, the V2.0 version of the algorithm is used as an example. Similar parameters can be added to other versions of the algorithm.
ros2 launch hobot_stereonet stereonet_model_web_visual_v2.0.launch.py \
mipi_image_width:=640 mipi_image_height:=352 mipi_lpwm_enable:=True mipi_image_framerate:=30.0 \
need_rectify:=False height_min:=-10.0 height_max:=10.0 pc_max_depth:=5.0 \
save_image_all:=True save_freq:=4 save_dir:=./online_result save_total:=10

Parameter descriptions:

NameValueDescription
save_image_allSet to TrueEnable image saving
save_freqSet to 4Save every 4 frames. Can be modified to any positive number.
save_dirSet to the image save directoryCan be set to the desired save location.
save_totalSet to 10Save a total of 10 images. Set to -1 to save continuously.

stereonet_save_log

stereonet_save_files

(2) Offline Image Replay on Local Machine​

  • If you want to evaluate the algorithm's performance using local images, you can use the following command to specify the algorithm's operating mode, image data path, and camera intrinsic parameters. Ensure that the image data has been rectified and epipolar aligned. The image format is shown below. The first left-eye image is named left000000.png, the second left-eye image is named left000001.png, and so on. The corresponding first right-eye image is named right000000.png, the second right-eye image is named right000001.png, and so on. The algorithm traverses the images in sequence until all images have been processed:

stereonet_rdk

  • The offline operation mode of the algorithm is as follows. Refer to the online startup command above, remove the mipi-related parameters from the online startup command, add use_local_image=True to enable offline inference, set local_image_path to the offline image directory, set camera_fx/camera_fy/camera_cx/camera_cy/base_line to the rectified camera parameters, set save_image_all:=True to enable saving results, and set save_dir to control the save directory (automatically created if it does not exist). Note: The replayed images must be epipolar rectified, and correct camera parameters must be set; otherwise, the saved results may be incorrect. 
# Configure tros.b humble environment
source /opt/tros/humble/setup.bash

# Here, the V2.0 version of the algorithm is used as an example. Similar parameters can be added to other versions of the algorithm.
ros2 launch hobot_stereonet stereonet_model_web_visual_v2.0.launch.py \
need_rectify:=False height_min:=-10.0 height_max:=10.0 pc_max_depth:=5.0 \
use_local_image:=True local_image_path:=./online_result \
camera_fx:=216.696533 camera_fy:=216.696533 camera_cx:=335.313477 camera_cy:=182.961578 base_line:=0.070943 \
save_image_all:=True save_dir:=./offline_result

stereonet_offline_log

Parameter descriptions:

NameValueDescription
use_local_imageSet to TrueEnable image replay mode
local_image_pathSet to the offline data directoryDirectory of replayed images
need_rectifySet to FalseReplayed images must be epipolar rectified, so this switch does not need to be enabled, but rectified parameters must be manually passed in
camera_fxSet to the rectified camera intrinsic parameter fxCamera intrinsic parameters
camera_fySet to the rectified camera intrinsic parameter fyCamera intrinsic parameters
camera_cxSet to the rectified camera intrinsic parameter cxCamera intrinsic parameters
camera_cySet to the rectified camera intrinsic parameter cyCamera intrinsic parameters
base_lineSet to the rectified camera baselineBaseline distance in meters
height_minSet to -10.0Minimum point cloud height is -10.0m
height_maxSet to 10.0Maximum point cloud height is 10.0m
pc_max_depthSet to 5.0Maximum point cloud distance is 5.0m
save_image_allSet to TrueSave replayed results
save_dirSet to the image save directoryCan be set to the desired save location.
  • After the algorithm runs successfully, the result data can also be displayed via the web interface and rviz2. Refer to the corresponding settings above.

(3) Startup with ZED Stereo Camera​

  • The ZED stereo camera is shown below:

zed_cam

  • Connect the ZED camera to RDK via USB, then start the stereo algorithm. Connect to RDK via SSH and execute the following command:

Note: RDK must be connected to the internet when running the ZED camera because ZED needs to download calibration files online. 

# Only supports RDK X5

# Configure tros.b humble environment
source /opt/tros/humble/setup.bash

# Start the stereo model launch file, which includes the startup of the algorithm and stereo camera nodes
ros2 launch hobot_stereonet stereonet_model_web_visual_zed_v2.0.launch.py use_mipi_cam:=False \
height_min:=-10.0 height_max:=10.0 pc_max_depth:=5.0 uncertainty_th:=0.10

stereonet_zed_run_success_log

When connected to the internet, the program will automatically download the calibration file. If RDK is not connected to the internet, you can manually download the calibration file and upload it to RDK: According to the log information, open a browser on your PC and enter (https://calib.stereolabs.com/?SN=38085162) to download the calibration file SN38085162.conf. Note that each ZED camera has a different SN code. Use the calibration file corresponding to the error message提示. Upload the calibration file to the /root/zed/settings/ directory on RDK. If the directory does not exist, create one.

  • View the depth map via the web interface by entering http://ip:8000 in the browser (ip is the corresponding IP address of RDK). To view the point cloud and save images, refer to the corresponding settings above.

hobot_stereonet Package Description​

Subscribed Topics​

NameMessage TypeDescription
/image_combine_rawsensor_msgs::msg::ImageTopic for the combined left and right images published by the stereo camera node, used for model inference to generate depth
/camera_info_topicsensor_msgs::msg::CameraInfoTopic for the camera parameters published by the stereo camera node, used for converting between disparity maps and depth maps

Published Topics​

NameMessage TypeDescription
/StereoNetNode/stereonet_depthsensor_msgs::msg::ImagePublished depth image, with pixel values representing depth in millimeters
/StereoNetNode/stereonet_visualsensor_msgs::msg::ImagePublished intuitive visualization rendering image
/StereoNetNode/stereonet_pointcloud2sensor_msgs::msg::PointCloud2Published point cloud depth topic

Other Important Parameters​

NameValueDescription
stereo_image_topicDefault /image_combine_rawTopic name for subscribing to stereo image messages
camera_info_topicDefault /image_right_raw/camera_infoTopic name for subscribing to camera calibration parameter messages
need_rectifyDefault TrueWhether to specify a custom calibration file for image rectification
stereo_calib_file_pathDefault stereo.yamlWhen need_rectify=True, load the calibration file from this path for calibration
stereo_combine_modeDefault 1Left and right images are often combined into one image and then published. 1 indicates vertical combination, 0 indicates horizontal combination, indicating how the stereo algorithm splits the image
KMeanDefault 10Number of neighboring points for each point when filtering sparse outliers. The distances to the nearest 10 points are statistically analyzed.
stdvDefault 0.01Threshold for determining outliers when filtering sparse outliers. The standard deviation multiplier is set to 0.01.
leaf_sizeDefault 0.05Sets the unit density of the point cloud, indicating that there is only one point within a 3D sphere with a radius of 0.05 meters.
Last updated on