Skip to main content

5.4.4 Robot Human Following

Overview​

The robot human following App controls the robot to follow a person. The App consists of MIPI image capture, human detection and tracking, human following strategy, image encoding/decoding, and a Web display client. The workflow is shown below:

The App uses a virtual robot in the PC-side Gazebo simulation environment as an example. The published control commands can also be used directly to control a physical robot.

Code repository: (https://github.com/D-Robotics/body_tracking)

Supported Platforms​

PlatformRuntime Environment
RDK X3, RDK X3 ModuleUbuntu 20.04 (Foxy), Ubuntu 22.04 (Humble)
RDK X5, RDK X5 ModuleUbuntu 22.04 (Humble)

Preparation​

RDK Platform​

  1. The RDK has been flashed with the Ubuntu system image.

  2. TogetheROS.Bot has been successfully installed on the RDK.

  3. A MIPI or USB camera has been installed on the RDK.

  4. A PC on the same network as the RDK (wired or on the same Wi-Fi, with the first three octets of the IP address matching). The PC requires the following environment:

sudo apt-get install ros-foxy-gazebo-*
sudo apt install ros-foxy-turtlebot3
sudo apt install ros-foxy-turtlebot3-simulations

Usage​

RDK Platform​

After running the robot human following App, the robot motion control package selects the person closest to the front of the robot (the person with the largest detection box width) as the follow target. When the person is far from the robot, the robot moves forward to approach the person and keeps the person directly in front of the robot.

After the App starts, images published by the sensor and corresponding algorithm results can be rendered and displayed in a PC browser (enter http://IP:8000 in the browser, where IP is the RDK's IP address).

Start the simulation environment on the PC:

source /opt/ros/foxy/setup.bash
export TURTLEBOT3_MODEL=burger
ros2 launch turtlebot3_gazebo empty_world.launch.py

After successful startup, the robot in the simulation environment appears as follows:

Publish images using MIPI camera

# 配置tros.b环境
source /opt/tros/setup.bash
# 从TogetheROS的安装路径中拷贝出运行示例需要的配置文件。
cp -r /opt/tros/${TROS_DISTRO}/lib/mono2d_body_detection/config/ .

# 配置MIPI摄像头
export CAM_TYPE=mipi

# 启动launch文件
ros2 launch body_tracking body_tracking_without_gesture.launch.py

Publish images using USB camera

# 配置tros.b环境
source /opt/tros/setup.bash
# 从tros.b的安装路径中拷贝出运行示例需要的配置文件。
cp -r /opt/tros/${TROS_DISTRO}/lib/mono2d_body_detection/config/ .

# 配置USB摄像头
export CAM_TYPE=usb

# 启动launch文件
ros2 launch body_tracking body_tracking_without_gesture.launch.py

Result Analysis​

The RDK terminal outputs the following information:

[body_tracking-7] [WARN] [1653430533.523069034] [ParametersClass]: TrackCfg param are
[body_tracking-7] activate_wakeup_gesture: 0
[body_tracking-7] track_serial_lost_num_thr: 100
[body_tracking-7] activate_robot_rotate_thr: 45
[body_tracking-7] activate_robot_move_thr: 5
[body_tracking-7] move_step: 0.3
[body_tracking-7] rotate_step: 0.5
[body_tracking-7] img_width: 960
[body_tracking-7] img_height: 544
[body_tracking-7] 
[body_tracking-7] [WARN] [1653430533.712812076] [TrackingManager]: update frame_ts 395787, 873
[body_tracking-7] [WARN] [1653430533.713105576] [TrackingManager]: Tracking body start!, track_id: 1, frame_ts: 395787, tracking_sta(0:INITING, 1:TRACKING, 2:LOST): 1, gesture: 0
[body_tracking-7] [WARN] [1653430535.018442618] [TrackingManager]: Do move! body_rect_width: 353, thr: 864, move_step_ratio: 1, body_rect_to_top: 20, img_height: 544, move_step: 0.3
[body_tracking-7] [WARN] [1653430535.220268535] [TrackingManager]: Do rotate move, ts sec: 3397, nanosec: 387800000
[body_tracking-7] [WARN] [1653430535.220408576] [RobotCmdVelNode]: RobotCtl, angular: 0 0 0, linear: 0.3 0 0, pub twist ts: 1653430535220394

The log above shows a segment of output after the App starts. After startup, related configuration (TrackCfg param) is printed first. Once a person is detected, the robot enters the following state (tracking_sta value is 1) and moves forward at 0.3 m/s (RobotCtl, angular: 0 0 0, linear: 0.3 0 0) to approach the person.

Use the ros2 topic list command on the PC terminal to query RDK topic information:

$ ros2 topic list
/camera_info
/cmd_vel
/hbmem_img04054242060426080500012020112713
/hobot_mono2d_body_detection
/image
/parameter_events
/rosout

/image is the JPEG-encoded image published by the RDK after capturing from the MIPI sensor. /hobot_mono2d_body_detection is the algorithm message published by the RDK containing human detection results. /cmd_vel is the motion control command published by the RDK.

Use the ros2 topic echo /cmd_vel command on the PC terminal to view motion control commands published by the RDK:

linear:
  x: 0.5
  y: 0.0
  z: 0.0
angular:
  x: 0.0
  y: 0.0
  z: -0.5
---
linear:
  x: 0.5
  y: 0.0
  z: 0.0
angular:
  x: 0.0
  y: 0.0
  z: -0.5
---

In the PC simulation environment, the robot follows the person. The simulation result is shown below:

Last updated on