Skip to main content

BEV Perception Algorithm

Feature Introduction​

The BEV perception algorithm is a multi-task BEV model trained on the nuscenes dataset using OpenExplorer.

The algorithm takes six groups of image data as input: front, front-left, front-right, rear, rear-left, and rear-right views. The model outputs detection results for ten object categories along with their corresponding 3D bounding boxes—including obstacles, various vehicle types, traffic signs, etc.—as well as semantic segmentation results for lane markings, sidewalks, and road edges.

This example uses local image data as input, performs inference on the BPU, publishes rendered images containing perception results, and displays these results in a web browser on a PC.

Code repository: (https://github.com/D-Robotics/hobot_bev.git)

Supported Platforms​

PlatformRuntime EnvironmentExample Functionality
RDK UltraUbuntu 20.04 (Foxy)Uses local data replay and displays inference results via web
RDK S100, RDK S100PUbuntu 22.04 (Humble)Uses local data replay and displays inference results via web

Prerequisites​

  1. The RDK has been flashed with an Ubuntu 20.04 or Ubuntu 22.04 system image.
  2. TogetheROS.Bot has been successfully installed on the RDK.
  3. Ensure your PC can access the RDK over the network.

Usage Instructions​

Using Local Dataset Replay​

Local dataset replay is used to feed images into the algorithm. After inference, the algorithm publishes rendered image messages containing perception results. These are displayed in a web browser on the PC via the websocket package.

Preparing the Replay Dataset

# Download dataset on the board
wget http://archive.d-robotics.cc/TogetheROS/data/hobot_bev_data.tar.gz

# Extract the archive
mkdir -p hobot_bev_data
tar -zxvf hobot_bev_data.tar.gz -C hobot_bev_data

# After extraction, the dataset resides under hobot_bev_data/data

Replaying the Dataset

# Set up the tros.b environment
source /opt/tros/setup.bash

# Launch the websocket service
ros2 launch websocket websocket_service.launch.py

# Launch the execution script and specify the dataset path
ros2 launch hobot_bev hobot_bev.launch.py image_pre_path:=hobot_bev_data/data

Result Analysis​

The following output appears in the terminal during execution:

[INFO] [launch]: All log files can be found below /root/.ros/log/2023-07-05-17-47-07-232907-hobot-2627970
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [hobot_bev-1]: process started with pid [2627972]
[INFO] [websocket-2]: process started with pid [2627974]
[hobot_bev-1] [WARN] [1688579227.907268364] [bev_node]:
[hobot_bev-1]  image_pre_path: hobot_bev_data/data
[hobot_bev-1] [BPU_PLAT]BPU Platform Version(1.3.3)!
[hobot_bev-1] [HBRT] set log level as 0. version = 3.14.25.0
[hobot_bev-1] [DNN] Runtime version = 1.12.3_(3.14.25 HBRT)
[hobot_bev-1] [WARN] [1688579228.714778531] [dnn]: Run default SetOutputParser.
[hobot_bev-1] [WARN] [1688579228.714925489] [dnn]: Set output parser with default dnn node parser, you will get all output tensors and should parse output_tensors in PostProcess.
[hobot_bev-1] [WARN] [1688579228.886846489] [bev_node]: loop 0/1002
[hobot_bev-1] [WARN] [1688579229.474568573] [bev_node]: loop 1/1002
[hobot_bev-1] [WARN] [1688579230.058551781] [bev_node]: loop 2/1002
[hobot_bev-1] [WARN] [1688579230.691667198] [bev_node]: loop 3/1002
[hobot_bev-1] [WARN] [1688579231.324658782] [bev_node]: loop 4/1002
[hobot_bev-1] [WARN] [1688579231.365145532] [bev_node]: input fps: 2.47, out fps: 2.52, infer time ms: 12, post process time ms: 659
[hobot_bev-1] [WARN] [1688579231.915645741] [bev_node]: loop 5/1002
[hobot_bev-1] [WARN] [1688579231.996993824] [bev_node]: input fps: 2.47, out fps: 2.52, infer time ms: 12, post process time ms: 658

Enter http://IP:8000 in your PC's web browser to view the rendered images and algorithm results (replace IP with the RDK's IP address):

Last updated on