Skip to main content

5.2.2 Data Display

Web Display​

Overview​

Web display is used to preview camera images (JPEG format) and algorithm results. Images and algorithm results are transmitted over the network to a PC browser for rendering. The display client also supports showing video only without rendering intelligent results.

Code repository: https://github.com/D-Robotics/hobot_websocket

Supported Platforms​

PlatformRuntime EnvironmentExample Function
RDK X3, RDK X3 ModuleUbuntu 20.04 (Foxy), Ubuntu 22.04 (Humble)Start MIPI camera and display images via Web
RDK X5, RDK X5 Module,RDK S100Ubuntu 22.04 (Humble)Start MIPI camera and display images via Web
RDK S600Ubuntu 24.04 (Jazzy)Start MIPI camera and display images via Web

Prerequisites​

RDK Platform​

  1. Confirm the F37 camera is correctly connected to RDK

  2. Confirm the PC can access RDK over the network

  3. Confirm TogetheROS.Bot has been successfully installed

Usage​

RDK Platform​

  1. Log in to RDK via SSH and start the board-side programs

    a. Start mipi_cam

    # Configure tros.b environment
    source /opt/tros/setup.bash
    # Configure tros.b environment
    source /opt/tros/humble/setup.bash
    # Configure tros.b environment
    source /opt/tros/jazzy/setup.bash
     ros2 launch mipi_cam mipi_cam.launch.py mipi_video_device:=F37

    b. Start encoding

    # Configure tros.b environment
    source /opt/tros/setup.bash
    # Configure tros.b environment
    source /opt/tros/humble/setup.bash
    # Configure tros.b environment
    source /opt/tros/jazzy/setup.bash
    ros2 launch hobot_codec hobot_codec_encode.launch.py

    c. Start websocket

    # Configure tros.b environment
    source /opt/tros/setup.bash
    # Configure tros.b environment
    source /opt/tros/humble/setup.bash
    # Configure tros.b environment
    source /opt/tros/jazzy/setup.bash
    ros2 launch websocket websocket.launch.py websocket_image_topic:=/image_jpeg websocket_only_show_image:=true

  2. Enter http://IP:8000 in a PC browser (Chrome/Firefox/Edge) to view the image, where IP is the RDK IP address.

    Web UI showing RDK live images via WebSocket in the browser

Notes​

  1. websocket requires port 8000. If the port is occupied, startup will fail. Solutions:

    • Use the lsof -i:8000 command to check which process is using port 8000, then use kill <PID> to close the process occupying port 8000, and restart websocket.

    • If you do not want to stop the service currently using port 8000, you can modify the listen port number in the /opt/tros/${TROS_DISTRO}/lib/websocket/webservice/conf/nginx.conf configuration file to a port number greater than 1024 that is not in use. After changing the port number, the URL used in the browser must also be updated accordingly.

HDMI Display​

Overview​

This section describes how to display camera NV12 images via HDMI. Connect RDK to a display via HDMI to show real-time image output, corresponding to the hobot_hdmi package.

Code repository: https://github.com/D-Robotics/hobot_hdmi

Supported Platforms​

PlatformRuntime EnvironmentExample Function
RDK X3, RDK X3 ModuleUbuntu 20.04 (Foxy), Ubuntu 22.04 (Humble)Start MIPI camera and display images via HDMI
RDK X5, RDK X5 ModuleUbuntu 22.04 (Humble)Start MIPI camera and display images via HDMI
Note

HDMI display EOL notice:

  • RDK X3 and RDK X3 Module platforms are supported up to version 2.1.0 , corresponding to TROS version 2.2.0 (2024-04-11) .
  • RDK X5 and RDK X5 Module platforms are supported up to version 2.4.2 , corresponding to TROS version 2.3.1 (2024-11-20) .

Prerequisites​

RDK Platform​

  1. RDK has been flashed with the Ubuntu system image.

  2. TogetheROS.Bot has been successfully installed on RDK.

  3. RDK is connected to a display via HDMI.

Usage​

RDK Platform​

Log in to the development board via SSH and start the board-side programs:

# Configure tros.b environment
source /opt/tros/setup.bash

When using RDK X5, run the following additional commands:

# Disable desktop display
sudo systemctl stop lightdm
# Copy runtime dependencies
cp -r /opt/tros/${TROS_DISTRO}/lib/hobot_hdmi/config/ .
# HDMI image rendering
ros2 launch hobot_hdmi hobot_hdmi.launch.py device:=F37

Result Analysis​

The terminal outputs the following information during execution:

[INFO] [launch]: All log files can be found below /root/.ros/log/2022-07-27-15-27-26-362299-ubuntu-13432
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [mipi_cam-1]: process started with pid [13434]
[INFO] [hobot_hdmi-2]: process started with pid [13436]

The display shows the image as follows:

Terminal log showing the hobot_hdmi display output node started successfully

RViz2 Display​

Overview​

TogetheROS.Bot is compatible with ROS2. To conveniently preview image output, images can be obtained through RViz2.

Supported Platforms​

PlatformRuntime Environment
RDK X3, RDK X3 ModuleUbuntu 20.04 (Foxy), Ubuntu 22.04 (Humble)
RDK X5, RDK X5 Module, RDK S100Ubuntu 22.04 (Humble)
RDK S600Ubuntu 24.04 (Jazzy)

Prerequisites​

RDK Platform​

  1. RDK has been flashed with the Ubuntu desktop system image.

  2. tros.b has been successfully installed on RDK.

  1. The PC has Ubuntu 20.04/Ubuntu 22.04, ROS2 Foxy/Humble desktop edition, and the RViz2 data visualization tool installed, and is on the same network segment as RDK (first three digits of IP address are the same).

    • ROS2 installation reference: Foxy version, Humble version

    • Install RViz2 on PC: sudo apt install ros-$ROS_DISTRO-rviz-common ros-$ROS_DISTRO-rviz-default-plugins ros-$ROS_DISTRO-rviz2 . Where $ROS_DISTRO is the ROS2 version, such as foxy or humble .

  1. The PC has Ubuntu 22.04, ROS2 Humble desktop edition, and the RViz2 data visualization tool installed, and is on the same network segment as RDK (first three digits of IP address are the same).

    • ROS2 installation reference: Humble version

    • Install RViz2 on PC: sudo apt install ros-$ROS_DISTRO-rviz-common ros-$ROS_DISTRO-rviz-default-plugins ros-$ROS_DISTRO-rviz2 . Where $ROS_DISTRO is the ROS2 version, such as humble .

  1. The PC has Ubuntu 24.04, ROS2 Jazzy desktop edition, and the RViz2 data visualization tool installed, and is on the same network segment as RDK (first three digits of IP address are the same).

    • ROS2 installation reference: Jazzy version

    • Install RViz2 on PC: sudo apt install ros-$ROS_DISTRO-rviz-common ros-$ROS_DISTRO-rviz-default-plugins ros-$ROS_DISTRO-rviz2 . Where $ROS_DISTRO is the ROS2 version, such as jazzy .

Usage​

RDK Platform​

  1. Log in to RDK via SSH and start the board-side programs

    # Configure tros.b environment
    source /opt/tros/setup.bash
    # Configure tros.b environment
    source /opt/tros/humble/setup.bash
    # Configure tros.b environment
    source /opt/tros/jazzy/setup.bash
    # Start mipi camera to publish BGR8 format images
    ros2 launch mipi_cam mipi_cam.launch.py mipi_out_format:=bgr8 mipi_image_width:=480 mipi_image_height:=272 mipi_io_method:=ros mipi_video_device:=F37

    Note: Do not change mipi_out_format arbitrarily. RViz2 only supports image formats such as RGB8, RGBA8, BGR8, and BGRA8.

    If the program outputs the following information, the node has started successfully:

    [INFO] [launch]: All log files can be found below /root/.ros/log/2022-08-19-03-53-54-778203-ubuntu-2881662
    [INFO] [launch]: Default logging verbosity is set to INFO
    [INFO] [mipi_cam-1]: process started with pid [2881781]

  2. Open a new window on RDK and query topics. Command and return result:

    # Configure tros.b environment
    source /opt/tros/setup.bash
    # Configure tros.b environment
    source /opt/tros/humble/setup.bash
    # Configure tros.b environment
    source /opt/tros/jazzy/setup.bash
    # Query topics
    ros2 topic list

    Output:

    /camera_info
    /image_raw
    /parameter_events
    /rosout

  3. Start RViz2 on RDK to subscribe to topics and preview camera data;

source /opt/tros/foxy/setup.bash
source /opt/tros/humble/setup.bash
source /opt/tros/jazzy/setup.bash
# Install RViz2
sudo apt install ros-${TROS_DISTRO}-rviz-common ros-${TROS_DISTRO}-rviz-default-plugins ros-${TROS_DISTRO}-rviz2
# Start RViz2
ros2 run rviz2 rviz2

Note: To run rviz on RDK, use tools such as MobaXterm for SSH connection, or add the "-Y" parameter when connecting via command-line SSH.

In the RViz2 interface, first click the add button, then select the published image by topic. In this example, the topic name is /image_raw, then click image:

RViz2 configuration for adding an Image display and selecting the /image_raw topic

The image output looks like this:

RViz2 visualization after successfully displaying a camera image topic

RQt Display​

Overview​

TogetheROS.Bot is compatible with ROS2 and supports previewing compressed format images through RQt, which can significantly reduce network bandwidth consumption. The example in this section starts the MIPI camera on RDK to capture images, then uses RQt on RDK for preview.

Supported Platforms​

PlatformRuntime Environment
RDK X5, RDK X5 Module, RDK S100Ubuntu 22.04 (Humble)
RDK S600Ubuntu 24.04 (Jazzy)

Prerequisites​

RDK Platform​

  1. RDK has been flashed with the Ubuntu desktop system image.

  2. tros.b has been successfully installed on RDK.

  1. The PC has Ubuntu 20.04/Ubuntu 22.04, ROS2 Foxy/Humble desktop edition, and the RQt data visualization tool installed, and is on the same network segment as RDK (first three digits of IP address are the same).

    • ROS2 installation reference: Foxy version, Humble version

    • Install rqt-image-view on PC: ros-$ROS_DISTRO-rqt-image-view ros-$ROS_DISTRO-rqt . Where $ROS_DISTRO is the ROS2 version, such as foxy or humble .

  1. The PC has Ubuntu 22.04, ROS2 Humble desktop edition, and the RQt data visualization tool installed, and is on the same network segment as RDK (first three digits of IP address are the same).

    • ROS2 installation reference: Humble version

    • Install rqt-image-view on PC: ros-$ROS_DISTRO-rqt-image-view ros-$ROS_DISTRO-rqt . Where $ROS_DISTRO is the ROS2 version, such as humble .

  1. The PC has Ubuntu 24.04, ROS2 Jazzy desktop edition, and the RQt data visualization tool installed, and is on the same network segment as RDK (first three digits of IP address are the same).

    • ROS2 installation reference: Jazzy version

    • Install rqt-image-view on PC: ros-$ROS_DISTRO-rqt-image-view ros-$ROS_DISTRO-rqt . Where $ROS_DISTRO is the ROS2 version, such as jazzy .

Usage​

RDK Platform​

  1. Log in to the RDK development board via SSH and start the mipi camera:

    # Configure tros.b environment
    source /opt/tros/setup.bash
    # Configure tros.b environment
    source /opt/tros/humble/setup.bash
    # Configure tros.b environment
    source /opt/tros/jazzy/setup.bash
    ros2 launch mipi_cam mipi_cam.launch.py mipi_image_width:=640 mipi_image_height:=480 mipi_video_device:=F37

  2. Start hobot_codec on RDK to publish compressed format images:

    # Configure tros.b environment
    source /opt/tros/setup.bash
    # Configure tros.b environment
    source /opt/tros/humble/setup.bash
    # Configure tros.b environment
    source /opt/tros/jazzy/setup.bash
    ros2 launch hobot_codec hobot_codec_encode.launch.py codec_out_format:=jpeg codec_pub_topic:=/image_raw/compressed

  3. Subscribe to topics on RDK and preview camera data;

source /opt/tros/foxy/setup.bash
source /opt/tros/humble/setup.bash
source /opt/tros/jazzy/setup.bash
# Install rqt
sudo apt install ros-${TROS_DISTRO}-rqt-image-view ros-${TROS_DISTRO}-rqt ros-${TROS_DISTRO}-compressed-image-transport
# Start rqt
ros2 run rqt_image_view rqt_image_view

Note: To run rqt on RDK, use tools such as MobaXterm for SSH connection, or add the "-Y" parameter when connecting via command-line SSH.

Select the topic /image_raw/compressed . The image output looks like this:

Image view in rqt after subscribing to /image_raw/compressed

Foxglove Display​

Overview​

Foxglove is an open-source toolkit that includes online and offline versions, designed to simplify robot system development and debugging. It provides a series of features for building robot applications.

This section mainly uses Foxglove's data recording and playback functionality: Foxglove allows recording ROS2 topic data to files for subsequent playback and analysis. This is very useful for system fault diagnosis, performance optimization, and algorithm debugging.

In the demo, we use the hobot_visualization package developed by TogetheROS to convert intelligent inference results into ROS2 rendering topic information.

Code repository: https://github.com/D-Robotics/hobot_visualization

Supported Platforms​

PlatformRuntime Environment
RDK X3, RDK X3 ModuleUbuntu 20.04 (Foxy), Ubuntu 22.04 (Humble)
RDK X5, RDK X5 Module, RDK S100Ubuntu 22.04 (Humble)
RDK S600Ubuntu 24.04 (Jazzy)

Prerequisites​

RDK Platform​

  1. Confirm the F37 camera is correctly connected to RDK

  2. Confirm the PC can access RDK over the network

  3. Confirm TogetheROS.Bot has been successfully installed

Usage​

RDK Platform​

  1. Log in to the RDK platform via SSH and start the board-side programs:
# Configure tros.b environment
source /opt/tros/setup.bash
# Configure tros.b environment
source /opt/tros/humble/setup.bash
# Configure tros.b environment
source /opt/tros/jazzy/setup.bash
export CAM_TYPE=fb
cp -r /opt/tros/${TROS_DISTRO}/lib/dnn_node_example/config/ .

ros2 launch hobot_visualization hobot_vis_render.launch.py dnn_example_config_file:=config/yolov2workconfig.json

At the same time, use SSH to log in to another terminal and record topic information on the board:

# Configure tros.b environment
source /opt/tros/setup.bash
# Configure tros.b environment
source /opt/tros/humble/setup.bash
# Configure tros.b environment
source /opt/tros/jazzy/setup.bash
# Record rosbag data, which will be generated in the current working directory
ros2 bag record -a
  1. Play rosbag data on the Foxglove online page

  1. Enter (https://foxglove.dev/studio) in a PC browser (Chrome/Firefox/Edge) to access the Foxglove website

    Foxglove Studio official website home page

PS: Registration is required for first-time use. You can register using a Google account or a third-party email.

Foxglove Studio account registration page for first-time users

  1. Enter the visualization interface

    Foxglove Studio main visualization workspace

  2. Click to select the local rosbag file

    Foxglove dialog for selecting a local rosbag file

  3. Open the layout interface. In the upper right corner of the layout interface, click Settings, select the icon, and enable marker rendering message playback

    Foxglove layout settings for enabling Marker message rendering

  4. Click Play

    Foxglove UI after Marker rendering settings are configured

  5. View the data

    Final Foxglove visualization while playing a rosbag

Notes​

  1. For Foxglove to visualize image data, ROS2 official message formats must be used with image encoding formats supported by Foxglove. For details, see (https://foxglove.dev/docs/studio/panels/image).

  2. When recording messages with rosbag, topic information from other devices may also be recorded. To ensure clean rosbag data, you can set export ROS_DOMAIN_ID=xxx , such as export ROS_DOMAIN_ID=1 .

Last updated on