About | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange

Configuring a Robot's Diagnostic Aggregator

Description: This tutorial will help you set up and configure a diagnostic aggregator, using the PR2 as an example. The Aggregator can be configured for a robot or type of robot to perform basic analysis on the diagnostics.

Tutorial Level: INTERMEDIATE

Next Tutorial: Creating a Diagnostic Analyzer

Diagnostic Aggregator Basics

The diagnostic_aggregator package contains the tools for collecting, categorizing, and analyzing diagnostics data from a robot. The Aggregator class subscribes to raw data on the /diagnostics topic, and publishes /diagnostics_agg at 1Hz. To process the data, the aggregator loads "diagnostic analyzers" at startup. These analyzers can examine any data item, and output processed diagnostics data.

To keep this simple, we'll only use the diagnostic_aggregator::GenericAnalyzer and diagnostic_aggregator::AnalyzerGroup analyzer types, which are both in the diagnostic_aggregator package. It's useful to know a little about the GenericAnalyzer, so you may want to look at the tutorial Using the GenericAnalyzer if you have questions.

We'll go through the process of setting up a diagnostic aggregator, or aggregator_node, for a PR2. We'll divide up the diagnostics data into different types, and set up our analyzers to categorize the raw data. We can test this using bag files of raw data and the robot_monitor.

Looking at the Raw Diagnostics

To start with, we can view the raw data from a PR2. We can grab a bag file of recorded diagnostics data, or just subscribe to an actual robot. To view it, use the runtime_monitor. All raw diagnostics is published on the /diagnostics as a diagnostic_msgs/DiagnosticArray topic.

Looking at the raw diagnostics data on a PR2, we can see that we have broad categories:

• EtherCAT devices (motors)
• Joints
• Computers
• Power System
• Sensors and Drivers
  • Hokuyos
  • IMU
  • Forearm/Dual Stereo Camera
  • Prosilica
  • Joystick

The easiest way to analyze the diagnostics is to simply group everything into categories. We'll set up an analyzer for each group or sub group above.

Setting up the Aggregator

To configure our aggregator_node, we'll start by grouping all the EtherCAT Devices together. These devices are the PR2 actuators, plus the LED projector board and EtherCAT hubs.

To get the aggregator_node to load our analyzers, we'll need to set up the analyzers in the private parameter space of the aggregator_node. Make a YAML file to store the parameters. You can make it in any package, or just use the existing "diagnostic_aggregator/demo/pr2_analyzers.yaml". This configuration file will load into the private parameter namespace of an aggregator_node.

pub_rate: 1.0 # Optional
base_path: '' # Optional, prepended to all diagnostic output
analyzers:
  motors:
    type: GenericAnalyzer
    path: Motors
    startswith: 'EtherCAT'

In the analyzers namespace, we made our first analyzer in the motors namespace. Let's look at each parameter under motors.

• type: This is mandatory for all analyzers. It tells the aggregator_node which plugin class to load.

• path: All output from this GenericAnalyzer be under this name. (Ex: "EtherCAT Device (head_pan_motor)" -> "Motors/EtherCAT Device (head_pan_motor)")

• startswith: This tells the GenericAnalyzer to analyze any component that starts with 'EtherCAT'. See the tutorial Using the GenericAnalyzer for details.

As noted above, the pub_rate and base_path parameters are optional. They default to 1.0 and "", respectively.

Using AnalyzerGroup

When we categorized the diagnostics, the sensors went under "Sensors", but in different sub categories. To use sub categories in the aggregator_node, we use the AnalyzerGroup.

To the same YAML file above, add:

  sensors:
    type: AnalyzerGroup
    path: Sensors
    analyzers:
      base_hk:
        type: GenericAnalyzer
        path: Base Hokuyo
        timeout: 5.0
        find_and_remove_prefix: base_hokuyo_node
        num_items: 3
      tilt_hk:
        type: GenericAnalyzer
        path: Tilt Hokuyo
        timeout: 5.0
        find_and_remove_prefix: tilt_hokuyo_node
        num_items: 3
      imu:
        type: GenericAnalyzer
        path: IMU
        timeout: 5.0
        find_and_remove_prefix: imu_node
        num_items: 3

This sets up an AnalyzerGroup to cover "Sensors", and moves the Hokuyo's and IMU into a sub category. An incoming message "imu_node: Connection Status" will become "Sensors/IMU/Connection Status", and be 2 levels down in the Robot Monitor.

The AnalyzerGroup will initialize "sub-analyzers" for any Analyzer specified in its analyzers namespace.

Adding Remaining Analyzers

To add more analyzers to the aggregator_node, add the parameters to YAML setup file.

  computers:
    type: GenericAnalyzer
    path: Computers
    contains: [
      'HD Temp',
      'CPU Usage',
      'CPU Temperature',
      'HD Usage',
      'NFS']
  joints:
    type: GenericAnalyzer
    path: Joints
    startswith: 'Joint'
  power:
    type: GenericAnalyzer
    path: 'Power System'
    timeout: 5.0
    startswith: [
      'Smart Battery',
      'Power board']
    expected: [
      'IBPS 1',
      'IBPS 2',
      'IBPS 3',
      'IBPS 0']

This will "fill out" the Analyzer with the categories for the power system, computers, and joints.

Launching the Aggregator

When you're done making the above YAML file, add it to a roslaunch file so it loads in the aggregator_node's namespace.

Your launch file should look something like this:

<launch>
  <node pkg="diagnostic_aggregator" type="aggregator_node"
        name="diagnostic_aggregator" >
    <!-- Load the file you made above -->
    <rosparam command="load" 
              file="$(find my_robot_package)/analyzers.yaml" />
  </node>
</launch>

Save this launch file as "aggregator.launch", and launch it.

Testing

It's important to test your newly configured diagnostic aggregator. You'll have to view it with the robot_monitor. Make sure you have the robot monitor up to date and built. See Using the Robot Monitor for details.

To test, pull a diagnostics logfile from your robot. On the PR2, these files are recorded automatically (stored in /hwlog on the robot). If your robot does not automatically record, you will have to manually record one using rosrecord. Copy the file to your desktop.

Play back and verify that the log file works using the runtime_monitor. Open three terminals.

roscore # Terminal 1
rosplay YOUR_LOGFILE # Terminal 2
rosrun runtime_monitor monitor # Terminal 3

You should see raw diagnostics data from the robot. Leave these terminals up.

Now, launch the file you just made:

roslaunch aggregator.launch

View with the robot_monitor:

rosrun robot_monitor robot_monitor

Compare the robot_monitor and the runtime_monitor. The robot_monitor should be categorized according to your configuration. Diagnostic items in error or warning state should make their parents in error or warning as well.

Shutdown your aggregator with Ctrl-C, and the robot_monitor should report stale and turn gray in a few seconds.

If you wish to alter categories, modify your YAML file and re-launch. If you're satisfied with the aggregator configuration, add the launch file to the robot's main launch file.

<include file="$(find my_pkg)/aggregator.launch" />

Existing Configurations

See pr2_bringup for existing diagnostics aggregators on robots. The file pr2_bringup/config/analyzers.yaml on the PR2 robot holds the configuration instructions, and the file pr2_bringup/pr2.launch starts the aggregator_node on startup.