Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Diagnostics] Add diagnostics of execution time and periodicity of the controllers and controller_manager #1871

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
15b3377
add MovingAverageStatistics to ControllerSpec
saikishor Sep 25, 2024
b6eb1d4
Use `ControllerUpdateStatus` object to return trigger_update status w…
saikishor Nov 6, 2024
c25866e
add stats of execution time and the periodicity of the controllers
saikishor Nov 6, 2024
66135e6
Add periodicity stats to the controller manager
saikishor Nov 6, 2024
6531ab2
update the diagnostics of the controller manager with periodicity inf…
saikishor Nov 6, 2024
348b590
make execution time in the ControllerUpdateStatus optional for async …
saikishor Nov 7, 2024
8129cf0
adapt periodicity statistics for async controllers
saikishor Nov 7, 2024
e873706
Add controller statistics diagnostics
saikishor Nov 7, 2024
1a57fdf
use current trigger time from AsyncFunctionHandler to calculate the r…
saikishor Nov 8, 2024
eae02c1
Update the logic to use the period from ControllerUpdateStatus
saikishor Nov 8, 2024
2856a29
Only publish the diagnostics stats for the active controllers
saikishor Nov 8, 2024
be64141
fix CM periodicity stat
saikishor Nov 8, 2024
2c77713
Fix the tests by initializing the pointers on construction
saikishor Nov 8, 2024
b1ad07e
Don't add the initial periodicity upon activation
saikishor Nov 9, 2024
9e45ce0
set the first update period to zero and do not count for periodicity
saikishor Nov 9, 2024
62f5832
pass current_time to the trigger_update method
saikishor Nov 9, 2024
6e9be7d
Fix the test and add tests of the statistics
saikishor Nov 9, 2024
02eb08d
Add some reasonable checks about the execution time
saikishor Nov 9, 2024
f13289a
Add more statistics tests
saikishor Nov 9, 2024
186b5ff
Publish periodicity only if it is an async controller
saikishor Nov 10, 2024
0d5b4e8
Add an initial way to retrieve threshold parameters from paramserver
saikishor Nov 10, 2024
a9fa9ab
Don't use const reference as it is going to be changed in different t…
saikishor Nov 10, 2024
cad7a74
Update async test to check if the first period is set to zero or not
saikishor Nov 10, 2024
3f9f9eb
Update the tests to also check for the Min and Max of the periodicity
saikishor Nov 10, 2024
5969df7
Add tests to check for the async update with different rate with stats
saikishor Nov 10, 2024
9ba1f50
Add first version of the statistics information
saikishor Nov 10, 2024
3f5faad
use the direct error and std dev instead of the percentage way
saikishor Nov 10, 2024
d22e206
Change the parameter to mean_error
saikishor Nov 10, 2024
853abe7
Apply the statistics to the controller manager periodicity
saikishor Nov 10, 2024
615ccc6
Use from a variable instead of calling methods everytime
saikishor Nov 10, 2024
cb00998
add first version of the documentation
saikishor Nov 11, 2024
aee43bd
Update documentation
saikishor Nov 11, 2024
973decc
Add first version of the GPL configuration
saikishor Nov 12, 2024
8d65eb3
Use forward declaration for ParamListener
saikishor Nov 13, 2024
8d2a9c0
Use parameters from the GPL library
saikishor Nov 13, 2024
a125ba8
Add validators to the parameters
saikishor Nov 17, 2024
ccf482c
Add hardware_components_initial_state parameters to GPL
saikishor Nov 17, 2024
33d531f
Use hardware_components_initial_state parameters from the GPL directly
saikishor Nov 17, 2024
2af19ea
Add parameters_context.yaml and move parameters documentation
saikishor Nov 17, 2024
faf9c75
Fix hardware management service tests
saikishor Nov 17, 2024
91f549a
Add suggestions from code review
saikishor Dec 2, 2024
de5d617
Merge branch 'master' into add/statistics/diagnostics
saikishor Dec 2, 2024
18e67f6
Rename the variable `ok` to `successful` and add documentation
saikishor Dec 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,25 @@ struct ControllerUpdateStats
unsigned int total_triggers;
unsigned int failed_triggers;
};

/**
* Struct to store the status of the controller update method.
* The status contains information if the update was triggered successfully, the result of the
* update method and the execution duration of the update method. The status is used to provide
* feedback to the controller_manager.
* @var successful: true if the update was triggered successfully, false if not.
* @var result: return_type::OK if update is successfully, otherwise return_type::ERROR.
* @var execution_time: duration of the execution of the update method.
* @var period: period of the update method.
*/
struct ControllerUpdateStatus
{
bool successful = true;
return_type result = return_type::OK;
std::optional<std::chrono::nanoseconds> execution_time = std::nullopt;
std::optional<rclcpp::Duration> period = std::nullopt;
};

/**
* Base interface class for an controller. The interface may not be used to implement a controller.
* The class provides definitions for `ControllerInterface` and `ChainableControllerInterface`
Expand Down Expand Up @@ -175,13 +194,11 @@ class ControllerInterfaceBase : public rclcpp_lifecycle::node_interfaces::Lifecy
*
* \param[in] time The time at the start of this control loop iteration
* \param[in] period The measured time taken by the last control loop iteration
* \returns A pair with the first element being a boolean indicating if the async callback method
* was triggered and the second element being the last return value of the async function. For
* more details check the AsyncFunctionHandler implementation in `realtime_tools` package.
* \returns ControllerUpdateStatus. The status contains information if the update was triggered
* successfully, the result of the update method and the execution duration of the update method.
*/
CONTROLLER_INTERFACE_PUBLIC
std::pair<bool, return_type> trigger_update(
const rclcpp::Time & time, const rclcpp::Duration & period);
ControllerUpdateStatus trigger_update(const rclcpp::Time & time, const rclcpp::Duration & period);

CONTROLLER_INTERFACE_PUBLIC
std::shared_ptr<rclcpp_lifecycle::LifecycleNode> get_node();
Expand Down
24 changes: 21 additions & 3 deletions controller_interface/src/controller_interface_base.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -159,12 +159,14 @@ const rclcpp_lifecycle::State & ControllerInterfaceBase::get_lifecycle_state() c
return node_->get_current_state();
}

std::pair<bool, return_type> ControllerInterfaceBase::trigger_update(
ControllerUpdateStatus ControllerInterfaceBase::trigger_update(
const rclcpp::Time & time, const rclcpp::Duration & period)
{
ControllerUpdateStatus status;
trigger_stats_.total_triggers++;
if (is_async())
{
const rclcpp::Time last_trigger_time = async_handler_->get_current_callback_time();
const auto result = async_handler_->trigger_async_callback(time, period);
if (!result.first)
{
Expand All @@ -174,12 +176,28 @@ std::pair<bool, return_type> ControllerInterfaceBase::trigger_update(
"The controller missed %u update cycles out of %u total triggers.",
trigger_stats_.failed_triggers, trigger_stats_.total_triggers);
}
return result;
status.successful = result.first;
status.result = result.second;
const auto execution_time = async_handler_->get_last_execution_time();
if (execution_time.count() > 0)
{
status.execution_time = execution_time;
}
if (last_trigger_time.get_clock_type() != RCL_CLOCK_UNINITIALIZED)
{
status.period = time - last_trigger_time;
}
}
else
{
return std::make_pair(true, update(time, period));
const auto start_time = std::chrono::steady_clock::now();
status.successful = true;
status.result = update(time, period);
status.execution_time = std::chrono::duration_cast<std::chrono::nanoseconds>(
std::chrono::steady_clock::now() - start_time);
status.period = period;
}
return status;
}

std::shared_ptr<rclcpp_lifecycle::LifecycleNode> ControllerInterfaceBase::get_node()
Expand Down
11 changes: 10 additions & 1 deletion controller_manager/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ set(THIS_PACKAGE_INCLUDE_DEPENDS
rclcpp
realtime_tools
std_msgs
libstatistics_collector
generate_parameter_library
)

find_package(ament_cmake REQUIRED)
Expand All @@ -27,6 +29,10 @@ foreach(Dependency IN ITEMS ${THIS_PACKAGE_INCLUDE_DEPENDS})
find_package(${Dependency} REQUIRED)
endforeach()

generate_parameter_library(controller_manager_parameters
src/controller_manager_parameters.yaml
)

add_library(controller_manager SHARED
src/controller_manager.cpp
)
Expand All @@ -35,6 +41,9 @@ target_include_directories(controller_manager PUBLIC
$<BUILD_INTERFACE:${PROJECT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include/controller_manager>
)
target_link_libraries(controller_manager PUBLIC
controller_manager_parameters
)
ament_target_dependencies(controller_manager PUBLIC ${THIS_PACKAGE_INCLUDE_DEPENDS})

# Causes the visibility macros to use dllexport rather than dllimport,
Expand Down Expand Up @@ -236,7 +245,7 @@ install(
DESTINATION include/controller_manager
)
install(
TARGETS controller_manager
TARGETS controller_manager controller_manager_parameters
EXPORT export_controller_manager
RUNTIME DESTINATION bin
LIBRARY DESTINATION lib
Expand Down
21 changes: 21 additions & 0 deletions controller_manager/doc/parameters_context.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
hardware_components_initial_state: |
Map of parameters for controlled lifecycle management of hardware components.
The names of the components are defined as attribute of ``<ros2_control>``-tag in ``robot_description``.
Hardware components found in ``robot_description``, but without explicit state definition will be immediately activated.
Detailed explanation of each parameter is given below.
The full structure of the map is given in the following example:

.. code-block:: yaml

hardware_components_initial_state:
unconfigured:
- "arm1"
- "arm2"
inactive:
- "base3"

diagnostics.threshold.controllers.periodicity: |
The ``periodicity`` diagnostics will be published only for the asynchronous controllers, because any affect to the synchronous controllers will be reflected directly in the controller manager's periodicity.

diagnostics.threshold.controllers.execution_time: |
The ``execution_time`` diagnostics will be published for all controllers. The ``mean_error`` for a synchronous controller will be computed against zero, as it should be as low as possible. However, the ``mean_error`` for an asynchronous controller will be computed against the controller's desired update period, as the controller can take a maximum of the desired period cycle to execute it's update cycle.
36 changes: 10 additions & 26 deletions controller_manager/doc/userdoc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,32 +57,6 @@ robot_description [std_msgs::msg::String]
Parameters
-----------

hardware_components_initial_state
Map of parameters for controlled lifecycle management of hardware components.
The names of the components are defined as attribute of ``<ros2_control>``-tag in ``robot_description``.
Hardware components found in ``robot_description``, but without explicit state definition will be immediately activated.
Detailed explanation of each parameter is given below.
The full structure of the map is given in the following example:

.. code-block:: yaml

hardware_components_initial_state:
unconfigured:
- "arm1"
- "arm2"
inactive:
- "base3"

hardware_components_initial_state.unconfigured (optional; list<string>; default: empty)
Defines which hardware components will be only loaded immediately when controller manager is started.

hardware_components_initial_state.inactive (optional; list<string>; default: empty)
Defines which hardware components will be configured immediately when controller manager is started.

update_rate (mandatory; integer)
The frequency of controller manager's real-time update loop.
This loop reads states from hardware, updates controller and writes commands to hardware.

<controller_name>.type
Name of a plugin exported using ``pluginlib`` for a controller.
This is a class from which controller's instance with name "``controller_name``" is created.
Expand All @@ -99,6 +73,16 @@ update_rate (mandatory; integer)
The fallback controllers activation is subject to the availability of the state and command interfaces at the time of activation.
It is recommended to test the fallback strategy in simulation before deploying it on the real robot.

.. generate_parameter_library_details::
../src/controller_manager_parameters.yaml
parameters_context.yaml

**An example parameter file:**

.. generate_parameter_library_default::
../src/controller_manager_parameters.yaml


Handling Multiple Controller Managers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@

namespace controller_manager
{
class ParamListener;
class Params;
using ControllersListIterator = std::vector<controller_manager::ControllerSpec>::const_iterator;

CONTROLLER_MANAGER_PUBLIC rclcpp::NodeOptions get_cm_node_options();
Expand Down Expand Up @@ -346,7 +348,7 @@ class ControllerManager : public rclcpp::Node

// Per controller update rate support
unsigned int update_loop_counter_ = 0;
unsigned int update_rate_ = 100;
unsigned int update_rate_;
std::vector<std::vector<std::string>> chained_controllers_configuration_;

std::unique_ptr<hardware_interface::ResourceManager> resource_manager_;
Expand All @@ -357,6 +359,8 @@ class ControllerManager : public rclcpp::Node
const std::string & command_interface);
void init_controller_manager();

void initialize_parameters();

/**
* Clear request lists used when switching controllers. The lists are shared between "callback"
* and "control loop" threads.
Expand Down Expand Up @@ -473,6 +477,8 @@ class ControllerManager : public rclcpp::Node
*/
rclcpp::NodeOptions determine_controller_node_options(const ControllerSpec & controller) const;

std::shared_ptr<controller_manager::ParamListener> cm_param_listener_;
std::shared_ptr<controller_manager::Params> params_;
diagnostic_updater::Updater diagnostics_updater_;

std::shared_ptr<rclcpp::Executor> executor_;
Expand Down Expand Up @@ -603,6 +609,8 @@ class ControllerManager : public rclcpp::Node
rclcpp::Subscription<std_msgs::msg::String>::SharedPtr robot_description_subscription_;
rclcpp::TimerBase::SharedPtr robot_description_notification_timer_;

controller_manager::MovingAverageStatistics periodicity_stats_;

struct SwitchParams
{
void reset()
Expand Down
13 changes: 13 additions & 0 deletions controller_manager/include/controller_manager/controller_spec.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,13 @@
#include <vector>
#include "controller_interface/controller_interface_base.hpp"
#include "hardware_interface/controller_info.hpp"
#include "libstatistics_collector/moving_average_statistics/moving_average.hpp"

namespace controller_manager
{

using MovingAverageStatistics =
libstatistics_collector::moving_average_statistics::MovingAverageStatistics;
/// Controller Specification
/**
* This struct contains both a pointer to a given controller, \ref c, as well
Expand All @@ -35,9 +39,18 @@ namespace controller_manager
*/
struct ControllerSpec
{
ControllerSpec()
{
last_update_cycle_time = std::make_shared<rclcpp::Time>(0, 0, RCL_CLOCK_UNINITIALIZED);
execution_time_statistics = std::make_shared<MovingAverageStatistics>();
periodicity_statistics = std::make_shared<MovingAverageStatistics>();
}

hardware_interface::ControllerInfo info;
controller_interface::ControllerInterfaceBaseSharedPtr c;
std::shared_ptr<rclcpp::Time> last_update_cycle_time;
std::shared_ptr<MovingAverageStatistics> execution_time_statistics;
std::shared_ptr<MovingAverageStatistics> periodicity_statistics;
};

struct ControllerChainSpec
Expand Down
2 changes: 2 additions & 0 deletions controller_manager/package.xml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@
<depend>ros2param</depend>
<depend>ros2run</depend>
<depend>std_msgs</depend>
<depend>libstatistics_collector</depend>
<depend>generate_parameter_library</depend>

<test_depend>ament_cmake_gmock</test_depend>
<test_depend>ament_cmake_pytest</test_depend>
Expand Down
Loading
Loading