Grid Loadbalancer

Overview

The Cabana::Grid LoadBalancer generates a new GlobalGrid with better balanced domain boundaries.

This is done using the ALL library, which must be available via CMake to use these features. The class is only available if the CMake flag Cabana_ENABLE_ALL is set, which can be required by setting Cabana_REQUIRE_ALL.

The class is in use for the ExaMPM and CabanaMD mini apps. A more hands on example is present there. More details, albeit less close to the actual (complex) use case, is available in this repository under the grid tutorial section.

In general, the library and class require a scalar work from each rank that is used as the metric for balancing.

After the creation of the new grid, the local variables and all classes that depend on it, must be updated or recreated.

The domain boundaries will always be aligned to the nearest grid cell boundary. For a molecular dynamics simulation code such as CabanaMD, this is not needed because there is no inherent grid. In this case, the cell size of the global grid can be chosen arbitrarily small. No arrays are allocated based on this cell size by the load balancer or the other required methods and classes. A more detailed implementation can be found in CabanaMD.

In some cases a minimum domain width is required, so that communication only occurs between neighbors. This can be set to the same value for all dimensions, or for each dimensions separately as an optional argument to the constructor.

The used domain boundaries are always aligned to the underlying grid. However, ALL works on continuous boundaries. The new balanced coordinates are then rounded to the nearest cell boundary. In cases of small load imbalances, this can cause no change for the aligned boundaries. To allow small imbalances to be balanced, the domain boundaries used for load balancing are continuous and can change over several balance steps until the aligned boundaries change as well. These continuous boundaries are called internal boundaries. They are given to the library during balancing.

The load balancer has a few additional methods, that are only useful in some cases. The vertices and internal vertices can be retrieved with getVertices and getInternalVertices respectively. The current load imbalance as defined by (wmax-wmin)/(wmax+wmin) with the maximum and minimum work across all ranks is returned by getImbalance. Note, this method causes all to all communication.

Implementation

Header Files: Cabana_Grid_LoadBalancer.hpp

Interface

template <class MeshType>
class LoadBalancer

Examples

A very basic framework is:

auto _global_mesh = Cabana::Grid::createUniformGlobalMesh(global_low_corner, global_high_corner, num_cell);
auto _global_grid = Cabana::Grid::createGlobalGrid(comm, _global_mesh, periodic, partitioner);
auto _lb = Cabana::Grid::Experimental::createLoadBalancer(_comm, _global_grid, min_domain_width);
for(int i=0; i<10; ++i)
{
    double work = /* Measure of work, e.g.,  number of particles */
    _global_grid = _lb->createBalancedGrid(*_global_mesh, *_partitioner, work);
    /* Update everything depending on the global grid */
}

A more in depth example is available in the grid tutorial section here.

In the ExaMPM mini app the load balancer is called from the Solver class. (TODO PR #29) The optional load balancer calls are in ExaMPM_ENABLE_LB preprocessor guards.

Similarly, it is called from the CbnMD class in CabanaMD. (TODO PR #104) Look for CabanaMD_ENABLE_LB to find the optional load balancer calls.

The main changes in the mini apps from the framework shown above, are logistical or for clarity. In ExaMPM the update of the dependents of the global grid is done in the ProblemManager class via updateMesh, since it contains most of those. Hence, the main loop is still clear to read and not cluttered with a lot of detailed reallocations or recreations of classes.

This is part of the Programming Guide series