Extensible interface for non-contiguous data handling

[cwpearson]

As seen here: https://github.com/cwpearson/kokkos-comm/compare/develop..experiments/non-contig-trait

I was trying to separate out non-contiguous data handling from MPI transport, and I came up with something like this:

template<..., typename NC>
KC::send(view, count, type, dest, comm, ...) {

  auto plan = NC::pre_send(view)
  for (msg in plan) {
    MPI_Send(msg.buf, msg.count, msg.type, ...)
  }
  NC::post_send(plan);

This interface supports anything you can do to a buffer as one shot: packing, transposing, splitting into multiple messages, constructing MPI datatypes...

You could use a different (compatible!) strategy at the receiver, or provide different strategies for send and recv buffers:

template<..., typename SNC, // send side
              typename RNC // recv side
>
KC::reduce(sendView, recvView, ...) {

  if (rank == root) {
    auto sendPlan = SNC::pre_send(sendView);
    auto recvPlan = RNC::pre_recv(recvView)
    for (smsg, rmsg in zip(sendPlan, recvPlan)) {
      MPI_Reduce(smsg.buf, rmsg.buf, smsg.count, smsg.type, ...)
    }
    SNC::post_send(..., sendPlan);
    RNC::post_recv(..., recvPlan);
  } else {
    auto sendPlan = SNC::pre_send(sendView);
    for (msg in sendPlan) {
      MPI_Reduce(msg.buf, nullptr, smsg.count, smsg.type, ...)
    }
    SNC::post_send(..., sendPlan);
  }

}

The basic interface would just define a few hooks

 // shared by all Non Contiguous Planner types
struct Plan {
  struct Msg {
    void * buf;
    MPI_Datatype type;
    int count;
  };

  std::vector<Msg> msgs;
};

// we provide these, or a user can write their own
template <typename ExecSpace, typename View>
struct NonContiguousPlanner {
  static Plan pre_send(...) { 
    // e.g. scratch allocations, pre-send data wrangling, configuring your experimental hardware
  }

  static Plan pre_recv(...) {
    // e.g. scratch allocations
  }

  static void post_send(..., plan) {
    // e.g. some cleanup? Most usually a no-op?
  }

  static void post_recv(..., plan) {
     // e.g. post-recv unpacking
  }
}

Wish list items that I haven't figured out yet

1. Interleaving MPI operations and datatype handling
2. As proposed the Plan object has some dynamic allocations in it, which would be a hindrance on the GPU

Another missing thing that might be useful

1. A stateful NC (all state must only exist in the plan). This is a minor limitation that could be addressed with a default argument in addition to the template parameter.

[janciesko]

This looks better than the pack_traits and packer types before. I like the idea of a Plan which to me sounds like a MPI operation plan. To what extent the Plan needs to be generic, accepting any Ctx or Plan base-type and to what extent the KokkoComm MPI API should be type specialized on those Plans could become apparent from application ports and early users.
In any case, having that option to provide alternative MPI API implementations based on custom plans sounds useful.

For now, partial specialization of the API on DefaultNonContig and expecting some Ctx or Plan with those members looks useful.