FLTUND in KF in physmon at PointwiseMaterialInteraction::covarianceContributions

[AJPfleger]

Problem

I tried to run the KF with the current physmon settings but with 100k events. This leads on the gitlab CI to an FPE of type FLTUND. This happend in the call

Traceback (most recent call last):
  File "/builds/acts/ci-bridge/src/CI/physmon/workflows/physmon_trackfitting_gx2f_vs_kf.py", line 38, in <module>
    s.run()
RuntimeError: Sequencer terminated abnormally
::error::🟥 Dataset generation failed: CI/physmon/workflows/physmon_trackfitting_gx2f_vs_kf.py -> ec=1

https://gitlab.cern.ch/acts/ci-bridge/-/jobs/46363712

Stack trace

11:24:40    Sequencer      INFO      52900 / 100000 events processed
11:24:40    Sequencer      INFO      53000 / 100000 events processed
11:24:41    Sequencer      INFO      FPE of type FLTUND exceeded configured per-event threshold of 0 (mask: NONE) (seen: 1 FPEs)
 0# Acts::detail::PointwiseMaterialInteraction::covarianceContributions(bool, bool) at /builds/acts/ci-bridge/src/Core/src/Propagator/detail/PointwiseMaterialInteraction.cpp:43
 1# void Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> >::materialInteractor<Acts::PropagatorState<Acts::PropagatorOptions<Acts::SympyStepper::Options, Acts::Navigator::Options, Acts::ActorList<Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> >, Acts::PathLimitReached> >, Acts::SympyStepper::State, Acts::Navigator::State, Acts::KalmanFitterResult<Acts::VectorMultiTrajectory> >, Acts::SympyStepper, Acts::Navigator>(Acts::Surface const*, Acts::PropagatorState<Acts::PropagatorOptions<Acts::SympyStepper::Options, Acts::Navigator::Options, Acts::ActorList<Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> >, Acts::PathLimitReached> >, Acts::SympyStepper::State, Acts::Navigator::State, Acts::KalmanFitterResult<Acts::VectorMultiTrajectory> >&, Acts::SympyStepper const&, Acts::Navigator const&, Acts::MaterialUpdateStage const&) const [clone .isra.0] at /builds/acts/ci-bridge/src/Core/include/Acts/TrackFitting/KalmanFitter.hpp:876
 2# void Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> >::act<Acts::PropagatorState<Acts::PropagatorOptions<Acts::SympyStepper::Options, Acts::Navigator::Options, Acts::ActorList<Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> >, Acts::PathLimitReached> >, Acts::SympyStepper::State, Acts::Navigator::State, Acts::KalmanFitterResult<Acts::VectorMultiTrajectory> >, Acts::SympyStepper, Acts::Navigator>(Acts::PropagatorState<Acts::PropagatorOptions<Acts::SympyStepper::Options, Acts::Navigator::Options, Acts::ActorList<Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> >, Acts::PathLimitReached> >, Acts::SympyStepper::State, Acts::Navigator::State, Acts::KalmanFitterResult<Acts::VectorMultiTrajectory> >&, Acts::SympyStepper const&, Acts::Navigator const&, Acts::KalmanFitterResult<Acts::VectorMultiTrajectory>&, Acts::Logger const&) const [clone .isra.0] at /builds/acts/ci-bridge/src/Core/include/Acts/TrackFitting/KalmanFitter.hpp:383
 3# Acts::Result<void, std::error_code> Acts::Propagator<Acts::SympyStepper, Acts::Navigator>::propagate<Acts::PropagatorState<Acts::PropagatorOptions<Acts::SympyStepper::Options, Acts::Navigator::Options, Acts::ActorList<Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> >, Acts::PathLimitReached> >, Acts::SympyStepper::State, Acts::Navigator::State, Acts::KalmanFitterResult<Acts::VectorMultiTrajectory> > >(Acts::PropagatorState<Acts::PropagatorOptions<Acts::SympyStepper::Options, Acts::Navigator::Options, Acts::ActorList<Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> >, Acts::PathLimitReached> >, Acts::SympyStepper::State, Acts::Navigator::State, Acts::KalmanFitterResult<Acts::VectorMultiTrajectory> >&) const at /builds/acts/ci-bridge/src/Core/include/Acts/Propagator/Propagator.ipp:77
 4# Acts::Result<Acts::TrackContainer<Acts::VectorTrackContainer, Acts::VectorMultiTrajectory, std::shared_ptr>::TrackProxy, std::error_code> Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::fit_impl<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis>, Acts::PropagatorOptions<Acts::SympyStepper::Options, Acts::Navigator::Options, Acts::ActorList<Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> > > >, Acts::KalmanFitterResult<Acts::VectorMultiTrajectory>, Acts::TrackContainer<Acts::VectorTrackContainer, Acts::VectorMultiTrajectory, std::shared_ptr> >(Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> const&, Acts::PropagatorOptions<Acts::SympyStepper::Options, Acts::Navigator::Options, Acts::ActorList<Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::Actor<Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> > > > const&, Acts::TrackContainer<Acts::VectorTrackContainer, Acts::VectorMultiTrajectory, std::shared_ptr>&) const at /builds/acts/ci-bridge/src/Core/include/Acts/TrackFitting/KalmanFitter.hpp:1245
 5# Acts::Result<Acts::TrackContainer<Acts::VectorTrackContainer, Acts::VectorMultiTrajectory, std::shared_ptr>::TrackProxy, std::error_code> Acts::KalmanFitter<Acts::Propagator<Acts::SympyStepper, Acts::Navigator>, Acts::VectorMultiTrajectory>::fit<__gnu_cxx::__normal_iterator<Acts::SourceLink const*, std::vector<Acts::SourceLink, std::allocator<Acts::SourceLink> > >, Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis>, Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis>, Acts::TrackContainer<Acts::VectorTrackContainer, Acts::VectorMultiTrajectory, std::shared_ptr> >(__gnu_cxx::__normal_iterator<Acts::SourceLink const*, std::vector<Acts::SourceLink, std::allocator<Acts::SourceLink> > >, __gnu_cxx::__normal_iterator<Acts::SourceLink const*, std::vector<Acts::SourceLink, std::allocator<Acts::SourceLink> > >, Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> const&, Acts::KalmanFitterOptions<Acts::VectorMultiTrajectory> const&, Acts::TrackContainer<Acts::VectorTrackContainer, Acts::VectorMultiTrajectory, std::shared_ptr>&) const at /builds/acts/ci-bridge/src/Core/include/Acts/TrackFitting/KalmanFitter.hpp:1134
 6# (anonymous namespace)::KalmanFitterFunctionImpl::operator()(std::vector<Acts::SourceLink, std::allocator<Acts::SourceLink> > const&, Acts::GenericBoundTrackParameters<Acts::ParticleHypothesis> const&, ActsExamples::TrackFitterFunction::GeneralFitterOptions const&, ActsExamples::MeasurementCalibratorAdapter const&, Acts::TrackContainer<Acts::VectorTrackContainer, Acts::VectorMultiTrajectory, std::shared_ptr>&) const at /builds/acts/ci-bridge/src/Examples/Algorithms/TrackFitting/src/KalmanFitterFunction.cpp:137
 7# ActsExamples::TrackFittingAlgorithm::execute(ActsExamples::AlgorithmContext const&) const at /builds/acts/ci-bridge/src/Examples/Algorithms/TrackFitting/src/TrackFittingAlgorithm.cpp:141

Analysis

We end up here with a multiplication of small floats.
Core/src/Propagator/detail/PointwiseMaterialInteraction.cpp:43:

const float sigmaQoverP =
    computeEnergyLossLandauSigmaQOverP(slab, mass, qOverP, absQ);
varianceQoverP = sigmaQoverP * sigmaQoverP;

These small floats are coming from the multiplication of many other small floats in
Core/src/Material/Interactions.cpp:310:

float Acts::computeEnergyLossLandauSigmaQOverP(const MaterialSlab& slab,
                                               float m, float qOverP,
                                               float absQ) {
  const RelativisticQuantities rq{m, qOverP, absQ};
  const float fwhm = detail::computeEnergyLossLandauFwhm(slab, rq);
  const float sigmaE = convertLandauFwhmToGaussianSigma(fwhm);
  //  var(q/p) = (d(q/p)/dE)² * var(E)
  // d(q/p)/dE = d/dE (q/sqrt(E²-m²))
  //           = q * -(1/2) * 1/p³ * 2E
  //           = -q/p² E/p = -(q/p)² * 1/(q*beta) = -(q/p)² * (q/beta) / q²
  //  var(q/p) = (q/p)^4 * (q/beta)² * (1/q)^4 * var(E)
  //           = (1/p)^4 * (q/beta)² * var(E)
  // do not need to care about the sign since it is only used squared
  const float pInv = qOverP / absQ;
  const float qOverBeta = std::sqrt(rq.q2OverBeta2);
  return qOverBeta * pInv * pInv * sigmaE;
}

Next steps

As discussed with @andiwand we should wait, until we switch the material from float to double and see, if the problem persists.