What happened? (You can include a screenshot if it helps explain)

It is unclear if the number in the progress widget (in center of circle) is representing the number of FMs completed, or if it is the number of which FM is currently running. This should be the same number, but there can be a situation where an event is lost/dropped, then this number number will be off.
In the attached image, realization 4 is shown as 101/108 FMs are completed, yet the detailed table view shows that FM 72 is still running while 73,74,75,76... has successfully finished.

This happened with max_submit=2, so there might be something buggy with how we do status counting when resubmitting realizations.

What did you expect to happen?

FMs after a running FM should not be marked as finished if the proceeding one has not finished. In case an event is dropped, we should be able to conclude that it has succeeded if the FMs after it are running/finished.

steps to reproduce

TBD

Environment where bug has been observed

• python 3.8
• python 3.10
• python 3.11
• python 3.12
• macosx
• rhel7
• rhel8
• local queue
• lsf queue
• slurm queue
• openPBS queue