Report termination information

[afmagee42]

Revised per synchronous discussion, this PR now:

• Tracks simulation termination
• Excludes simulations from computation when they stop due to max_infections
  • It also checks for 0 successful simulations and throws an st.error instead of a runtime error.
• Warns users if max_infections is being hit and requires them to acknowledge the potentially-fraught results.
• Adds to the spin wheel text to say that simulations being very slow is probably bad news.

Known issues:

• I'm pretty sure if you say "show me anyways" it re-runs the simulations. We could maybe wrap the entire display code into a function and decorate it with an st.fragment but since this is already ill-advised usage, that seems like overkill.

[swo]

I agree it's a good idea to indicate whether you terminated because of hitting max # of infections

But I think the stuff about "last complete generation" can be retrieved from the simulation results. I'd rather compute all the summary values (e.g., total # of infections, last complete generation, etc.) in the summary output widget/postprocessing.

We can incorporate the "hit max infections" flag into summary outputs. Like, we can define "successful control" as <max infections AND zero infections in CoCoCs.

Thoughts?

[afmagee42]

But I think the stuff about "last complete generation" can be retrieved from the simulation results. I'd rather compute all the summary values (e.g., total # of infections, last complete generation, etc.) in the summary output widget/postprocessing.

I don't see how? Once the queue is gone, how do I know if a generation was in-progress or not?

Example:

set max_infections = 5
infection `0` draws infections at time 1, 2, 3
at time 1, new infection `1` draws times 1.1, 1.2, 1.3, 1.4
len(queue) > max_infections
stop

We never hit the infection at time 2, so no generations are complete (or, I guess technically generation 0 is), so Simulation.infections just contains ["0", "1"], and would appear to contain people from the first generation, but that generation is not complete.

We can incorporate the "hit max infections" flag into summary outputs. Like, we can define "successful control" as <max infections AND zero infections in CoCoCs.

I don't follow

[swo]

I don't see how? Once the queue is gone, how do I know if a generation was in-progress or not?

I see your point: if infection N_max occurred in generation G, you don't know if the N+1-th would have been in G or G+1

But my hypothesis is that we don't need to know in precisely which ring the simulation finished for any of the relevant outputs? Like, I'm seeing hitting max # of infections as a sign that you're asking this widget to perform outside its specs

We can incorporate the "hit max infections" flag into summary outputs. Like, we can define "successful control" as <max infections AND zero infections in CoCoCs.

I don't follow

If we hit max # of infections, I'd like the main output to be: hey you either need to change the params or you can kiss control goodbye, probably

Like, maybe there's some world where you want some % of chains to go extinct, but you're also OK with some chains growing to >100, but that just doesn't feel epidemiologically relevant

I think the outputs I want are:

• Did you achieve control? where "control" means you had fewer than the max # of infections AND no infected contacts-of-contacts-of-contacts
• How many infections total were there? Show a histogram(?) truncated at the max # of infections

If we really do need more rigorous stats, I'd rather just demand that we push the max # of infections higher. Again, this is just a widget to give you an idea of outcomes.

[afmagee42]

I agree that if you don't set max_infections to something ridiculous (like 5) then hitting it is probably a sign you've blown past control.

I suppose whether we discard those simulations or incorrectly use them, we're still stuck with the fact that conditional summaries are biased towards things looking rosier.

[afmagee42]

In other words:

• To properly compute Pr(containment | # infections < M) or Pr(# infections = m | # infections < M) we need this information that we're not tracking, but I believe Pr(containment | # infections < M) > Pr(containment) and E[# infections = m | # infections < M] < E[# infections = m].
• If we take the event of maxing-out the infecteds to be non-containment
  • We might have a decent approximation to Pr(containment)
  • I don't know what the probability distribution on sizes is here, it's not a well-specified quantity

[swo]

Right, I'm saying that the constant M should be large enough that Pr(containment | # infections >= M) is negligible, so that Pr(containment) ~= P(cont | #inf < M) * P(#inf < M).

But for now, let's just have a big warning sign that says "you hit max # of infections in x% of simulations; interpret results with care" and see if this is actually a problem before we try to repair the statistics

[afmagee42]

This is going to bug me big time if we don't get it satisfactorily right-ish the first time.

But on reflection, I was being a bit pessimistic and I do think we can get something which is both interpretable and sensible if we:

Track early termination
Force max_infections to not be tiny (say, a minimum of 100)
Then we can

Make the "maxing out means failed containment" assumption to appropriately compute Pr(containment)
≈
1 - Pr(hit max) - Pr(no containmnent | did not hit max)
Provide your suggested conditional summary of sizes, the probability distribution Pr(size | did not hit max) ≡ Pr(size | size < max), this is just the histogram of final sizes of all non-early-terminated simulations (including extinct ones)

(sorry, @swo tried to quote this comment but instead edited it and now I can't revert it)

[afmagee42]

We can also add warnings about things blowing up impacting results, and a "if your simulations are taking too long, you've probably got an absurdly large $R_e$, maybe don't do that" warning too

[swo]

2. Force max_infections to not be tiny (say, a minimum of 100)

Yes, this was my thought; this is mostly an escape valve so that you're not sitting for a long time and waiting for a simulation to finish

But we should also maybe make this less possible via more sensible slider rangers

[swo]

LGTM; some things to maybe change downstream; I'll open issues

[swo]

I think there's some way to refactor all this for mild cleanliness; I'll open a PR if I have something useful to say

[afmagee42]

Yeah, that logic is definitely not the prettiest, happy if there's a cleaner way to express "plot the results unless the user pushes the big red override button"

[afmagee42]

@swo I well and truly appreciate all the improvements but I will not approve any more PRs into this PR above #45. Anything else can be handled as a new PR into main later