Use eko.runner.managed.solve

[felixhekhorn]

Use the new "jets runner"

Remember that there are still upcoming changes in eko NNPDF/eko#296 which might affect pineko ...

cc @alecandido @scarlehoff

[scarlehoff]

But should this be default if it is slower? Maybe it should be a cmdlije option.

[felixhekhorn]

But should this be default if it is slower? Maybe it should be a cmdlije option.

I would argue yes, because (as you experienced yourself) the current default may not converge always and it shouldn't be much slower: yes, there are more write-to-disk operations, but compared to the numerical integration (~mins to hours) this should be negligible (to be proven, but I'm quite sure)

[niclaurenti]

But should this be default if it is slower? Maybe it should be a cmdlije option.

With the same number of cores it maybe be slightly slower, but the fact that it uses less memory allows us to increase the number of cores to make faster. (This is what I did to compute the evolution eko for QED)

[alecandido]

I would argue yes, because (as you experienced yourself) the current default may not converge always and it shouldn't be much slower: yes, there are more write-to-disk operations, but compared to the numerical integration (~mins to hours) this should be negligible (to be proven, but I'm quite sure)

Yes, and also it was just a transition. We don't want to maintain both, so at some point the legacy one will be fully dropped.

[alecandido]

With the same number of cores it maybe be slightly slower, but the fact that it uses less memory allows us to increase the number of cores to make faster. (This is what I did to compute the evolution eko for QED)

In principle, it could even be distributed on multiple nodes. The computation is happening in two steps:

0. (recipes to detail what to compute are generated)
1. many independent parts are integrated (the heavy lift)
2. the parts are merged in full-blown operators

We never finished implementing the infrastructure for splitting recipes and merging parts, but it just boils down to file system operations (moving files around).
Then you could split your point 1. computation in as many batches as you wish, and distribute wherever you want.

[scarlehoff]

Please wait to merge because I'm still not sure this solves the problem.

[andreab1997]

Please wait to merge because I'm still not sure this solves the problem.

I guess this worked at the end right? So we can probably merge.

[scarlehoff]

Well, I was not able to create the fktables so not sure if the eko is correct or anything but I guess other people has already tested this a lot?

[andreab1997]

Well, I was not able to create the fktables so not sure if the eko is correct or anything but I guess other people has already tested this a lot?

Yes of course, we know that it is possible, you just need more memory. Of course it is not ideal but it is not related to this PR.

[felixhekhorn]

Can this be merged now?

[alecandido]

@felixhekhorn since Pineko is the main EKO user, if this works for Pineko, shouldn't you make the managed.solve just the default in EKO itself, and throw away the legacy one? (together with eko.evolution_operator.grid)

[felixhekhorn]

@felixhekhorn since Pineko is the main EKO user, if this works for Pineko, shouldn't you make the managed.solve just the default in EKO itself, and throw away the legacy one? (together with eko.evolution_operator.grid)

We may want to do this at some point, but since I consider this is a major change (although the user should(tm) never notice), I would delay that

[alecandido]

We may want to do this at some point, but since I consider this is a major change (although the user should(tm) never notice), I would delay that

I agree it's a major change, but eventually it's the same if you do it in Pineko or EKO :P

I know that EKO has also other users beyond Pineko, and that you'd like to group this major change with other ones. But if the alternative is to release the major in a couple of months, you could well release a major now with just that, and another one in a couple of months.