Add usage of EMParams

[johschnee]

Changes

• Class EMParams is now used by the ExpectationMaximizationModel in tools/_em_model.py.

Related issues

Closes #1026, #1025.

[johschnee]

@WeilerP Currently, I have the following questions about the future design of the API:

1. Should all the fitted parameters still be written in the AnnData object by default when calling ExpectationMaximiziationModel.fit()? Or should the EMParams class become the new way of handling the inferred parameters and only be written to the AnnData object when calling ExpectationMaximiziationModel.export_results_adata() (not yet implemented)?
2. My current implementation relies on the methods _read_pars() and _write_pars() in tools/_em_model_core.py. Should these methods be kept in the implementation at all or do you aim to remove them with this restructuring?
3. In the old method recover_dynamics() the parameter copy is false by default. In the new implementation ExpectationMaximizationModel.fit() it is true by default. Is this on purpose?

[WeilerP]

Thanks, @johschnee. I added two comments which should also answer two of your questions. Regarding the third: Since we'll not be writing to AnnData in the fit method, the copy argument will become redundant. So it doesn't really matter what the default argument is, ATM (although it's a good point).

[WeilerP]

@johschnee, is there an update on this PR?

[johschnee]

Yes, I was working on it actually today. I added the implementation of the abstract methods to this PR, because export_results_adata is used instead of _write_pars.

[WeilerP]

@johschnee, I left another few comments. Mainly two things:

1. I updated/refactored the _initialize_state_dict function. Can you please double check that it is still working?
2. Can you please und all changes irrelevant to this PR (i.e., refactoring the code to rely on EMParams)? These changes (e.g., function implementations) should be there own PR accompanied by an issue.

[WeilerP]

FYI, @johschnee, the ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (4,) + inhomogeneous part. error is caused by a new Numpy version (see #1058).

[johschnee]

Hi @WeilerP,
this branch now contains the changes we discussed. In particular, I changed the following:

• Added metadata to the attributes of EMParams and read/write methods: The metadata allows a different handling of vector and matrix parameters without hardcoding them in the read/write methods. I decided to add the read/write methods to EMParams because they rely a lot on the attributes of EMParams.
• In the method fit(), I used the class EMParams. Since the AnnData object should not be modified during fit(), I defined some private attributes to store results and additional settings: _loss, _fit_connected_states, and _use_raw. They are written to an AnnData object when export_results_adata() is called.
• As we discussed, I copied _align_dynamics() to the class ExpectationMaximizationModel and adapted it slightly to use the EMParams. Now, ExpectationMaximizationModel only uses the class DynamicsRecovery from _em_model_core, but no other methods.

Overall,

model = ExpectationMaximizationModel(adata)
model.fit()
adata_return = model.export_results_adata()

and

scv.tl.recover_dynamics(adata)

should lead to the same results. I tested it with simulated data and the pancreas dataset.

[johschnee]

Hi @WeilerP,
I changed the behavior of export_to_adata() such that arrays containing only nan values are not written anymore.
I also compared the results of the old and the new API for the simulated data and the pancreas dataset using numpy. As expected, they were identical.
Hence, it would be great if you could review this PR.

[WeilerP]

Thanks, @johschnee!