PySpark/Spark implementation

[hjk612]

Are there any plans to implement DoubleML in either PySpark or Spark Scala? The current implementation doesn’t scale efficiently to large datasets because scikit-learn is not designed for distributed training. I recognize that it can utilize all available cores, but it still falls short for very large datasets.

[SvenKlaassen]

Currently, there are no plans on a PySpark or Spark Scala implementation.
I agree with the argument on distributed training. Therefore we implemented the option to use external predictions.
It is not optimal, but enables a more control over the nuisance predictions (e.g. tuning etc.)
As a downside, the user has to take care of the correct sample-splitting or cross-fitting.

[hjk612]

Makes sense! Thanks for your prompt reply. I will look into external predictions. Would it be of interest if I create a spark equivalent of doubleML?

[SvenKlaassen]

Generally, I think this would be very interesting, but I am not sure which features you would like to support.
If you are interested, we could just schedule a short talk.

[hjk612]

Absolutely, would love to connect and discuss further. What would be the right mechanism to connect?