Add faithfulness metric

[pat-alt]

Observations

• Posterior sampling is very sensitive to scale of underlying data (step size for SGLD is in absolute terms).

To Do

• Need to ensure that is standardized/normalized before running sampling. Can instead just make the parameters for the sampling rule (ImproperSGLD) dependent on the second moment of the prior distribution.
• Get rid of clipping again.
• Support for mini-batch sampling?
• Make EnergySampler part of model M.fitresult so that it can be recycled.
• Add MLJFlux extension Add extensions for MLJFlux models #469
• Actually train a good model in JEM.jl
• Complete tutorial
• Closer look at role of reg loss
• Let decay depend on maximum number of iterations
• Why not use Normalized Compression Distance? Only available package only applicable to strings
• Cosine distance has advantage of being (-1,1)
• Add EnergySamplers dependency

Notes

Cosine distance works kind of well, but the problem is that scale independence leads to issues such as this one

julia> dist([1,2],[1,0])
0.5527864045000421

julia> dist([100,200],[1,0])
0.5527864045000421

where the cosine distance is the same in both cases even though the counterfactual is way off domain.

[pat-alt]

@alifarukyucel to see

[mschauer]

Is faithful here different or equal to from graphical "faithfulness"? https://www.cmu.edu/dietrich/philosophy/docs/scheines/introtocausalinference.pdf

[pat-alt]

Is faithful here different or equal to from graphical "faithfulness"? https://www.cmu.edu/dietrich/philosophy/docs/scheines/introtocausalinference.pdf

I'm not familiar with this work but from briefly looking at their definition of faithfulness, it's different. We define faithful counterfactuals as "consistent with what the model has learned about the data" in our AAAI paper: https://scholar.google.com/citations?view_op=view_citation&hl=en&user=e7KRRa8AAAAJ&citation_for_view=e7KRRa8AAAAJ:_FxGoFyzp5QC

Happy to chat more about it in one of our meetings

[codecov]

Codecov Report

Attention: Patch coverage is 20.09804% with 163 lines in your changes missing coverage. Please review.

Project coverage is 81.80%. Comparing base (0d5d16e) to head (509fc05).
Report is 19 commits behind head on main.

❗ Current head 509fc05 differs from pull request most recent head 7d50a87

Please upload reports for the commit 7d50a87 to get more accurate results.

Files	Patch %	Lines
src/evaluation/faithfulness/utils.jl	0.00%	92 Missing ⚠️
ext/JEMExt/jem.jl	0.00%	33 Missing ⚠️
src/objectives/penalties.jl	47.05%	9 Missing ⚠️
src/evaluation/plausibility/plausibility.jl	0.00%	8 Missing ⚠️
src/evaluation/utils.jl	0.00%	6 Missing ⚠️
src/evaluation/faithfulness/faithfulness.jl	0.00%	5 Missing ⚠️
ext/JEMExt/pretrained.jl	0.00%	4 Missing ⚠️
src/extensions/JEMExt.jl	0.00%	2 Missing ⚠️
src/counterfactuals/core_struct.jl	0.00%	1 Missing ⚠️
src/models/core_struct.jl	85.71%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #455      +/-   ##
==========================================
- Coverage   90.37%   81.80%   -8.58%     
==========================================
  Files          80       86       +6     
  Lines        1559     1671     +112     
==========================================
- Hits         1409     1367      -42     
- Misses        150      304     +154     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pat-alt]

Hi @MaritRadder, tagging you here already but the tutorial is still bare bones (just code). I'll try to fill it with some text asap and will probably merge this then and complete the remaining tasks here in another PR.

[pat-alt]

@MaritRadder the code here is already updated: TaijaBase.Samplers has been moved into its own new package EnergyBasedSamplers.