Include dataset hash in job search

[mvdbeek]

This means we find equivalent jobs if an input hda either points at the same dataset id (existing behavior), or if the dataset source_uri, transform and hashes match. All further restrictions still apply (same metadata etc).

Builds on #19108, and #19110 is probably also a good idea.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[bgruening]

Running a Galaxy Training Academy without large computational resources - yeah, yeah!

[jmchilton]

We have no clue about extra files in this context right - how can we make this conclusion without that? It isn't a -1 but I am uncomfortable with the direction this is all heading. I feel like we should be working on tightening up job search and not loosening it more and more before we've installed the guard rails it needs IMO.

[jmchilton]

Maybe to clarify this - I absolutely think we should be able to hash the input to be used with job search. But I don't think this hash is what I would use - I would implement a hash of the dataset and not of the primary file of the dataset. 95% of the time they could be the same but we should verify that before using it in this context.

[mvdbeek]

I would implement a hash of the dataset and not of the primary file of the dataset. 95% of the time they could be the same but we should verify that before using it in this context.

If we match the transform would it still be possible to get different extra files for uploaded content (considering we already match on the datatype) ? I added 4abc6e6 because we don't record the transform for local files, but we could of course do that.

There's of course also value in calculating the hash for datasets as they are written to the object store, but if we can get reliable equivalence using the upload hash + transform we could make nice progress in figuring out other edge cases for IWC workflows ?

[jmchilton]

I had the realization reading your response that you only care about upload or think these fields can all only be set during uploads. I believe any tool can produce hashes and source hashes and I believe we have an API for hashing files after the fact anyway. Could we restrict all this logic to just fetch and upload1 - then it feels pretty close to being correct? This also probably explains my unease with #19110 - I think we don't validate the hashes outside the fetch tool but we can create the hashes in other tools I think - it feels like what we need is a hash validated field if we want to act on that data in this fashion but maybe it would be sufficient to be more proactive in validating the hash field whenever it is set.

[nsoranzo]

You may want to rebase this now that #19181 has been merged (thanks @jmchilton !), since it contained some type annotation fixes copied from here.

If we match the transform would it still be possible to get different extra files for uploaded content (considering we already match on the datatype) ?

Not sure it's the same problem, but in #19181 I found out that a type of transform (grooming) is not reproducible (because the BAM header contains paths as part of the samtools command used to sort the data), so you'd get different hashes if you materialize twice from the same source.

So I think I agree it's safer to match on the actual dataset hashes instead of the dataset source hashes.

[mvdbeek]

Uhm, I'm confused, is that not what I'm doing ?

[nsoranzo]

Uhm, I'm confused, is that not what I'm doing ?

Ah, yes, I guess I was confused by the discussion above and the PR title, shouldn't it be "Include dataset hash in job search"? "Source hash" hints at the DatasetSourceHash model class to me (instead of DatasetHash, which is what you are using).

[nsoranzo]

@mvdbeek Still draft?

[mvdbeek]

I think so ? I need to think about all the ways it could break, and extra files is a valid concern, I think ?

[nsoranzo]

I think so ? I need to think about all the ways it could break, and extra files is a valid concern, I think ?

Right, should we only use dataset hashes if there are no extra files for the moment?

[mvdbeek]

That sounds good to me. I'm hacking away on something else right now, if you want to add to the PR that would be really cool, otherwise I'll come back to it later.