hash all downloads

[rabernat]

Just noting how easy it is to hash all downloads.

In our current copy function

pangeo-forge-recipes/pangeo_forge_recipes/storage.py

Lines 35 to 43 in f64a654

	with output_opener as target:
	start = time.time()
	interval = 5 # seconds
	bytes_read = log_count = 0
	while True:
	data = source.read(BLOCK_SIZE)
	if not data:
	break
	target.write(data)

we could easily insert this

md5hash = hashlib.md5()
with fsspec.open(fname, mode='rb') as fp:
    while True:
        data = fp.read(block_size)
        if not data:
            break
        md5hash.update(data)

I have confirmed that a hash built this way matches the md5 command line results.

[cisaacstern]

Where would we store this hash and what would it be used for?

[rabernat]

There are three basic uses:

• Validation against data corruption (most effective if the original hash is available somewhere)
• Determining if we need to re-download a cached file
• Indexing for content-addressable storage such as IPFS

We could store it in the metadata store for now.

We might also eventually want to be saving the list of ingested files in the database (e.g. to enable appending; #37). Storing the hash along with the file would be a good practice here.