Reading EBSP patterns and settings from other formats than Bruker's

[hakonanes]

Problem
I don't have Bruker eSprit, but pattern intensities stored in a 4D HDF5 file and a separate text file with settings (such as grid dimensions, step size, pattern resolution in pixels etc.), acquired using NORDIF. I have made a simple reader for this format (in my fork of this repo), enabling me to index my patterns using AstroEBSD (thank you so much!).

If this reader is to be included in the main repo, what format should the patterns be in (e.g. grid dimensions or line?), and how should the settings be included in the HDF5 file?

Ideally this reader should be made so that more people can use it for their patterns, and not only users of the NORDIF software.

Suggested solution
One HDF5 file with both patterns and settings is best. As far as I can tell, AstroEBSD only needs for the following Matlab structures (following your conventions in Tutorial: Crystal orientations and EBSD — Or which way is up?):

• MicroscopeData:
  • NROWS: # patterns in Xs
  • NCOLS: # patterns in Ys
  • WD: working distance in mm, Zs
  • SampleTilt: in degrees
  • CameraTilt: in degrees
  • PatternWidth: # pixels in Xd
  • PatternHeight: # pixels in Yd
  • XSTEP: step size in Xs in microns
  • YSTEP: step size in Ys in microns
  • CoordSystems: e.g. Top Right Z Plus for Bruker coordinates, or Top Left Z Minus for EDAX coordinates (?)
• MapData: nothing, as these matrices (detector distances, pattern centres etc.) are created during indexing?
• EBSD_DataInfo:
  • PW: pattern width, # pixels in Xd
  • PH: pattern height, # pixels in Yd

These settings should be stored in the HDF5 file structure. Is there any best way to do this?

Additional context
Is there a good reason to structure the HDF5 file the same way Bruker does?

[benjaminbritton]

Hi - thanks for this note.

It would be good to have this in a formation which is as close as possible to the HDF5 format that Bruker does this. We have a number of ancillary codes that work with this format (and for instance there is a MTEX reader too), so it would be convenient to keep this similar. This would also provide a method to enable users of other software to 'wrap' their patterns and information into a similar file.

In terms of key items in the file, I think you have these in the list. MapData is not needed (as this would be created in indexing - as you note). For CoordSystems - we assume TRZP for the code at present, but this may need changing for other systems - I would not worry about this too much at the moment, but demonstrate that you can get the correct orientation for a non symmetric unit cell cell (as per the tutorial paper).

If you are able to write a short converter file to make the HDF5 file from your inputs, and willing to share one example file pair, we can add the converter into the main branch of the code, so people can have a 'pre step' of building the HDF5 as an example for their own use. This will also motivate me to build a post processor to wrap a H5 format which can be stored for offline use & quick reading by MTEX.

Regards,
Ben

[hakonanes]

Sorry for the late reply.

Since I started this issue I have been working on a 'subpackage' of HyperSpy for EBSD patterns and have gotten familiar with their way of loading data into HyperSpy structures. They have a general load/save function which calls a custom reader/writer function based upon the file extension passed to it. If I want to read my NORDIF EBSD patterns from a binary Pattern.dat file, I only have to write file_reader and file_writer functions as per HyperSpy's specification. This way additions of new plugins are streamlined.

I think this approach is better for AstroEBSD than my initial suggestion of creating HDF5 files of all my NORDIF .dat files before reading them into AstroEBSD. As far as I can see, only custom versions of bReadHDF5 and bReadEBSP needs to be added for every new plugin, providing the above list of parameters.

[benjaminbritton]

Hi,

Delayed too. I've written a HDF5 writer function (it's in the EBSD 2019 pre-release branch - we are just bug testing - look for get/bCreateHDF5.m and decks/EBSD_2019_Astro2.m for an alternative).

Having different file readers is find in principal though (but this gets messy quickly). If you have an example dat file to share & your file reader then I can peek at how we can join these together.

Ben

[hakonanes]

Hi Ben,

in my fork in a branch named nordif-reader, https://github.com/hwagit/AstroEBSD/tree/nordif-reader, I've created a Nordif file reader and made the necessary changes to AstroEBSD for it to read either from Bruker's HDF5 format or the Nordif binary format. I rewrote the README explaining all changes I've made. Although I've made an effort to not introduce bugs, I've not tested the code for other use cases than my own...

A small Nordif example data set can be found in another repo, https://github.com/hwagit/astroebsd-nordif-tutorial/tree/master/data, in which I've also made available a brief tutorial PDF aimed at students at our university (NTNU) with an interest in indexing their Nordif data using AstroEBSD.

Håkon