Break the DIALS/xfel circular dependency (#872)

Brief summary:

* A new top level project in cctbx_project has been created: serialtbx
* Any code needed by DIALS and dxtbx that was in xfel has been moved to serialtbx or directly into DIALS/dxtbx. That mostly includes code to read data at XFELs used by dxtbx, and stills-specific indexing/integration methods.
* serialtbx has no imports of DIALS or dxtbx and so it can be directly configured by the conda packages cctbx and cctbx-base, @bkpoon. That said, there are no dispatchers or libtbx_config file so nothing specifically needs to be done to bootstrap.py, I believe.
* All imports of xfel in DIALS and dxtbx have been updated to use the new serialtbx or DIALS/dxtbx imports
* All other known repos that have xfel imports have been updated to use the new serialtbx or DIALS/dxtbx imports
* xfel will remain configured in the DIALS binary bundles on dials.github.io and distributed by phenix. For conda, the plan is to make a cctbx.xfel conda package that includes all its dependencies (MPI, MySQL, etc.)

Se also cctbx/dxtbx#627 and dials/dials#2404

Detailed listing of code movements:
* xfel.util.jungfrau → serialtbx.detector

From xfel.cftbx.detector:
* cspad_detector → serialtbx.detector.legacy_metrology
* generic_detector → serialtbx.detector.legacy_metrology
* metrology → serialtbx.detector.legacy_metrology

From xfel.mono_simulation:
* max_like → serialtbx.mono_simulation
* util → serialtbx.mono_simulation

From xfel.cftbx.detector.cspad_cbf_tbx:
* Constants moved to serialtbx.detector.cspad
* get_psana_corrected_data → serialtbx.detector.cspad
* cbf_wrapper → merged with dxtbx.format.FormatCBFMultiTile.cbf_wrapper
* angle_and_axis → dxtbx.format.FormatCBFMultiTile.cbf_wrapper
* center → serialtbx.detector
* basis → serialtbx.detector
* basis_from_geo → serialtbx.detector.xtc
* read_slac_metrology → serialtbx.detector.cspad
* add_frame_specific_cbf_tables → dxtbx.format.cbf_writer

From xfel.command_line.cspad_detector_congruence:
* iterate_detector_at_level → serialtbx.detector
* iterate_panels → serialtbx.detector
* id_from_name → serialtbx.detector
* get_center → serialtbx.detector

From xfel.command_line.frame_extractor:
* ConstructFrame → serialtbx.util.construct_frame

From xfel.cxi.cspad_ana.cspad_tbx:
* Constants moved to serialtbx.detector.cspad
* address_split → serialtbx.detector.xtc
* dpack → serialtbx.detector.cspad
* get_ebeam → serialtbx.detector.xtc
* env_distance → serialtbx.detector.xtc
* evt_wavelength → serialtbx.detector.xtc
* env_detz → serialtbx.detector.xtc
* old_address_to_new_address → serialtbx.detector.xtc

From xfel.cxi.cspad_ana.rayonix_tbx.py:
* Constants moved to serialtbx.detector.rayonix
* get_rayonix_pixel_size → serialtbx.detector.rayonix
* get_rayonix_detector_dimensions → serialtbx.detector.rayonix
* get_data_from_psana_event → serialtbx.detector.rayonix

C++ code from xfel/ext.cpp:
* radial_average → dxtbx.ext

From xfel.util:
* sublattice_helper → dials.algorithms.integration

Commits:
* Move xfel.mono_simulation.max_like to serialtbx
* Move time functionality from cspad_tbx to serialtbx/util/time.py
* Move xfel.mono_simulation.util to serialtbx/mono_simulation
* xfel.util.sublattice_helper moved to dials.algorithms.integration
* Move xfel.command_line.frame_extractor.ConstructFrame to serialtbx.util
* Move radial average c++ code to dxtbx
* Move dpack to serialtbx.detector.cspad
* Move jiffy functions from cspad_detector_congruence to serialtbx.detector
* Move utility functions and constants out of cspad_tbx into serialtbx.detector.cspad and serialtbx.detector.xtc
* Move movement of functions into serialtbx
* Moving out of cspad_cbf_tbx:
- Into serialtbx.detector: basis, center
- Into serialtbx.detector.cspad: read_slac_metrology
- Into serialtbx.detector.xtc: basis_from_geo
* Use dxtbx's cbf_wrapper by adding in the only function in the xfel subclass of cbf_wrapper
* Move add_frame_specific_cbf_tables to dxtbx
* Move xfel.util.jungfrau to serialtbx.detector
* cspad constants and data reading moved to serialtbx
* Move rayonix functions and constants into serialtbx
* Move old_address_to_new_address to serialtbx
* Move legacy image pickle metrology code to serialtbx.detector.legacy_metrology
* Missing imports and a few more functions moving into serialtbx
* Temporarily check out serial_tbx branches on other repos
* Duplicated code
* Fix import
* Add serialtbx to CCIBuilder configuration list. This should ensure serialtbx is included in the conda cctbx-base package
* Revert "Temporarily check out serial_tbx branches on other repos"

---------

Co-authored-by: Billy K. Poon <[email protected]>

libtbx/auto_build/bootstrap.py

@@ -2146,6 +2146,7 @@ class CCTBXLiteBuilder(CCIBuilder):
     'cctbx',
     'cctbx_website',
     'scitbx',
+    'serialtbx',
     'libtbx',
     'iotbx',
     'mmtbx',

rstbx/command_line/slip_viewer.py

@@ -69,7 +69,7 @@ def run(argv=None):
   frame.settings_frame.panel.collect_values()
 
   if (work_params.effective_metrology is not None):
-    from xfel.cftbx.detector.metrology import \
+    from serialtbx.detector.legacy_metrology.metrology import \
       master_phil, metrology_as_transformation_matrices
 
     stream = open(work_params.effective_metrology)

rstbx/slip_viewer/calibration_frame.py

@@ -106,7 +106,7 @@ def OnRestoreMetrology(self, event):
     if dialog.ShowModal() == wx.ID_OK:
       path = dialog.GetPath()
       if (path != ""):
-        from xfel.cftbx.detector.metrology import \
+        from serialtbx.detector.legacy_metrology.metrology import \
           master_phil, metrology_as_transformation_matrices
         from libtbx import phil
 

rstbx/slip_viewer/tile_generation.py

@@ -50,8 +50,8 @@ def _get_flex_image(
 
 def _get_flex_image_multipanel(panels, raw_data, beam, brightness=1.0,
                                binning=1, show_untrusted=False, color_scheme=0):
-  # From xfel.cftbx.cspad_detector.readHeader() and
-  # xfel.cftbx.cspad_detector.get_flex_image().  XXX Is it possible to
+  # From serialtbx.detector.legacy_metrology.cspad_detector.readHeader() and
+  # serialtbx.detector.legacy_metrology.cspad_detector.get_flex_image().  XXX Is it possible to
   # merge this with _get_flex_image() above?  XXX Move to dxtbx Format
   # class (or a superclass for multipanel images)?
 
@@ -61,7 +61,7 @@ def _get_flex_image_multipanel(panels, raw_data, beam, brightness=1.0,
   from libtbx.test_utils import approx_equal
   from scitbx.array_family import flex
   from scitbx.matrix import col, rec, sqr
-  from xfel.cftbx.detector.metrology import get_projection_matrix
+  from serialtbx.detector.legacy_metrology.metrology import get_projection_matrix
 
   assert len(panels) == len(raw_data), (len(panels), len(raw_data))
 

serialtbx/detector/__init__.py

@@ -0,0 +1,124 @@
+from __future__ import division
+from scitbx import matrix
+
+def center(coords):
+  """ Returns the average of a list of vectors
+  @param coords List of vectors to return the center of
+  """
+  for c in coords:
+    if 'avg' not in locals():
+      avg = c
+    else:
+      avg += c
+  return avg / len(coords)
+
+class basis(object):
+  """ Bucket for detector element information """
+  def __init__(self, orientation = None, translation = None, panelgroup = None, homogenous_transformation = None, name = None):
+    """
+    Provide only orientation + translation or a panelgroup or a homogenous_transformation.
+
+    @param orientation rotation in the form of a quarternion
+    @param translation vector translation in relation to the parent frame
+    @param panelgroup dxtbx panelgroup object whose local d matrix will represent the
+    basis shift
+    @param homogenous_transformation 4x4 matrix.sqr object representing a translation
+    and a rotation. Must not also contain a scale as this won't be decomposed properly.
+    @param name optional name for this basis shift
+    """
+    self.include_translation = True
+    self.name = name
+
+    if panelgroup is not None:
+      d_mat = panelgroup.get_local_d_matrix()
+      fast = matrix.col((d_mat[0],d_mat[3],d_mat[6])).normalize()
+      slow = matrix.col((d_mat[1],d_mat[4],d_mat[7])).normalize()
+      orig = matrix.col((d_mat[2],d_mat[5],d_mat[8]))
+
+      v3 = fast.cross(slow).normalize()
+
+      r3 = matrix.sqr((fast[0],slow[0],v3[0],
+                       fast[1],slow[1],v3[1],
+                       fast[2],slow[2],v3[2]))
+
+      self.orientation = r3.r3_rotation_matrix_as_unit_quaternion()
+      self.translation = orig
+
+      if not self.name:
+        self.name = panelgroup.get_name()
+
+    elif orientation is not None or translation is not None:
+      assert orientation is not None and translation is not None
+      self.orientation = orientation
+      self.translation = translation
+
+    else:
+      # Decompose the homegenous transformation assuming no scale factors were used
+      h = homogenous_transformation
+      self.orientation = matrix.sqr((h[0],h[1],h[2],
+                                     h[4],h[5],h[6],
+                                     h[8],h[9],h[10])).r3_rotation_matrix_as_unit_quaternion()
+      self.translation = matrix.col((h[3],
+                                     h[7],
+                                     h[11]))
+      assert h[12] == h[13] == h[14] == 0 and h[15] == 1
+
+def iterate_detector_at_level(item, depth = 0, level = 0):
+  """
+  Iterate through all panel groups or panels of a detector object at a given
+  hierarchy level
+  @param item panel group or panel. Use detector.hierarchy().
+  @param depth current dept for recursion. Should be 0 for initial call.
+  @param level iterate groups at this level
+  @return next panel or panel group object
+  """
+  if level == depth:
+    yield item
+  else:
+    for child in item:
+      for subitem in iterate_detector_at_level(child, depth+1, level):
+        yield subitem
+
+def iterate_panels(panelgroup):
+  """
+  Find and iterate all panels in the given panel group, regardless of the hierarchly level
+  of this panelgroup
+  @param panelgroup the panel group of interest
+  @return the next panel
+  """
+  if panelgroup.is_group():
+    for child in panelgroup:
+      for subitem in iterate_panels(child):
+        yield subitem
+  else:
+    yield panelgroup
+
+def id_from_name(detector, name):
+  """ Jiffy function to get the id of a panel using its name
+  @param detector detector object
+  @param name panel name
+  @return index of panel in detector
+  """
+  return [p.get_name() for p in detector].index(name)
+
+def get_center(pg):
+  """ Find the center of a panel group pg, projected on its fast/slow plane """
+  if pg.is_group():
+    # find the average center of all this group's children
+    children_center = matrix.col((0,0,0))
+    count = 0
+    for p in iterate_panels(pg):
+      children_center += get_center(p)
+      count += 1
+    children_center /= count
+
+    # project the children center onto the plane of the panel group
+    pgf = matrix.col(pg.get_fast_axis())
+    pgs = matrix.col(pg.get_slow_axis())
+    pgn = matrix.col(pg.get_normal())
+    pgo = matrix.col(pg.get_origin())
+
+    return (pgf.dot(children_center) * pgf) + (pgs.dot(children_center) * pgs) + (pgn.dot(pgo) * pgn)
+  else:
+    s = pg.get_image_size()
+    return matrix.col(pg.get_pixel_lab_coord((s[0]/2, s[1]/2)))