2019.06.20

2019.06.20

Contributors: Chris Barnes, Albert Cardona, Andrew Champion, Stephan Gerhard, Pat Gunn, Tom Kazimiers, William Patton

Notes

• A virtualenv update is required.

• All *.pyc files in the folders django/applications/catmaid/migrations/, django/applications/performancetests/migrations/ and django/applications/pgcompat/migrations/ need to be deleted.

• Python 3.7 is now supported.

• Heads-up: The next CATMAID version will require Postgres 11, PostGIS 2.5 and Python 3.6 or 3.7.

• Should migration 0057 fail due a permission error, the Postgres extension "pg_trgm" has to be installed manually into the CATMAID database using a Postgres superuser: sudo -u postgres psql -d <catmaid-db> -c 'CREATE EXTENSION pg_trgm;'

• CATMAID's version information changes back to a plain git describe format. This results overall in a simpler setup and makes live easier for some third-party front-ends, because the commit counter is included again. The version display is now the same git describe format for both regular setups and Docker containers.

• Tile loading is clamped to (0,0) again, i.e. there are no negative tile coordinates anymore by default. If you need them, set the respective stack's metadata field to {"clamp": false}.

• To write to CATMAID through its API using an API token, users need to have now dedicated "API write" permission, called "Can annotate project using API token" in the admin UI. To restore the previous behavior (regular annotate permission allows API write access) the settings.py variable REQUIRE_EXTRA_TOKEN_PERMISSIONS can be set to False. This is done as a safety measure to prevent accidental changes through automation.

• If R based NBLAST is used, make sure to execute to update all dependencies: manage.py catmaid_setup_nblast_environment.

• The main documentation on catmaid.org has now a place for widget specific documentation as well. Only a few widgets have been updated yet, but more will follow.

Features and enhancements

Tracing tool:

• Add a new Tracing Tool icon button to compute the distance between two nodes on a skeleton. Respects virtual nodes.

• The globally nearest node can now be selected and brought into view using Alt + G, as opposed to selecting the nearest node in the current section using the G key without modifier.

• Using the H shortcut now without an active skeleton, the most recently edited node of the current user (Alt: anyone) in the last active skeleton will be selected. Using Shift will look in all skeletons.

• Using Shift + Alt + Click with a connector selected, will will now create a presynaptic node. This is consistent with the already existing Shift + Alt + Click behavior with a treenode selected, which creates a presynaptic connector.

Graph widget:

• GraphML files can now be imported, positions and colors are respected. This is useful if layouting is done in e.g. Gephi and coloring should be done in CATMAID. The help page explains a possible workflow for this.

• The button "Group equally colored" in the "Main" tab will group all skeletons with the same color into a single group. The user is asked for group names for each color.

3D viewer:

• A new synapse coloring mode has been added: polyadicity. The number of partner nodes for each connector is color coded (for synaptic connectors, this is the number of postsynapses). The colors and ranges can be configured through the "Polyadicity colors" button in the "Shading parameters" tab. This is basically a configurable version of an absolute "N with partner" coloring.

• Branches with leaf nodes tagged "not a branch" can now be collapsed using the 'Collapse "not a branch"' checkbox in the Skeleton Filters tab.

• The visibility of radius information can now be controlled using the "Show radius" checkbox in the "Views settings" tab.

• History animations can now be exported in full length without requiring to guess the number of frames for the export. The animation export dialog will show an additional checkbox ("Complete history") if a history animation should be exported. If complete history is enabled, CATMAID will export the complete history of the exported skeletons.

• Animations can now be exported as stream directly to a file, which allows for much larger exports (32GB maximum at the moment).

• Fractional rotations are now allowed in the animation export.

• The default time per rotation in the animation export is set to 15 seconds now, slowing down the default by a factor of 3, which makes it easier to look at.

• Stack Z slices can now be animated. Configurable are the change frequency and the change step in terms of sections. This is available for animation exports as well.

• Stack Z slices can now be thresholded to replace a background color with another color. If enabled and the sum of all channels is in a configurable range [a,b] it will be replaced with another color.

• The rotation time for animations can now specified in seconds rather than angular distance.

• The background color is now fully adjustable through the "backgorund" button in the "View settings" tab.

• Basic support for Virtual Reality is now available on Windows platforms using a Mixed Reality / SteamVR.3D setup and Firefox >= v65. To enable, check the VR checkbox in the "View" tab and click the "Enter" button right next to it.

Skeleton history widget:

• A basic view of the change of a set of skeleton IDs over time based on all nodes that are part of a given skeleton ID or that have been in the past.

• Skeleton history can also be used with past skeleton IDs to see into what skeleton they changed (if any).

• All past and present treenodes with a passed in skeleton ID are tracked through the complete history and their path of skeleton ID changes is recorded along with the number of treenodes following a given skeleton path.

• The widget shows a graph from origin skeletons to the final skeleton IDs in every available path, summing the treenode counts for each contributing path.

• Existing skeletons are colored in yellow, past skeletons are colored in cyan. Selected skeletons are colored green.

• Ctrl+Click on skeleton will select it and go to the closest location in it. Shift+Click allows selecting multiple skeletons. All selected skeletons are available through the Skeleton Source interface.

Node and skeleton filters:

• Filter rules support now an "invert" option during creation, which allows to create filters that include everything but whatever is matched by a particular filter strategy. This can be useful e.g. during neuron review to only look at segments that have been created by people other than oneself or connectivity everywhere excluding a particular compartment.

Measurement table:

• Node filters are now supported. Like in other widgets, the respective panel can be opened through the funnel icon in the widget title bar. Measurements are displayed for all independent fragments of a skeleton after a set of node filter rules has been applied.

• With the help of the "Sum fragments" toggle all fragments that result from a node filter application can be aggregated into one row by summing individual values.

Remote CATMAID instances

• Some tools in CATMAID gained support to communicate with other CATMAID instances, e.g. the Landmark Widget (see below). To enable this functionality, remote CATMAID instances can now be added to the user settings.

• The Settings Widget contains now a section labeled "Other CATMAID instances". It allows to manage a list of other CATMAID servers based on their URL, an API key and optional HTTP authentication. This information is stored under an alias.

Landmarks:

• Clicking on the name of a landmark group in the Landmarks tab will now open the group member edit dialog that was previously accessible by clicking any landmark name. Clicking a landmark name will now show a new dialog, which allows to specify of which landmark groups the clicked landmark is a member of.

• The color of each active display transformation can be established separately in the respective table row. Color changes are now also visible in the 3D Viewer.

• It is now possible to load skeletons and landmarks from other CATMAID instances. Point matches are done on the basis of matching landmark names. To enable the UI for this, check the "Source other projects" checkbox in the Display tab.

• With the "other projects" UI enabled and a remote CATMAID instance configured in the Setting Widget (see above), it is now possible to select a remote CATMAID instance from the "Source remote" dropdown menu. Alternatively, the local instance can be selected if another project from the same instance should be used.

• Next the source project in the selected instance needs to be specified. This list is updated when a different remote instance is selected.

• Skeletons from a remote instance are collected through annotations. The respective annotation has to be entered in the "Source skeleton annotation". With the help of the "Preview" button it is possible to load the matching skeletons from their remote CATMAID to inspect if the correct ones are selected.

• As a last step for the remote data configuration, the source landmark group has to be defined. This list is updated if the source project changes. Landmarks from this group are mapped to the selected target group. The matching is done by name, i.e. no landmarks can have the same names in a group.

• Adding such a transformation adds it to the list at the bottom of the widget, just like with regular transformations and they can be used in the same way. The can be shown in 3D Viewers, superimposed on the Tracing Layer and used in NBLAST queries from the Neuron Similarity Widget (see below).

• The new checkbox 'Multiple mappings' displays additional user interface elements to add multiple source and target landmark group mappings for a single display transformation. Point matches are created independently for each pair and only merged for finding the final transformation.

• Using a transformation is now optional (but enabled by default). To disable landmark transformations, uncheck the "Apply transformation" checkbox. This allows e.g. to display skeletons from remote CATMAID instances without modification.

Neuron Similarity:

• The computation of the default "mean" normalized similarity scores is now considerably faster.

• The new "Top N" option allows to store only the Top N best matches in the result set. For "mean" normalization, it will also only compute the mean score for the top N forward hits (mean is the average between forward and backward score). A value of zero disables this cutoff (default). This cutoff can be used speed up computation for very large sets of neurons or large point clouds.

• The new "Reverse" option allows to rank similar objects by there reverse score. That means that the stored score for a particular object pair is target versus query rather than query versus target. NBLAST uses the query neuron as reference, and thereforoe the reverse option can make a difference as long as no "mean" scoring is used.

• It is now possible to use display transformations defined in the Landmark Widget that reference another CATMAID server as their data source. They can be used both as source and target for NBLAST comparisons. This makes it essentially possible to compare skeletons from another CATMAID project, possibly on another CATMAID server, to local skeletons using NBLAST.

• Similarity matrices can now include transformed skeletons in their set of similar objects. To do so, create a Display Transformation in the Landmark Widget that represents the wanted transformation and the select it from the "Sim. transformed skeletons" drop down in the Configurations tab of the Neuron Similarity Widget when creating a similarity matrix.

• Similarity matrices can now be created with more control over which skeletons are looked at as similar. The drop down menu "Similarity mode" allows to select different groupings of the input skeletons (and transformed skeletons). It is for instance possible to group a neuron and its contralateral partner that has been transformed to the side of the first neuron as similar. A confirmation dialog presents the computed grouping before actually creating a similarity matrix.

• Similarity matrices can now be imported from CSV files, optionally including the distance binning and dot binning as header row and column. As part of the import, the distance binning can be scaled to adjust for differences in datasets. This is available through the "Create from CSV file" button in the Configurations tab.

• Similarity query results can now be used as skeleton source in other widgets, if the target type of the query are skeletons.

System check widget:

• A subset of important database statistics are now displayed if a user has the "can_administer" permission in a project. These statistics include e.g. cache hit ratio, temporary files, requested checkpoints and replication lag. For most of them a rule of thumb suggestion on how a value should behave is provided as well.

Extension: CATMAID-autoproofreader:

• This extension is written by Will Patton and has to be installed separately, because it requires additional setup steps for segmentation data handling. Details can be found here: https://github.com/pattonw/CATMAID-autoproofreader.

• This tool suggests locations in a skeleton reconstructions where branches might be missing or wrongly connected. Users can step through these suggestions to fix potential problems. After reviewing a node it can be marked as reviewed for future reference.

• In the CATMAID front-end the autoproofreader provides a widget that relies on a compute server to handle the computations involved with automatically proofreading a neuron reconstruction. Proofreading jobs are performed asyncronously and might take a few minutes to complete.

Administration:

• A grid based node query cache can now be used to speed up tracing data retrieval by precomputing intersection results. This can be useful for larger field of views in big data set. It supports multiple levels of detail per grid cell and can therefore provide a somewhat uniform sampling when limiting the number of displayed nodes. The sorting of this LOD configuration can be controlled so that e.g. only the top N largest skeletons will be shown per cell with growing LOD. The documentation has more details. This cache can be kept up updated using two extra worker processes that listen to database changes, implemented as management commands catmaid_spatial_update_worker and catmaid_cache_update_worker.

• The database can now emit notifications on tracing data changes using the "catmaid.spatial-update" channel and when grid cache cells get marked dirty on the "catmaid.dirty-cacche" channel. These can be subscribed to using Postgres' LISTEN command. To enable emitting these events and let cache update workers work, set SPATIAL_UPDATE_NOTIFICATIONS = True in settings.py. This is disabled by default, because sending events for spatial cache updates can add slightly more time to spatial queries, which will mainly be relevant for importing and merging large skeleotons.

• Projects can now be deleted along with the data that reference them (e.g. treenodes, ontologies, volumes). To do so select the projects to delete in the admin project list and select "Delete selected" from action drop down.

• Users can now be set to an inactive state after a specified amount of time. This time is configured for a user group and it applies to users if they are members of such a group. A message can optionally be displayed as well as users to contact that could potentially help. This is configurable as "Group inactivity period" in the admin view. Users to contact for support, can be either configured there or from individual user views.

CLI Exporter:

• If both required annotations and exclusion annotations are provided, the importer will now only exclude a skeleton if it is not also annotated with a sub-annotation of the required annotations set. This behavior can be disabled and exclusion can be enforced when a skeleton is annotated with the exclusion annotation or one of its sub-annotations. To do so, use the new "--exclusion-is-final" switch.

• Volumes can now be exported by using the --volumes switch. By default all volumes of the exported project will be included. This can be further constrained by using --volume-annotation <annotation-name> arguments.

• The way she exported objects are specified through the command line interface changed. Instead of writing e.g. --notreenodes to not import treenodes from a source, --treenodes false has to be used now. This is the case for treenodes, connectors, tags and annotations. The defaults (when the argument is not provided) stay the same. To explicitly disable the export of a type false, no, f, n and 0 can be provided as argument, e.g. -treenodes false or --users n. For a positive parameter use true, yes, t, and 1.

CLI Importer:

• Precompute materializations (edges, connectors) explicitly only for imported data, which improves performance in typical scenarios (import is small compared to existing data). If the old behavior of recomputing everything in the target project should be used in cases where a full data base is imported, the --update-project-materializations switch can be used.

• The way she imported objects are specified through the command line interface changed. Instead of writing e.g. --notreenodes to not import treenodes from a source, --treenodes false has to be used now. This is the case for treenodes, connectors, tags and annotations. The defaults (when the argument is not provided) stay the same. To explicitly disable the import of a type false, no, f, n and 0 can be provided as argument, e.g. -treenodes false or --users n. For a positive parameter use true, yes, t, and 1.

• Imported usernames can now be mapped to existing usernames and mapped selectively by using one or more parameters of the form --username-mapping="import-username=target-username". The example would map all references to the user import-username in the imported data to the existing user target-username. The mapping is performed even with --map-users off.

• It is now possible to import volumes from CATMAID export files and other projects..

Miscellaneous:

• Tracing layer: a minimum skeleton length can now be specified in the layer options, causing the Tracing Layer to show only nodes of skeletons of at least this cable length (nm).

• Tracing layer: a minimum number of nodes can now be configured to only show skeletons of at certain minimum size (just like the existing minimum cable length constraint). This is configurable through the layer configuration, accessible by clicking the blue/white box in the lower left corner of stack viewers.

• Settings widget: visibility group conditions can now be inverted.

• Confirming a radius selection (Shift + Y) can now also be done using the Enter key.

• Neuron name searches and annotations should be much faster on larger instances.

• Basic support for touch screens (e.g. phones and tablets) is now implemented.

• CATMAID can now export files (e.g. CSVs, videos, etc.) of much larger size than before, if "Save exported files in streaming mode" is enabled in the Settings Widget. To make this work, the browser settings (chrome://settings) "Ask where to save each file before downloading" has to be enabled.

• Context aware help: by clicking the new question mark icon in the upper right corner, a context aware help dialog can be displayed on top of all other CATMAID tools. It provides some general guidance for the intended workflow with individual tools. This can be enable to be displayed by default as part of the Settings Widget. The display of this can also be enabled in a link to a view by appending &help=true. This will be added automatically if a help dialog is open during URL creation.

• Connectivity widget: the new "List links' button will open a Connector List with all links of the selected neurons.

• Selection table: the new checkbox "Link visibility" allows to change the behavior of the visibility checkboxes. So far the pre/post/text/meta select-all controls were only affecting the respective visibility options of visible/selected skeletons (default). With unlinked visibility pre/post/text/meta can also be controlled for all neurons, regardless of their visibility. This allows e.g. to only show synapses, but no skeletons.

• General search widget: a more flexible table is now used for display, which allows sorting, filtering and pagination.

• Neuron dendrogram: select currently active node by default when loading the dendrogram for the active skeleton.

• The right end of the status bar contains now a button to toggle the visibility of the top toolbars.

• Updating the copy of client side annotations will only request new data if there are actually new annotations. This makes loading of e.g. the Neuron Search widget faster.

• Boss databases can now be used as tile source type so that image data is loaded from them. More details: https://docs.theboss.io/docs/image.

• Docker: HTTP basic authentication can be configured by using the environment variables HTTP_AUTH_ENABLED, HTTP_AUTH_USER and HTTP_AUTH_PASS in the web container of the docker-compose.yml file (an example is given).

• Performance: selecting the closest node in a skeleton should now be faster.

Bug fixes

• A race condition has been fixed that could in rare cases lead to inconsistent skeleton IDs in a skeleton that is merge into different skeletons by different requests.

• Landmark transformations: fix Moving Least Square transformations for skeleton fragments outside of source group bounding box.

• The Connectivity Graph Plot draws now individual bars in each sub-plot side-by-side and uses the same neuron names as other widgets (from CATMAID's neuron name service).

• Information based on historic information like the history animation in the 3D Viewer and the Neuron History widget includes merge information now correctly. Before, merge nodes were represented twice in a time slice that included the merge.

• The default connector type is now properly persisted as a setting.

• Neuron search: no duplicate entries are shown anymore, which could result e.g. when sub-annotations where allowed.

• Neuron similarity: when showing result skeletons in 3D, skeletons are now appended to the Selection Table of a 3D Viewer, rather than the 3D Viewer directly.

• Connectivity widget: the auto-connectivity CSV will now use formatted neuron names as column and row headers.

• Connectivity widget: the 'Export CSV' button will now respect name and annotation filters.

• Graph widget: ungrouping of one-element groups is now allowed.

• Graph widget: merging of grouped skeletons is now handled properly.

• 3D viewer: the depth test for connector partner spheres is now performed correctly and spheres should be rendered in the correct Z order.

• 3D viewer: mouse controls now work correctly in fullscreen mode.

• 3D viewer: the suggested width and height of the animation export are now by default a factor of two. This is required by the WebM encoder. Off values are reduce by one.

• 3D viewer: spatial select works now also without the active node highlighted.

• 3D viewer: no error is shown any more when attempting to move while in radius edit mode.

• 3D viewer: landmark transformed skeletons show now also radius spheres.

• Connector table: the section and tag columns are now part of the CSV export.

• Tracing tool: unneeded node updates are removed from initial tracing layer loading.

• Selection table: don't try to load missing skeletons from JSON file.

• Settings widget: default values for tracing layer skeleton limits can now be configured under Tracing Overlay > Tracing layer skeleton filters.

• The numpad delete key is now recognized as regular delete (if numlock is off).

• SWC neuron import: providing a neuron ID for an import works again and will now also set an optionally provided neuron name. Additionally, it is now required that a user importing skeleton data for an existing neuron ID has the rights to edit the skeleton instance as well as all its treenodes.

• Volume widget: volumes/meshes with annotation can now be removed properly.

• Volume widget: editing a volume by double clicking its table entry works again.

• Neuron dendrogram: correctly reload skeleton if it changes as a result of a split or merge. If a dendrogram node is selected, the respective skeleton will be reloaded after a split, even if its ID changed.

• The Notification Table can be opened again without errors.

• Exporter: connector links are now exported properly if the parameter --original-placeholder-context is used.

• Docker: the docker-compose setup now uses Postgis 2.5 internally and therefore allows upgrades from Postgres versions < 10 with Postgis 2.4.

• Docker: stale Postgres PID files will now be removed during a database upgrade in a docker-compose setup. PID files without actually running database processes prevented some updates before.

• Initial setup: create_configuration.py now also prints the media files directory as part of the webserver example configuration. This is the folder where e.g. some exported files are made available.

API changes

Additions

• GET /{project_id}/nodes/nearest: Replaces POST /{project_id}/node/nearest. The parameters are the same, but the API allows now to look globally for the nearest node in the project, if no skeleton ID or neuron ID is provided.

• GET /{project_id}/nodes/most-recent: Replaces POST /{project_id}/node/nearest. A skeleton_id parameter can still be provided, but now also a user_id parameter is available to further constrain.

Modifications

• POST|GET /{project_id}/node/list: Offers a new optional parameter "min_skeleton_length", which can be used to constrain the returned neurons to only those of at least this cable length.

• POST|GET /{project_id}/pointclouds/: Offers a new optional parameter "order_by", which accepts the strings 'id' and 'name' to define in what order the list of pointclouds should be retuned (default: id).

• POST /{project_id}/landmarks/{landmark_id}/: Offers a new optional parameter "group_ids", an array of integers, which allows to set the landmark group memberships of a specific landmark. The new boolean parameter "append_memberships" allows to only append new group IDs as memberships, without removing any. Otherwise the whole set of memberships is replaced.

• POST /{project_id}/skeletons/import: The new parameter skeleton_id makes it possible to request a particular skeleton ID during import, just like it is done for neurons using neuron_id. If a skeleton or neuron with this ID exists already, a new object is created and the existing one is not touched. If an error should be raised instead, set the auto_id parameter to false. If instead the passed in IDs should replace existing data, the force parameter can be set to true. Both options apply to both neurons and skeletons.

• POST /{project_id}/skeletons/connectivity/csv: The new optional parameter names makes it possible to pass in a mapping of skeleton IDs to names used in the CSV export as column and row headers. If this parameter is not provided, the plain skeleton IDs will be used as it was done before. If it is provided, it has to be a list of two-element lists, each of the form [, ], which provides the mapping.

• POST /{project_id}/skeletons/connectivity: The source_skeleton_ids parameter can now also be specified in regular form format (multiple arguments with the exact same name), rather than only the square braces style.

Deprecations

None.

Removals

• POST /{project_id}/node/nearest: Replaced with GET /{project_id}/nodes/nearest (note the plural of nodes).

• POST /{project_id}/node/most-recent: Replaced with GET /{project_id}/nodes/most-recent.