-
Notifications
You must be signed in to change notification settings - Fork 182
Publishing a new release
Quick summary of steps:
-
In the lead-up to release
- Generate changelog
- Prepare announcement post for community forum
- Ensure there are no merge conflicts - Merge to master
- Publish the release
-
After release is published
- Check generation of binary packages
- Upload anaconda packages
- Add and upload MSYS2 package to MRtrix3 repo
- Verify binary installers
- Verify updated docs on the documentation site
- Upload new container to DockerHub -
Once everything checks out
- Post release announcement on community forum
- Post on social media, etc.
Use the fetch_changelog script to automate the process. This produces a first cut of the changelog based on the merge commits from the last tag on the branch currently checked out on your repo.
To use this:
- [on initial setup] clone the fetch_changelog repo
- navigate the top-level folder of your main local MRtrix3 repo (from where you will be generating the changelog)
-
[on initial setup] create the
fetch_changelog_settings
file with the following content, edited as appropriate (you'll need to generate your own GitHub personal authentication token:repo = 'MRtrix3/mrtrix3' user = 'jdtournier' password = 'gngkasgndfgknldsafnfknxgkjd'
- invoke
fetch_changelog
and redirect the output to file../fetch_changelog/fetch_changelog > changelog.txt
- copy/paste the contents of the file to the community forum (or any other markdown editor), and edit as appropriate.
-
create a post in the staff category of the forum, to ensure the formatting is as it will be when published, and for other developers to comment on and edit as required.
-
if the changelog is sufficiently long to warrant a separate post rather than a reply to the announcement, insert link to changelog post in the announcement (the URL will need to be updated when released, since the changelog will likely be copy/pasted into a new post to clean up its history).
You will need to generate a new git
branch, within which the version information will be updated (more on this below). The construction of this branch will depend on the nature of the release:
-
If publishing a macro update (e.g.
3.0.x
), containing only bug fixes, then create a new branch based off ofmaster
; -
If publishing a micro update (e.g.
3.x.0
), containing feature additions and/or changes to behavior, then create a new branch based off ofdev
.
For the rest of this document, this new branch will be referred to as "next_release
"; but any non-reserved branch name can be used.
For a micro update, make sure that next_release
can be merged to master
without having to resolve merge conflicts. Attempting to create a pull request on GitHub to merge from next_release
to master
should show this information both prior to, and after, creation of the request. If such conflicts exist, this is typically resolved by:
- Merge
master
intonext_release
; - Manually resolve any conflicts that arise;
- Push the updated branch to GitHub.
This is very fiddly given the number of interactions that need to be handled properly. The main issues to deal with are:
- the version number in
core/version.h
needs to be updated (this is the version number that thebuild
script falls back to in case nogit
history is available). - the version of the code checked out during container building needs to correspond to the new tag, so that containers built based on the code at a particular tag will contain code compiled from that same tag.
- the most recent tag name needs to be updated to match: the build script will fail if they do not match (if the
git
history is available). - the tag needs to correspond to the actual merge commit on
master
that will constitute the release. - the documentation needs to be updated since the help page of some commands will make reference to the online documentation, and the release version will be substituted in these help pages (to ensure no broken links, etc.).
Getting all this right is tricky. This is the approach I recommend - note that it requires admin access for the final push:
-
Make sure everything is up to date:
git fetch --all git checkout next_release git pull
-
Edit files
core/version.h
,Dockerfile
andSingularity
with desired version information and commit - but do not push (indeed none of the steps below are pushed to GitHub until step 8):vim core/version.h
The
Dockerfile
andSingularity
files will normally point tomaster
, but need to be hard-coded to the exact tag name for that commit to make sure the release recipes always build the release version. Make sure to change these frommaster
to the release tag; these will be reverted back tomaster
immediately after release (later in the instructions).vim Dockerfile # Update content of environment variable "MRTRIX3_GIT_COMMITISH" vim Singularity # Update "git clone" call
Then commit the changes:
git add core/version.h Dockerfile Singularity git commit -m "updated version information"
-
Tag the resulting commit with the same version information using a signed or annotated commit (so that this is what shows up with
git describe
), e.g. (edit as appropriate):git tag -s 3.0.0 -m "tagged version 3.0.0"
-
Build and update the docs, then commit the changes:
./build ./docs/generate_user_docs.sh git add docs git commit -m "updated documentation to match version change"
-
Merge
next_release
intomaster
:git checkout master git pull git merge next_release
Hopefully this won't generate any merge conflicts - these should have been resolved well in advance of reaching this point...
-
Delete the previous tag, and re-create a signed tag with the same label on the merge commit from the previous step, e.g. (edit as appropriate):
git tag -d 3.0.0 git tag -s 3.0.0 -m "tagged version 3.0.0"
-
Merge
master
back ontonext_release
, ready to push for a final round of CI:git checkout next_release git merge master
The
git merge
operation should result in a fast-forward merge that leave the commit SHA unmodified. You should find thatHEAD
,master
,next_release
,origin/next_release
, and your tag all point to exactly the same commit - but notorigin/master
(check withgit log --decorate
). -
Push
next_release
to GitHub for a final round of CI. This is important since it can catch issues introduced by the previous steps, and highlight any omissions, etc.. Best to make sure everything builds correctly before pushing out a release that immediately fails...git push --follow-tags
and wait until the CI completes. Note that you need to push the tag as well, otherwise the CI builds will fail.
-
Once all checks have passed, push the same commit directly to
master
(this can only be done by admin users). You can't merge using the GitHub interface since it'll create a merge commit, which would mean the tag will not longer point to the tip ofmaster
:git checkout master git push --follow-tags
This needs to be done on the GitHub releases page. The tag you pushed in the previous steps should already show up. Edit this tag, give it a name and description (you can draw some inspiration from previous releases), and make sure it is not marked as pre-release.
Once created, the relevant GitHub Actions should trigger automatically to generate the packages. You can check progress on this page.
-
GitHub Actions should generate 4 assets (at the time of writing) that will be uploaded to the release:
- macOS application package installer
- conda package for macOS
- conda package for Linux
- MSYS2 package for MSYS2
If any of these fail, this will need to be investigated and fixed.
The macOS and Linux conda packages will need to be pushed to the MRtrix3 channel. This is not trivial, and needs to be done using the anaconda
tool, within a working conda
install:
-
[on initial setup]
conda install anaconda-client
-
[on initial setup, or as required]
conda activate
-
[on initial setup, or as required]
anaconda login
This assumes that you have an account with anaconda, and that you are a member of the MRtrix3 organisation. - download the two conda package files (
conda-linux-mrtrix3-XXX-XXX_0.tar.bz2
andconda-macos-XXX-XXX_0.tar.bz2
) - strip the
conda-linux-
andconda-macos-
prefixes from the files.
This is important: the next step will not work if the filenames are not of the form 'package-version-checksum_num.tar.bz2', and the command provides no information at all about the failure (or indeed that there even was a failure...). - upload both packages one at a time:
anaconda upload -u MRtrix3 mrtrix3-XXX-XXX_0.tar.bz2
Hopefully that should work OK.
This requires as many users and developers as possible to try installing the packages according to our own instructions, both on existing installs and on pristine environments if possible (although realistically we're not going to be able to check all of these scenarios). The main thing is to check that the packages install when they should, and that they work on the different platforms.
Verify updated docs on the documentation site
If everything looks OK, activate the new version in the builds section of readthedocs, to make sure it appears in the list of available versions (not just as latest
)
-
Access the code corresponding to the new tag:
git checkout 3.#.#
-
Locally build the Docker container:
docker build . -t mrtrix3
-
Push the container to DockerHub with the corresponding tag:
docker tag mrtrix3 mrtrix3/mrtrix3:3.#.# docker login docker push mrtrix3/mrtrix3:3.#.#
-
If the changelog is sufficiently long to warrant a standalone post, copy/paste the contents of the changelog into a fresh topic in the staff section, and once satisfied that it's good to go, re-categorise it to the 'uncategorised' section. This makes it available to view, but won't send notifications to users.
-
Copy/paste the contents of the announcement into a fresh topic in the Announcements section, and make sure the URL to the changelog points to the updated changelog created in the previous step. If the changelog is short, it can be posted as a reply to this topic.
-
Close both the announcement and changelog topics (prevents further comments), and pin them globally.
-
Make sure that the community forum announcement appears on the website as most recent announcement post. The script that does this runs once a day, but can be run manually by logging onto the DigitalOcean instance that hosts the forum, and running the website update script:
ssh [email protected] /etc/cron.daily/auto_gen_blog_post
You'll obviously need the right credentials to do this...
-
Verify that the announcement shows up correctly on website, and fix up any issues with images and/or emojis, etc.
-
Add links to announcement and changelog on forum to the GitHub release description.
While it is important for consistency of container builds over time that those recipes pull the exact tagged version of code corresponding to the version of the recipe, anyone who clones the master
branch and builds a container from one of the recipe files would expect the code compiled within the container to correspond to the current contents of master
. As such, following a tag, it is recommended to revert the container recipes back to pulling code from the master
branch:
git checkout master
git checkout -b revert_recipes_to_master
vim Dockerfile # Set content of environment variable "MRTRIX3_GIT_COMMITISH" back to "master"
vim Singularity # Revert "git clone" call to clone "master" only
git add Dockerfile Singularity
git commit --author="MRtrixBot <[email protected]>" -m "Revert container recipes to pull master"
git push --set-upstream origin revert_recipes_to_master
Then generate a Pull Request to merge this change into master
.
🍻