Add shallow checkout for azure pipelines

[Trzs]

• bootstrap.py always checks out the full repository
• the azure runner is severely disk limited, so the '.git' directories are deleted again
• kokkos needs git information to be build -> unwieldy work around to delete the '.git' dir in an extra step
• I propose to add --depth=1 to git clone commands in bootstrap.py, so that only the necessary data is downloaded in the first place
• can be reversed, where necessary, with git fetch --unshallow

Note to @nksauter, @bkpoon

[phyy-nx]

$ time git clone [email protected]:cctbx/cctbx_project.git
real	0m44.325s
$ du -hs cctbx_project/
255M	cctbx_project/

$ time git clone --depth=1 [email protected]:cctbx/cctbx_project.git
real	0m12.546s
$ du -hs cctbx_project/
157M	cctbx_project/

XFEL CI on Azure is so tight on space this kinda thing can help (plus it's good dinosaur management). But how does this affect developers? With depth=1, the old commit history is literally not there. Git log shows only the latest commit. So I think we'd only want this for Azure, right? Maybe a flag to bootstrap update that we can set in Azure?

[bkpoon]

How much more disk space is needed? There is a disk clean up step in

cctbx_project/.azure-pipelines/xfel/conda-linux.yml

Lines 112 to 128 in 9983275

	- script: |
	sudo mkdir -p /tmp/empty_dir
	sudo rsync --stats -a --delete /tmp/empty_dir/ /tmp/host_opt/hostedtoolcache
	for d in \
	lib/jvm \
	local/.ghcup \
	local/lib/android \
	local/share/powershell \
	share/dotnet \
	share/swift \
	; do
	echo Deleting $d
	sudo rsync --stats -a --delete /tmp/empty_dir/ /tmp/host_usr/$d
	done
	df -h
	displayName: Clean up host image
	continueOnError: true

You essentially have root access to the virtual machine on Azure so you can delete whatever you want. There is sort of a limit in that in this pipeline, you are already inside the Docker image, but you can always make other host directories writeable here

cctbx_project/.azure-pipelines/xfel/conda-linux.yml

Lines 27 to 33 in 9983275

	container:
	image: ${{ parameters.distribution }}:${{ join('.', parameters.version) }}
	options: --name ci-container --privileged
	volumes:
	- /usr/bin/docker:/tmp/docker:ro
	- /usr:/tmp/host_usr:rw
	- /opt:/tmp/host_opt:rw

For example, in some pipelines, I clean up more stuff and make a swap file

https://github.com/phenix-project/phenix-installer/blob/main/scripts/clean_linux.sh

But that is done in the normal Azure image, not the Docker image. I get about 47 GB free in / after this step. Also, you do not need to use the 14 GB partition, you can use any partition on the image.

I can probably help more next week once I get back.

[Trzs]

@phyy-nx to get the full git history, you can run either git fetch --unshallow or git pull --unshallow