upload docker images: retry, take 2 (#2832)

As it turns out, 'skopeo --retry 5' doesn't actually help us. Wrap it in
bash, then.

To work around:

```
*** Uploading /tmp/tmp.kIKRERgZ1H/image to quay.io/wire/spar-integration:4.25.22Getting image source signatures

Copying blob f4f33343fcb5 skipped: already exists

Copying blob a3ab88edf03d skipped: already exists

Copying blob 9360a695c022 skipped: already exists

Copying blob 62d7b43f88a6 skipped: already exists

Copying blob 134eff2df9f9 skipped: already exists

Copying blob 8834895fc941 skipped: already exists

Copying blob 52a0756d3ab1 done  ======================>----------] 30.0MiB / 40.3MiB

Copying blob fa04d4e808c5 done  ---------------------------------] 8.0b / 190.0KiB

Copying blob 6c806be006f4 skipped: already exists

FATA[0004] trying to reuse blob sha256:95218c34e1598cf423f77062d98259bedcc19c9b8f4d937d0905895ee7b0242e at destination: too many requests to registry

make: *** [Makefile:242: upload-images] Error 1

make: Leaving directory '/tmp/build/80754af9/wire-server'
```

hack/bin/upload-image.sh

@@ -27,6 +27,39 @@ if [[ "${DOCKER_USER+x}" != "" ]]; then
     credsArgs="--dest-creds=$DOCKER_USER:$DOCKER_PASSWORD"
 fi
 
+# Retry a command with exponential backoff
+# quay.io sometimes rate-limits us, so try again.
+# Also, skopeo's retry logic doesn't properly work, look here if you want to see very badly written go code:
+# https://github.com/containers/skopeo/blob/869d496f185cc086f22d6bbb79bb57ac3a415617/vendor/github.com/containers/common/pkg/retry/retry.go#L52-L113
+function retry {
+    local maxAttempts=$1
+    local secondsDelay=1
+    local attemptCount=1
+    local output=
+    shift 1
+
+    while [ $attemptCount -le "$maxAttempts" ]; do
+        output=$("$@")
+        local status=$?
+
+        if [ $status -eq 0 ]; then
+            break
+        fi
+
+        if [ $attemptCount -lt "$maxAttempts" ]; then
+            echo "Command [$*] failed after attempt $attemptCount of $maxAttempts. Retrying in $secondsDelay second(s)." >&2
+            sleep $secondsDelay
+        elif [ $attemptCount -eq "$maxAttempts" ]; then
+            echo "Command [$*] failed after $attemptCount attempt(s)" >&2
+            return $status
+        fi
+        attemptCount=$((attemptCount + 1))
+        secondsDelay=$((secondsDelay * 2))
+    done
+
+    echo "$output"
+}
+
 tmp_link_store=$(mktemp -d)
 # Using dockerTools.streamLayeredImage outputs an executable which prints the
 # image tar on stdout when executed. This is done so we don't store large images
@@ -42,4 +75,4 @@ image_file="$tmp_link_store/image"
 repo=$(skopeo list-tags "docker-archive://$image_file" | jq -r '.Tags[0] | split(":") | .[0]')
 printf "*** Uploading $image_file to %s:%s\n" "$repo" "$DOCKER_TAG"
 # shellcheck disable=SC2086
-skopeo --insecure-policy copy --retry-times 5 $credsArgs "docker-archive://$image_file" "docker://$repo:$DOCKER_TAG"
+retry 5 skopeo --insecure-policy copy --retry-times 5 $credsArgs "docker-archive://$image_file" "docker://$repo:$DOCKER_TAG"