[docs][tests] clarify documents about authentication and filter. Test…

… --no-resolve-image for local images.

README.md

@@ -69,8 +69,8 @@ You can configure the most behaviors of *Gantry* via environment variables.
 | Environment Variable  | Default | Description |
 |-----------------------|---------|-------------|
 | GANTRY_SERVICES_EXCLUDED         | | A space separated list of services names that are excluded from updating. |
-| GANTRY_SERVICES_EXCLUDED_FILTERS | `label=gantry.services.excluded=true` | A space separated list of [filters](https://docs.docker.com/engine/reference/commandline/service_ls/#filter), e.g. `label=project=project-a`. Exclude services which match the given filters from updating. The default value allows you to add label `gantry.services.excluded=true` to services to exclude them from updating. Note that multiple filters will be logical **ANDED**. |
-| GANTRY_SERVICES_FILTERS          | | A space separated list of [filters](https://docs.docker.com/engine/reference/commandline/service_ls/#filter) that are accepted by `docker service ls --filter` to select services to update, e.g. `label=project=project-a`. Note that multiple filters will be logical **ANDED**. Also see [How to filters multiple services by name](docs/faq.md#how-to-filters-multiple-services-by-name). |
+| GANTRY_SERVICES_EXCLUDED_FILTERS | `label=gantry.services.excluded=true` | A space separated list of [filters](https://docs.docker.com/engine/reference/commandline/service_ls/#filter), e.g. `label=project=project-a`. Exclude services which match the given filters from updating. The default value allows you to add label `gantry.services.excluded=true` to services to exclude them from updating. Note that multiple filters will be logical **ANDED**. An empty string means no filters, as a result *Gantry* will not exclude any services. |
+| GANTRY_SERVICES_FILTERS          | | A space separated list of [filters](https://docs.docker.com/engine/reference/commandline/service_ls/#filter) that are accepted by `docker service ls --filter` to select services to update, e.g. `label=project=project-a`. Note that multiple filters will be logical **ANDED**. An empty string means no filters, as a result *Gantry* will update all services. Also see [How to filters multiple services by name](docs/faq.md#how-to-filters-multiple-services-by-name). |
 
 > NOTE: *Gantry* reads labels on the services not on the containers. The labels need to go to the [deploy](https://docs.docker.com/reference/compose-file/deploy/#labels) section, if you are using docker compose files to setup your services.
 

docs/authentication.md

@@ -1,6 +1,6 @@
-## Authentication
+# Authentication
 
-### Single registry
+## Single registry
 
 If you only need to login to a single registry, you can use the environment variables `GANTRY_REGISTRY_USER`, `GANTRY_REGISTRY_PASSWORD`, `GANTRY_REGISTRY_HOST` and `GANTRY_REGISTRY_CONFIG` to provide the authentication information. You may also use the `*_FILE` variants to pass the information through files. The files can be added to the service via [Docker secret](https://docs.docker.com/engine/swarm/secrets/).
 
@@ -10,7 +10,7 @@ If you only need to login to a single registry, you can use the environment vari
 
 > NOTE: *Gantry* uses `GANTRY_REGISTRY_PASSWORD` and `GANTRY_REGISTRY_USER` to obtain Docker Hub rate when `GANTRY_REGISTRY_HOST` is empty or `docker.io`. You can also use their `_FILE` variants. If either password or user is empty, *Gantry* reads the Docker Hub rate for anonymous users.
 
-### Multiple registries
+## Multiple registries
 
 If the images of services are hosted on multiple registries that are required authentication, you should provide a configuration file to the *Gantry* and set `GANTRY_REGISTRY_CONFIGS_FILE` correspondingly. You can use [Docker secret](https://docs.docker.com/engine/swarm/secrets/) to provision the configuration file. The configuration file must be in the following format:
 
@@ -30,17 +30,19 @@ If the images of services are hosted on multiple registries that are required au
 
 You can use `GANTRY_REGISTRY_CONFIGS_FILE` together with other authentication environment variables.
 
-You can login to multiple registries using the same Docker configuration. However if you login to the same registry with different user names for different services, you should use different Docker configurations.
+You can login to multiple registries using the same Docker configuration. For example you can set all the configurations to the default Docker configuration location `${HOME}/.docker/`. However if you login to the same registry with different user names for different services, you need to use different Docker configurations.
 
-### Select Docker configurations for services
+## Selecting Docker configurations for services
 
-If you login to a single registry using `GANTRY_REGISTRY_USER`, `GANTRY_REGISTRY_PASSWORD` and `GANTRY_REGISTRY_HOST` without setting `GANTRY_REGISTRY_CONFIG`, the default Docker configuration is used. You don't need to set anything extra for authentication.
+If you login to a single registry using `GANTRY_REGISTRY_USER`, `GANTRY_REGISTRY_PASSWORD` and `GANTRY_REGISTRY_HOST` **without** setting `GANTRY_REGISTRY_CONFIG`, the default Docker configuration is used. When using the default Docker configuration, you don't need to set anything extra for authentication.
 
- *Gantry* creates or updates the docker configurations based on the locations set via `GANTRY_REGISTRY_CONFIG`, `GANTRY_REGISTRY_CONFIG_FILE` or `GANTRY_REGISTRY_CONFIGS_FILE`.
+ *Gantry* creates or updates the docker configurations based on the locations set via `GANTRY_REGISTRY_CONFIG`, `GANTRY_REGISTRY_CONFIG_FILE` or `GANTRY_REGISTRY_CONFIGS_FILE`. They could be same as the the default Docker configuration location. When using the default Docker configuration, you don't need to set anything extra for authentication.
 
- You can use environment variable [`DOCKER_CONFIG`](https://docs.docker.com/engine/reference/commandline/cli/#environment-variables) to apply the same Docker configuration to all docker commands, i.e. to all services.
+The default Docker configuration location is `/root/.docker/` inside the container created based on the image built from this repository, because the default user is `root`. You can use environment variable [`DOCKER_CONFIG`](https://docs.docker.com/engine/reference/commandline/cli/#environment-variables) to explicitly set the default docker configuration location, which applies to all docker commands, i.e. to all services.
 
-You need to add label `gantry.auth.config=<configuration>` to particular services to tell which Docker configuration to use for authentication, when you use different configurations for different registries. When *Gantry* finds the label `gantry.auth.config=<configuration>` on services, it adds `--config <configuration>` to the Docker commands for the corresponding services.
+Optionally you can use different configurations for different services, for example when you want to login to the same registry with different user names. In this case, besides using different configuration values in `GANTRY_REGISTRY_CONFIG` and `GANTRY_REGISTRY_CONFIG_FILE`, you need to add the label `gantry.auth.config=<configuration>` on the particular services to tell which Docker configuration to use for authentication. When *Gantry* finds the label `gantry.auth.config=<configuration>` on services, it adds `--config <configuration>` to the Docker commands for the corresponding services to overrides the default configuration location.
+
+## Adding `--with-registry-auth`
 
 *Gantry* automatically adds `--with-registry-auth` to the `docker service update` command for services for the following cases.
 
@@ -49,9 +51,9 @@ You need to add label `gantry.auth.config=<configuration>` to particular service
   * when `GANTRY_REGISTRY_USER`, `GANTRY_REGISTRY_PASSWORD` are set, while `GANTRY_REGISTRY_CONFIG` is empty.
   * when the configuration from `GANTRY_REGISTRY_CONFIG` or `GANTRY_REGISTRY_CONFIGS_FILE` is same as the default Docker configuration location `${HOME}/.docker/` or the location specified by `DOCKER_CONFIG`.
 
-You can manually add `--with-registry-auth` to `GANTRY_UPDATE_OPTIONS` if it is not added automatically for your case. Without `--with-registry-auth`, the service will be [updated to an image without digest](https://github.com/shizunge/gantry/issues/53#issuecomment-2348376336), and you might get a warning "*image \<image\> could not be accessed on a registry to record its digest. Each node will access \<image\> independently, possibly leading to different nodes running different versions of the image.*"
+You can manually add `--with-registry-auth` to `GANTRY_UPDATE_OPTIONS` if it is not added automatically for your case. When `--with-registry-auth` is missing but the registry requires authentication, the service will be [updated to an image without digest](https://github.com/shizunge/gantry/issues/53#issuecomment-2348376336), and you will get a warning *"image \<image\> could not be accessed on a registry to record its digest. Each node will access \<image\> independently, possibly leading to different nodes running different versions of the image."*
 
-### Use an existing Docker configuration
+## Using an existing Docker configuration
 
 You can use an existing Docker configuration from the host machines for authorization when you run *Gantry* as a Docker service. You need to do the followings.
 
@@ -60,4 +62,4 @@ You can use an existing Docker configuration from the host machines for authoriz
 * Set the environment variable `DOCKER_CONFIG` on the *Gantry* container to specify the location of the Docker configuration folder inside the container. You can skip this step when you mount the folder to the default Docker configuration location `/root/.docker/` inside the container.
 * Add `--with-registry-auth` to `GANTRY_UPDATE_OPTIONS` manually.
 
-> Note that [`docker buildx imagetools inspect`](https://docs.docker.com/engine/reference/commandline/buildx_imagetools_inspect/) writes data to the Docker configuration folder `${DOCKER_CONFIG}/buildx`, which therefore needs to be writable. You can set `GANTRY_MANIFEST_CMD` to `manifest` to avoid writing to the Docker configuration folder.
+> Note that [`docker buildx imagetools inspect`](https://docs.docker.com/engine/reference/commandline/buildx_imagetools_inspect/) writes data to the Docker configuration folder `${DOCKER_CONFIG}/buildx`, which therefore needs to be writable. If you want to use `buildx` and mount the configuration files read-only, you could just mount the file `config.json` and leave the folder writeable. If you have to mount the entire folder read-only, you can set `GANTRY_MANIFEST_CMD` to `manifest` to avoid writing to the Docker configuration folder. Also see [Which `GANTRY_MANIFEST_CMD` to use](../docs/faq.md#which-gantry_manifest_cmd-to-use).

docs/development.md

@@ -1,6 +1,6 @@
-## Development
+# Development
 
-### Internal Configurations
+## Internal Configurations
 
 The following configurations are set automatically by *Gantry* internally. User usually does not need to touch them.
 

docs/faq.md

@@ -1,49 +1,51 @@
-## FAQ
+# FAQ
 
-### How does *Gantry* work?
+## How does *Gantry* work?
 
 Fundamentally *Gantry* calls [`docker service update`](https://docs.docker.com/engine/reference/commandline/service_update/) CLI and let docker engine [applies rolling updates to a service](https://docs.docker.com/engine/swarm/swarm-tutorial/rolling-update/).
 
 Before updating a service, *Gantry* will try to obtain the manifest of the image used by the service to decide whether there is a new image.
 
 At the end of updating, *Gantry* optionally removes the old images.
 
-### How to update standalone docker containers?
+## How to update standalone docker containers?
 
 *Gantry* only works for docker swarm services. If you need to update standalone docker containers, you can try [*watchtower*](https://github.com/containrrr/watchtower). *Gantry* can launch *watchtower* via `GANTRY_PRE_RUN_CMD` or `GANTRY_POST_RUN_CMD`. See the [example](../examples/prune-and-watchtower).
 
-### How to filters multiple services by name?
+## How to filters multiple services by name?
 
 It will not work by setting multiple filters with different names, because filters are logical **ANDED**.
 
 To filter multiple services, you can set a label on each service then let *Gantry* filter on that label via `GANTRY_SERVICES_FILTERS`. Or you can run multiple *Gantry* instances.
 
-### How to run *Gantry* on a cron schedule?
+Setting `GANTRY_SERVICES_FILTERS` to an empty string means no filters, as a result *Gantry* will update all services.
+
+## How to run *Gantry* on a cron schedule?
 
 You can start *Gantry* as a docker swarm service and use [`swarm-cronjob`](https://github.com/crazy-max/swarm-cronjob) to run it at a given time. When use `swarm-cronjob`, you need to set `GANTRY_SLEEP_SECONDS` to 0. See the [example](../examples/cronjob).
 
-### How to update services with no running tasks?
+## How to update services with no running tasks?
 
 As discussed [here](https://github.com/docker/cli/issues/627), the CLI will hang when running `docker service update` on a service with no running tasks. We must add `--detach=true` option to the `docker service update`.
 
 *Gantry* will check whether there are running tasks in a service. If there is no running task, *Gantry* automatically adds the option `--detach=true`. In addition to the detach option, *Gantry* also adds `--replicas=0` for services in replicated mode. You don't need to add these options manually.
 
-### Which `GANTRY_MANIFEST_CMD` to use?
+## Which `GANTRY_MANIFEST_CMD` to use?
 
 Before updating a service, *Gantry* will try to obtain the image's meta data to decide whether there is a new image. If there is no new image, *Gantry* skips calling `docker service update`, leading to a speedup of the overall process.
 
 `buildx`, the default value, should work in most cases. [`docker buildx imagetools inspect`](https://docs.docker.com/engine/reference/commandline/buildx_imagetools_inspect/) is selected as the default, because `docker manifest inspect` could [fail on some registries](https://github.com/orgs/community/discussions/45779). Additionally, `docker buildx imagetools` can obtain the digest of multi-arch images, which could help reduce the number of calling the `docker service update` CLI when there is no new images.
 
 `manifest` is kept for debugging purpose. The only known advantage of [`docker manifest inspect`](https://docs.docker.com/engine/reference/commandline/manifest_inspect/) is that it does not require the write permission to the [Docker configuration folder](https://docs.docker.com/engine/reference/commandline/cli/#configuration-files). If the Docker configuration folder is read-only, you need to use `manifest` to avoid permission deny errors.
 
-`none` can be used to disable the image inspection. One use case of `none` is that you want to add `--force` to the `docker service update` command via `GANTRY_UPDATE_OPTIONS`, which updates the services even if there is nothing changed. Another use case is to debug image inspection. Please report the bug through a [GitHub issue](https://github.com/shizunge/gantry/issues), thanks.
+`none` can be used to disable the image inspection. One use case of `none` is that you want to update services using local built images that are not available on a registry, in which case you want to add `--force` to the `docker service update` command via `GANTRY_UPDATE_OPTIONS`. `none` can also be used to debug image inspection.
 
-### Can *Gantry* report Docker Hub rate for non-anonymous account?
+## Can *Gantry* report Docker Hub rate for non-anonymous account?
 
 When checking Docker Hub rate, *Gantry* reads the Docker Hub credential only from `GANTRY_REGISTRY_PASSWORD` and `GANTRY_REGISTRY_USER`, or their `_FILE` variants. `GANTRY_REGISTRY_HOST` or its `_FILE` variant must be either empty or `docker.io`.
 
 If you need to login to multiple registries, you can use `GANTRY_REGISTRY_CONFIGS_FILE` together with `GANTRY_REGISTRY_PASSWORD` and `GANTRY_REGISTRY_USER`. Credentials in `GANTRY_REGISTRY_CONFIGS_FILE` will be used for services updating, but they won't be used for checking Docker Hub rate. See [Authentication](../docs/authentication.md) for more information.
 
-### How to use authorization from the host machines?
+## How to use authorization from the host machines?
 
-See [use an existing Docker configuration](../docs/authentication.md#use-an-existing-docker-configuration).
+See [using an existing Docker configuration](../docs/authentication.md#using-an-existing-docker-configuration).

docs/migration.md

@@ -1,4 +1,4 @@
-## Migration from shepherd
+# Migration from shepherd
 
 *Gantry* started to fix the following problems I found in [*shepherd*](https://github.com/containrrr/shepherd), then it became refactored and totally rewritten, with [abundant tests](../tests/README.md).
 
@@ -15,7 +15,7 @@ Although I have tried to keep backward compatibility, not all configurations in
 
 This guide helps you migrate from *shepherd* to *gantry* by highlighting the difference between them. Please refer to the [README](../README.md) for the full description of the configurations.
 
-### Equivalent or similar configurations
+## Equivalent or similar configurations
 
 | *Shepherd* Env | Equivalent or similar *Gantry* Env  | Enhancement |
 |----------------|-------------------------------------|-------------|
@@ -37,7 +37,7 @@ This guide helps you migrate from *shepherd* to *gantry* by highlighting the dif
 
 The label on the services to select config to enable authentication is renamed to `gantry.auth.config`.
 
-### Deprecated configurations
+## Deprecated configurations
 
 | *Shepherd* Env | Workaround |
 |----------------|------------|
@@ -46,7 +46,7 @@ The label on the services to select config to enable authentication is renamed t
 | WITH_NO_RESOLVE_IMAGE  | Manually add `--no-resolve-image` to `GANTRY_UPDATE_OPTIONS`. |
 | RUN_ONCE_AND_EXIT      | Set `GANTRY_SLEEP_SECONDS` to 0. |
 
-### New configurations
+## New configurations
 
 You can enable these new features of *Gantry* through new configurations.
 
@@ -74,7 +74,7 @@ You can enable these new features of *Gantry* through new configurations.
 
 Besides the global configurations via environment variables, you can apply a different value to a particular service via [labels](../README.md#labels).
 
-### License
+## License
 
 *Shepherd* is under [MIT license](https://github.com/containrrr/shepherd/blob/master/LICENSE)
 

src/lib-gantry.sh

@@ -1037,18 +1037,29 @@ _update_single_service() {
     _static_variable_add_unique_to_list STATIC_VAR_SERVICES_UPDATE_FAILED "${SERVICE_NAME}"
     return 1
   fi
-  local TIME_ELAPSED=
-  TIME_ELAPSED=$(time_elapsed_since "${START_TIME}")
   local PREVIOUS_IMAGE=
-  local CURRENT_IMAGE=
   PREVIOUS_IMAGE=$(_get_service_previous_image "${SERVICE_NAME}")
+  PREVIOUS_DIGEST=$(extract_string "${PREVIOUS_IMAGE}" '@' 2)
+  [ -z "${PREVIOUS_DIGEST}" ] && log DEBUG "After updating, the previous image ${PREVIOUS_IMAGE} of ${SERVICE_NAME} does not have a digest."
+  local CURRENT_IMAGE=
   CURRENT_IMAGE=$(_get_service_image "${SERVICE_NAME}")
+  CURRENT_DIGEST=$(extract_string "${CURRENT_IMAGE}" '@' 2)
+  [ -z "${CURRENT_DIGEST}" ] && log WARN "After updating, the current image ${CURRENT_IMAGE} of ${SERVICE_NAME} does not have a digest."
+  local TIME_ELAPSED=
+  TIME_ELAPSED=$(time_elapsed_since "${START_TIME}")
   if [ "${PREVIOUS_IMAGE}" = "${CURRENT_IMAGE}" ]; then
-    log INFO "No updates for ${SERVICE_NAME}. Use ${TIME_ELAPSED}."
-    return 0
+    # The same new and old images indicate that the image is still being used.
+    # Removing image would fail due to that.
+    if [ -n "${CURRENT_DIGEST}" ]; then
+      log INFO "No updates for ${SERVICE_NAME}. Use ${TIME_ELAPSED}."
+      return 0
+    fi
+    # This (i.e. no digest in both old and new image.) could happen when the service is updated to a local built image.
+  else
+    # Remove PREVIOUS_IMAGE only when it is no longer used.
+    _static_variable_add_unique_to_list STATIC_VAR_IMAGES_TO_REMOVE "${PREVIOUS_IMAGE}"
   fi
   _static_variable_add_unique_to_list STATIC_VAR_SERVICES_UPDATED "${SERVICE_NAME}"
-  _static_variable_add_unique_to_list STATIC_VAR_IMAGES_TO_REMOVE "${PREVIOUS_IMAGE}"
   log INFO "Updated ${SERVICE_NAME}. Use ${TIME_ELAPSED}."
   return 0
 }