Anycable and Kamal 2 healthchecks

[pjpires]

Hello,

First of all, thanks for the great work - we've been using anycable in production in a rails app for a couple of years now, and it's been a pretty smooth experience. 🙏

That said, the project's deployment setup is a bit dated, and still relies on capistrano. I'm trying to explore Kamal 2, but I'm struggling with making it work with anycable.

My understanding of this section of the docs is that kamal-proxy will perform the healthcheck on all "proxy roles" at the specified path (or the default one, which is /up I believe). This turned my attention to anycable for an healthcheck endpoint, and I realized there's a way to setup one. However, it runs as a separate rack server, at a different port from anycable itself. From my testing, I think that invalidates the possibility of using kamal 2, because it doesn't allow me to configure a specific port for the healthcheck, only the path (it will always use the defined app_port from the config).

This is the relevant section of my deploy.yml file at the moment:

# service, image ...

servers:
  web:
    - <%= ENV['HOST'] %>

  anycable-rpc:
    hosts:
      - <%= ENV['WEBSOCKET_HOST'] %>
    cmd: bundle exec anycable
    env:
      clear:
        ANYCABLE_HTTP_HEALTH_PORT: 54321
    proxy:
      hosts:
        - <%= ENV['WEBSOCKET_HOST'] %>
      app_port: 50051
      healthcheck:
        path: /health # <-- problem is here I think, the healtcheck will be attempted at :50051/health

proxy:
  ssl: true
  hosts:
    - <%= ENV['HOST'] %>

# registry stuff, not relevant...

env:
  secret:
    - RAILS_MASTER_KEY
    - POSTGRES_PASSWORD
  clear:
    DB_HOST: my-app-db
    REDIS_URL: "redis://my-app-redis:6379"
    HOST: <%= ENV['HOST'] %>
    ANYCABLE_RPC_HOST: "0.0.0.0:50051"
    ANYCABLE_REDIS_URL: "redis://my-app-redis:6379/0"
    ANYCABLE_WEBSOCKET_URL: "wss://<%= ENV['WEBSOCKET_HOST'] %>/cable"

accessories:
  db:
    image: postgres:16
    # postgres config...

  redis:
    image: redis:7.0
    # redis config...

  anycable-go:
    image: anycable/anycable-go:1.5
    host: <%= ENV['HOST'] %>
    port: "127.0.0.1:8080:8080"
    env:
      clear:
        ANYCABLE_HOST: "0.0.0.0"
        ANYCABLE_PORT: 8080
        ANYCABLE_RPC_HOST: "my-app-anycable-rpc:50051"
        ANYCABLE_REDIS_URL: "redis://my-app-redis:6379/0"

When I try to deploy/boot the servers with this configuration I get this:

docker stderr: Error: target failed to become healthy

I'm not too familiar with Kamal (or deploying with docker, really), so it might just be a problem in my config. Anyway, I thought I'd post here in case someone is able to help.

Related: @palkan, I saw that there are a few examples as draft PRs related to deploying with a few different configurations in https://github.com/anycable/anycable_rails_demo, would it be worthy to add one specifically for Kamal 2?

Thanks in advance!

[palkan]

Hey!

it will always use the defined app_port from the config

The thing is that the 50051 is used by gRPC, while the HTTP health server runs on a different port, because it's a separate interface. (For gRPC we have gRPC health checks, but those are not supported by Kamal).

However, I don't think you need to run AnyCable RPC as a server in Kamal's terms; it's more like a worker, a Sidekiq. I'm not sure if Kamal supports health checks for workers though.

Alternatively, you may try specify the health check server port as the app_port; you should be still able to expose the gRPC port within the internal Kamal's Docker network.

[palkan]

Thanks a lot for the details!

Well, it looks like Kamal isn't meant (yet) for such setups.

Have you considered Thruster, by the way? With it, you can pack all three components (Rails, RPC, AnyCable-Go) together. That's how most Kamal 1 users did, as far as I know.

[palkan]

Can't this trick be useful to have stable RPC name within the Docker network? https://x.com/skryukov_dev/status/1861117727924273512

[pjpires]

Hey @palkan! Totally, that was my conclusion too, coincidentally I also posted about it here yesterday. 😅 I was about to share my findings here, I think I've managed to tie up all the loose ends.

Here’s my current setup:

servers:
  web:
    - <%= ENV['HOST'] %>

  # sidekiq ...

  anycable-rpc:
    hosts:
      - <%= ENV['WEBSOCKET_HOST'] %>
    cmd: bundle exec anycable
    options:
      network-alias: my-app-anycable-rpc
      
# other settings

accessories:
  # db, redis ...
  anycable-go:
    image: anycable/anycable-go:1.5
    host: <%= ENV['WEBSOCKET_HOST'] %>
    env:
      clear:
        ANYCABLE_HOST: "0.0.0.0"
        ANYCABLE_PORT: 8080
        ANYCABLE_RPC_HOST: "my-app-anycable-rpc:50051"
        ANYCABLE_REDIS_URL: "redis://gooze-booze-api-redis:6379/0"

After discussing it with a friend, I realized that ports don’t need to be explicitly published to be accessible within the same Docker network. That was a helpful insight! I was able to remove the publish: option that had been causing issues with redeployment and container rotation.

I also added an alias to my deployment script that re-registers anycable-go with kamal-proxy (step 3 above), like this:

aliases:
  # predefined aliases, like console, shell, etc.

  add-anycable-go-to-proxy: |
    server exec docker exec kamal-proxy kamal-proxy deploy my-app-anycable-go-staging
    --target my-app-anycable-go:8080
    --host <%= ENV['WEBSOCKET_HOST'] %>
    --health-check-path /health
    --tls

Then, I call it from .kamal/hooks/post-deploy like so:

#!/bin/sh

kamal add-anycable-go-to-proxy

I’ve run through this a few times, and everything seems to be working fine when deploying from my development machine. Now, I’m tackling the process from CI, but I’m hopeful it will be a smaller challenge.

Thanks for your support!

[pjpires]

Have you considered Thruster, by the way? With it, you can pack all three components (Rails, RPC, AnyCable-Go) together. That's how most Kamal 1 users did, as far as I know.

BTW, regarding Thruster: I've actually added the base thruster gem to the project, so I'm using it already. At some point I noticed you had a fork because of this demo PR, but I wasn't successful putting it all together. In any case, given that I was able to make it work with RPC it should be more performant than the HTTP version (using thruster), right?

[palkan]

given that I was able to make it work with RPC it should be more performant than the HTTP version (using thruster), right?

Yeah, from performance perspective, running a gRPC server in a separate process is better; with HTTP, resources are shared between regular traffic and AnyCable RPC; which is fine for smaller installations, but can become a bottleneck at scale.

[palkan]

different configurations in https://github.com/anycable/anycable_rails_demo, would it be worthy to add one specifically for Kamal 2?

Yeah, we have Kamal on the list of demo ideas for this repo. Not sure when we'll have to do that though.

[pjpires]

That would be neat! Since kamal seems to be picking up some traction (specially with the latest v2 release), it might be useful for other people. I wouldn't mind helping out, but I seem to be pretty stuck myself right now. 😅