Broadcasting options

[palkan]

Currently, AnyCable requires Redis as a dependency for the only one thing: sending messages from the application to the WebSocket server for broadcastings (ActionCable.server.broadcast ...).

The idea is to provide more options for the users and thus not to require Redis for everyone.

TODO adapters:

• HTTP REST – great for development/testing, small projects with the only one websocket server instance
• PostgreSQL pub/sub
• Redis Sentinel (see failover support anycable-rails#3)
• RabbitMQ
• NATS

Other possible options:

• UDP ?
• ...

[kobaltz]

Personally, I like having Redis as a dependency as most of my current applications use two Redis instances; persistent store and volatile. However, I'd always be open to something different if the maintainability/performance was the same or better. I would caution against PostgreSQL as a dependency as that would be more limiting than having a Redis dependency. While I do use PostgreSQL in some applications, we have to use other databases in other cases.

[maurcarvalho]

HTTP REST seems like an "out of external dependency" way to go.

@palkan can you say a little more about how do you imagine it?
will it be something like a http handler between WS and ActionCable.server.broadcast ...

[palkan]

@kobaltz The idea is to avoid additional dependency if it's possible. Most Rails applications already use Redis, so they are likely to be OK with it (and that's why for now Redis is the only option, btw).

But there are applications which don't use it. And that's the main reason why we're looking for alternatives.

There can also be one more reason: performance and scalability. For i-dunno-how-much-large apps, it would be better to use RabbitMQ or even Kafka.

We do not have to provide all possible options but an abstraction to build them (as separate gems).
On the other hand, to develop a good abstraction it would be useful to know these alternatives beforehand.

P.S. As you probably remember, Action Cable initially also depended on Redis, the community didn't like it.

[palkan]

@maurcarvalho Sure.

Agree, that's the easiest option (from the development point of view).

Assume that our WebSocket server implements HTTP endpoint to receive broadcast messages. Then we can write our #broadcast method like this:

def broadcast(channel, payload)
  Net::HTTP.post_form(AnyCable.config.http_endpoint, { "channel" => channel, "payload" => payload })
end

Of course, it's a very straightforward solution. It would be great to re-use connection ('keep-alive') of even have a connection pool, send requests asynchronously and deal with errors.
Maybe, we should put it to ActiveJob out-of-box.

Not so easy)

[AnthonySuper]

If you're already using GRPC, why not also use it to deliver messages to the server?

It can be something like:

service AnycableServer {
    rpc Broadcast (BroadcastingMesssage) returns (google.protobuf.Empty) {}
}

message Message {
    string channel = 1;
    string payload = 2;
}

Now, this would make a server a bit harder to implement, which is something we need to consider.

[palkan]

If you're already using GRPC, why not also use it to deliver messages to the server?

That's the reason:

... this would make a server a bit harder to implement

We have to run GRPC server as a part of our WebSocket server, which is not available yet for all languages (e.g. for Erlang).

And another problem: consider clustered configuration – several WebSocket servers, several application servers. Then we need a GRPC balancer – yet another problem.

[AnthonySuper]

Okay, gotcha. I didn't even consider the load balancing part, which is a pretty big issue now that I think about it.

An HTTP solution seems like it will likely be the best for now, in that case.

[pbrumm]

I think having nsq would be very interesting for the message passing.
http://nsq.io/components/nsqd.html

it enables the actioncable component to just put the message into a queue, and then each instance of the websocket server could register a channel on that queue with a name from a config file. This would allow multiple websocket servers to receive messages without concern for maintaining an active connection with the rails app.

the rails app could also be modified to just be reading messages from a queue with a single channel.

nsq has client libraries in many languages and is written in go.

[slayer]

What about MQTT ?

[palkan]

@slayer

What about MQTT ?

Although, MQTT-based broadcasting is possible, I think, it would too complicated for our use case. The main (IMO) feature of MQTT – quality of service – doesn't make sense in our case: if a WebSocket server is down and doesn't receive broadcast messages (through HTTP/Redis/queue), it's likely not to handle client connections too.

Broadcast messages a ephemeral from the WS server point of view. At least for now.

[jtoy]

love the idea of postgres pub/sub

[khalilovcmd]

Re:

HTTP REST – great for development/testing,

Nice! 👍

small projects with the only one websocket server instance

Just trying to understand more, if understand correctly, the main advantage of AnyCable is: to write a high-performant real-time application using Ruby as a language for implementing a business-logic.

Wouldn't depending on a HTTP transport medium null the performance benefits that is gained by AnyCable specifically for small projects?

I think it is also subjective of how small is a small project, but wondering to get more of your thoughts about it.

[palkan]

Wouldn't depending on a HTTP transport medium null the performance benefits that is gained by AnyCable specifically for small projects?

Yep, from the performance point of view there is no a huge benefit.
But you can still have benefits from the resources usage point of view.

how small is a small project

Yeah, that's a good question.
We should think about the number of simultaneous connections (peak and average) and the message rate/payload size.
I think, the threshold to start thinking about AnyCable (instead of just Action Cable) is somewhere between 500 and 1000 connections on average or 5k-10k during peak hours.

[jtoy]

What’s the largest known anycable project? We are testing now if we can support 60k concurrent connections. — Sent from my mobile

…

On Jan 23, 2019, at 10:34 AM, Vladimir Dementyev ***@***.***> wrote: Wouldn't depending on a HTTP transport medium null the performance benefits that is gained by AnyCable specifically for small projects? Yep, from the performance point of view there is no a huge benefit. But you can still have benefits from the resources usage point of view. how small is a small project Yeah, that's a good question. We should think about the number of simultaneous connections (peak and average) and the message rate/payload size. I think, the threshold to start thinking about AnyCable (instead of just Action Cable) is somewhere between 500 and 1000 connections on average or 5k-10k during peak hours. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[palkan]

What’s the largest known anycable project?

The largest project I'm aware of (and I'm not aware of the most of them, unfortunately) had ~10k concurrent connections (that's equipe.com, one year ago); they handled this with 4 1x dynos on Heroku (before switching to AnyCable they had 20 2x dynos for ActionCable).

We are testing now if we can support 60k concurrent connections.

Could you please share some information on your use case?

[khalilovcmd]

@palkan Thanks for the context! – I see you abstracted the broadcasting adapters already, I will give it a try with adding one of the broadcasting adapter options you listed above, I will tag you in a PR ☺️

[palkan]

@jtoy
Could you please share some information on your use case? I’ve added an issue to collect the feedback:
#68

[PeterLaudel]

hey we like the idea to add a rabbitmq adapter. Could you explain a bit more detailed which steps would be necessary to achive this? Is it right to also adapt the anycable-go app?

[palkan]

Yep, pub/sub adapter should be added to both RPC and WS server implementations.

We use a single stream/queue/channel to deliver messages from RPC to WS.
RPC server acts as publisher: it pushes a JSON-encoded command. Pubsub connection is initialized lazily in this case (during the first #broadcast call).

WS server (anycable-go) acts as subscriber: subscription is initialized on server start, messages are received, deserialized and passed to the app.

[ztsuperman]

Hi @palkan, Anycable is an amazing project! I'm trying to build a prototype for my work. I read through anycable-go's source code. It seems anycable-go only subscribes to one redis pubsub channel, which is anycable. According to official Actioncable guide, Actioncable creates multiple redis pubsub channels. Could you please confirm this is a difference between Anycable and Actioncable? If it is the case, could you please share more details on that? I have been reading source code and searching answers online. But, there's very little information about how Anycable/Actioncable uses redis channels. Thank you so much!

We are working for one of the largest tech company who is based on Ruby on Rails. Right now, we are building a concept proofing prototype using Anycable. Please help us, so we can persuade the leadership to invest resource on it. 🙏

[palkan]

Could you please confirm this is a difference between Anycable and Actioncable? If it is the case, could you please share more details on that?

Yes, AnyCable uses only a single Redis pub/sub channel. Unlike Action Cable, anycable-go manages the actual subscriptions by itself (see hub.go), we only need a single channel to get broadcasts from web apps to a WS server, which performs the actual retransmission. Check out https://docs.anycable.io/#/v1/misc/how_to_anycable_server

[ztsuperman]

Thank you so much for your quick answer! 🙏

[AlexeyMatskevich]

Are there any plans to support solid cable library?

[palkan]

What do you mean? Reading messages from Solid Cable tables in anycable-go to broadcast them to clients?

We consider adding database-backed features to AnyCable, but I suppose that we will go with a custom implementation (at least, different data schema to cover AnyCable-specific features). However, we may consider adding a compatibility layer with Solid Cable. The biggest problem here is that Solid Cable is still very new and I'm pretty sure its schema has not yet stabilized, so it's too early to talk about it.