-
Notifications
You must be signed in to change notification settings - Fork 170
Suro server
Suro server consists of four parts: ThriftServer, MessageQueue, MessageRouter, Sink.
Suro server is THsHaServer server with TNonblockingServerTransport. ThriftServer is receiving MessageSet from Suro client and put in MessageQueue.
As Suro asynchronous client has two types of internal message queue, Suro server also has the same types of MessageQueue including memory and file backed queue. We can take advantage of file queue's persistence when Suro server ends unexpectedly, we can keep data not consumed to sinks.
Sink is the destination of the data. It can be local file(LocalFileSink), remote file(RemoteFileSink such as S3FileSink), other data pipeline such as Kafka, Elasticsearch, Cassandra. Currently, the following sinks are provided:
- LocalFileSink: text file or Hadoop sequence file is supported.
- S3FileSink: It embeds LocalFileSink and upload the file to AWS S3.
- Kafka0.7Sink: It sends messages to Kafka 0.7 broker
MessageRouter is dispatching messages to sinks according to their routing key. User can define message routing map as Json string.