-
Notifications
You must be signed in to change notification settings - Fork 170
Suro server
Suro server consists of four components: ThriftServer, MessageSetProcessor, MessageRouter, and Sink.
Suro server is a THsHaServer server with TNonblockingServerTransport. The ThriftServer receives messages of TMessageSet type from Suro client and puts them in a queue managed by a MessageSetProcessor.
Similar to Suro's asynchronous client, a MessageSetProcessor can choose to use a memory queue or a file-backed queue. The advantage of file-backed queue is that messages are written to disk first, so they will not get lost even if Suro server terminates unexpectedly.
MessageRouter dispatches messages to sinks according to their routing key and filtering rules. User can define message routing map as Json string.
Sink is the destination of the data. It can be local file(LocalFileSink), remote file(RemoteFileSink such as S3FileSink), or other data service such as Kafka, Elasticsearch, and Cassandra. Currently, the following sinks are open sourced:
- LocalFileSink: text file or Hadoop sequence file is supported.
- S3FileSink: It embeds LocalFileSink and upload the file to AWS S3.
- HdfsFileSink: It embeds LocalFileSink and upload the file to HDFS.
- KafkaSink: Sends messages to Kafka 0.8 broker.
We may also later open source sinks for ElasticSearch, HTTP/HTTPs endpoints, and Cassandra.