Replicate across Kafka clusters in other data centers
UReplicator provides the ability to replicate across Kafka clusters in other data centers. Instead of publishing to a single Kafka cluster, you can publish data to multiple regional Kafka clusters and aggregate it all in one Kafka cluster.
Kafka's MirrorMaker design (0.8.2) consumes data from a given regional Kafka cluster using a Kafka high-level consumer. With this design, rebalancing in the high level consumer affects all the topics being replicated via that Mirrormaker (due to an addition/deletion of topics, source cluster problems, network issues and so on). uReplicator thus offers the following:
- Stability: Rebalance only occurs during startup (when a node is added/deleted)
- Simple operations: Easy to scale up cluster, no server restart for whitelisting topics
- High throughput: Max offset lag is consistently 0
- Time SLA (~5 min)