uReplicator

Replicate across Kafka clusters in other data centers

Fork me on Github

uReplicator

Replicate across Kafka clusters in other data centers

UReplicator provides the ability to replicate across Kafka clusters in other data centers. Instead of publishing to a single Kafka cluster, you can publish data to multiple regional Kafka clusters and aggregate it all in one Kafka cluster.

Kafka's MirrorMaker design (0.8.2) consumes data from a given regional Kafka cluster using a Kafka high-level consumer. With this design, rebalancing in the high level consumer affects all the topics being replicated via that Mirrormaker (due to an addition/deletion of topics, source cluster problems, network issues and so on). uReplicator thus offers the following:

  1. Stability: Rebalance only occurs during startup (when a node is added/deleted)
  2. Simple operations: Easy to scale up cluster, no server restart for whitelisting topics
  3. High throughput: Max offset lag is consistently 0
  4. Time SLA (~5 min)