User:Areveny/sandbox

Design
Kafka was originally designed to handle the challenges of very large-scale log and event data. Earlier approaches to handling this data were based on scraping log files, for which specialized distributed log aggregators, including Flume and Scribe were later built. These systems collected and loaded log data for later analysis. Kafka was developed to combine the benefits of these log aggregators with a messaging system to facilitate real-time consumption of log events.

To achieve this, Kafka keeps a replicated write-ahead commit log and offers a pull-based messaging system for subscribers to read messages.


 * General abstractions.

Stateless broker
 * Specific implementation concepts