User:What-a-good-jungle/sandbox

RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability. It is the third generation distributed messaging middleware open sourced by Alibaba in 2012. Through several years of technical improvement, RocketMQ is now capable of transferring trillions of concurrent online messages as of Alibaba's Nov. 11th Shopping Festival. On November 21, 2016, Alibaba donated RocketMQ to the Apache Software Foundation. Next year, on February 20, the Apache Software Foundation announced Apache RocketMQ as a Top-Level Project. Nowadays, RocketMQ is widely used by thousands of companies, and research and educational institutions around the world.

History
RocketMQ's evolution can be divided into three stages.

The first generation adopts the push mode in which data are stored in relational database. Low latency message delivery and distributed transactions meet Alibaba's demands and contribute to RocketMQ's wide adoption.

The second generation adopts the pull mode in which data storage is managed by file systems. Its performance on log collection is comparable to Kafka. Taking account of Alibaba's application scenarios, especially the requirement for high reliability during transaction processes, the second-generation RocketMQ does not blindly seek the throughput but also stresses stability and reliability. The long pulling mechanism makes the response time of the system no slower than that in the first generation.

The third generation, RocketMQ mainly uses the Pull mode along with some Push operations. Largely optimized, it has low latency and high performance, especially in high concurrency and massive amounts of data scenarios. Since 2012, RocketMQ has challenged the task of supporting Alibaba's annual Nov. 11th Global Shopping Festival multiple times. During the 2016 Festival, RocketMQ robustly provided stable infrastructure with a transfer throughput of more than one trillion messages.

Features
Much comparison have been made between all kinds of messaging solutions, there is a table demonstrates the comparison between RocketMQ, ActiveMQ and Kafka (Apache’s most popular messaging solutions according to awesome-java ) up to December 17, 2016:

Architecture


RocketMQ consists of four parts: name servers, brokers, producers and consumers. Each of them can be horizontally extended without a single point of Failure. As shown in image left.

NameServer Cluster
Name Servers provide lightweight service discovery and routing. Each Name Server records full routing information, provides corresponding reading and writing service, and supports fast storage expansion.

Broker Cluster
Brokers take care of message storage by providing lightweight TOPIC and QUEUE mechanisms. They support the Push and Pull model, contains fault tolerance mechanism (2 copies or 3 copies), and provides strong padding of peaks and capacity of accumulating hundreds of billion messages in their original time order. In addition, Brokers provide disaster recovery, rich metrics statistics, and alert mechanisms, all of which are lacking in traditional messaging systems.

Producer Cluster
Producers support distributed deployment. Distributed Producers send messages to the Broker cluster through multiple load balancing modes. The sending processes support fast failure and have low latency.

Consumer Cluster
Consumers support distributed deployment in the Push and Pull model as well. It also supports cluster consumption and message broadcasting. It provides real-time message subscription mechanism and can meet most consumer requirements. RocketMQ’s website provides a simple quick-start guide to interested users.

Applications
There are at least five aspects Apache RocketMQ could relate to:


 * The integration of heterogeneous systems；
 * Decoupling between applications;
 * Backbone in the event-driven mechanism and complex event architecture model;
 * Data replication channels;
 * Integration with flow calculation engine.

Open Source and Community Maintenance
The RocketMQ team have done much to active the community. and the team has adopted a strategy similar to Apache's other top-level projects as the team devotes itself to the high quality of its products. A series of standardized software development processes, such as milestone discussions, features design, code testing, peer reviews, integration tests, release discussions, and release announcements have been conducted in the RocketMQ version planning. On the other hand, the team interacts with the open source community through activities like line Meet-ups, Workshops, ApacheCon and Code Marathon, to attract new contributors and committers.

The team also produces a commercial distribution called Aliware MQ. It supports the TCP, HTTP and MQTT protocols, enhances the maintenance tools of ecological integration (including visualization of the trajectory, resource reporting statistics monitoring and alerts, as well as Kafka integration). Designed for enterprise, it has multi-region deployment with high availability disaster recovery characteristics on the public cloud and private cloud.

Rewards
2016 China’s most popular open source software award

2017 China’s most popular open source software award

12th CJK(China-Japan-South Korea) open source software outstanding technology award

Enterprises that use RocketMQ
Those use RocketMQ: • Alibaba Group

• Ant Financial

• HuaWei

• DiDi Global

• WeBank

• IQiyi

• China Minsheng Bank

• SF Express

• IFLYTEK

• China Telecom

• Midea Group

• Mogujie

• GOME.COM.CN

• Pactera

• Mobike

• Yunpian.com

• ShiHai Group

• Tree Financial

• YH Soft

• fenqile.com

• EGOVA

• ShouQI Group

• Decent

History
The development of RocketMQ can be divided into three stages.

The first generation uses the push mode in data transportation, and relational database in data storage. It shows low latency in message delivery and meets the command of a typical E-commerce platform with distributed transactions. Then many industry began to adopt it.

The second generation uses the pull mode in data transportation, and file system in data storage. It paid more attention to stability and reliability, and shows a comparable performance to the first generation in response time and Kafka on log collection.

The third generation combines the Pull mode with some Push operations. It inherits the advantages of the first and second generation, and shows high performance in concurrency and massive amounts of data scenarios. It's the key for the success of 2016 Alibaba's Nov. 11th Shopping Festival.

Features
Much comparison have been made between all kinds of messaging solutions, especially between Kafka and RocketMQ, there is a table demonstrates the comparison between RocketMQ, ActiveMQ and Kafka(the three most popular messaging solutions ) up to December 17, 2016: Because the characteristics of high performance, high reliability and high real time ability, more and more efforts have been make to the combination of RocketMQ and other protocol component in all kinds of messaging scenarios.

Architecture
RocketMQ consists of four parts: name servers, brokers, producers and consumers. Each of them can be horizontally extended without a single point of Failure. As shown in image left.

NameServer Cluster
The lightweight component for service discovery and they can be used to read and write routing information. Each one records global information, and supports fast storage expansion.

Broker Cluster
They use lightweight TOPIC and QUEUE mechanisms to manage data storage. To realize fault tolerance, 2 copies or 3 copies of data are provided. And Client can get message in Push and Pull model. In addition, disaster recovery and rich metrics statistics are also supported.

Producer Cluster
Producers can be distributed deployed, and messages from producers to brokers can be balanced through multi-path. In addition, fast failure and low latency are supported.

Consumer Cluster
Consumers can also be distributed deployed in the push and pull model, and they can subscribe message real-time, consume message in the unit of cluster. Message broadcasting is also supported.

Applications
There are at least five aspects Apache RocketMQ could relate to:


 * The integration of heterogeneous systems；
 * Decoupling between applications;
 * Backbone in the event-driven mechanism and complex event architecture model;
 * Data replication channels;
 * Integration with flow calculation engine.

Community Maintenance
The RocketMQ team have done much to active the community. Meetups, Workshops, ApacheCon and Code Marathon are regular held in BeiJing, ShenZhen, HangZhou to to attract new contributors and committers. As for version management, a series of standardized software development processes are adopted. The latest version is 4.2.0, and 4.3.0 is on the way. More information can be reached in here.

Papers

 * A MQTT Protocol Message Push Server Based on RocketMQ

Rewards

 * 2016 China’s most popular open source software award
 * 2017 China’s most popular open source software award
 * 12th CJK(China-Japan-South Korea) open source software outstanding technology award

Enterprises that use RocketMQ
Those use RocketMQ:• Alibaba Group

• Ant Financial

• HuaWei

• DiDi Global

• WeBank

• IQiyi

• China Minsheng Bank

• SF Express

• IFLYTEK

• China Telecom

• Midea Group

• Mogujie

• GOME.COM.CN

• Pactera

• Mobike

• Yunpian.com

• ShiHai Group

• Tree Financial

• YH Soft

• fenqile.com

• EGOVA

• ShouQI Group

• Decent

History
The development of RocketMQ can be divided into three stages.

The first generation uses the push mode in data transportation, and relational database in data storage. It shows low latency in message delivery and meets the command of a typical E-commerce platform with distributed transactions.

The second generation uses the pull mode in data transportation, and file system in data storage. It paid more attention to stability and reliability, and shows a comparable performance to the first generation in response time and Kafka on log collection.

The third generation combines the Pull mode with some Push operations. It inherits the advantages of the first and second generation, and shows high performance in concurrency and massive amounts of data scenarios.

Features
Much comparison have been made between all kinds of messaging solutions, and it is widely known that when the number of topics increases dramatically, the throughput of RocketMQ dropped much less than Kafka.

Because the characteristics of high performance, high reliability and high real time ability, more and more efforts have been make to the combination of RocketMQ and other protocol component in all kinds of messaging scenarios such as MQTT. It is

Architecture
RocketMQ consists of four parts: name servers, brokers, producers and consumers. Each of them can be horizontally extended without a single point of Failure. As shown in image left.

NameServer Cluster
The lightweight component for service discovery and they can be used to read and write routing information. Each one records global information, and supports fast storage expansion.

Broker Cluster
They use lightweight TOPIC and QUEUE mechanisms to manage data storage. To realize fault tolerance, 2 copies or 3 copies of data are provided. And Client can get message in Push and Pull model. In addition, disaster recovery and rich metrics statistics are also supported.

Producer Cluster
Producers can be distributed deployed, and messages from producers to brokers can be balanced through multi-path. In addition, fast failure and low latency are supported.

Consumer Cluster
Consumers can also be distributed deployed in the push and pull model, and they can subscribe message real-time, consume message in the unit of cluster. Message broadcasting is also supported.

Applications
There are at least five aspects Apache RocketMQ could relate to:


 * The integration of heterogeneous systems；
 * Decoupling between applications;
 * Backbone in the event-driven mechanism and complex event architecture model;
 * Data replication channels;
 * Integration with flow calculation engine.

Community Maintenance
The RocketMQ team have done much to active the community. Meetups, Workshops, ApacheCon and Code Marathon are regular held in BeiJing, ShenZhen, HangZhou to to attract new contributors and committers. The OpenMessaging benchmarking suites are currently available for the RocketMQ and it makes RocketMQ keep the pace with the global standard for distributed messaging. As for version management, a series of standardized software development processes are adopted. The latest version is 4.2.0, and 4.3.0 is on the way. More information can be reached in here.

Awards

 * 2016 China’s most popular open source software award
 * 2017 China’s most popular open source software award
 * 12th CJK(China-Japan-South Korea) open source software outstanding technology award