In the first article in this RabbitMQ series we looked at what message queueing is and a brief look at RabbitMQ. In this article we will look in more detail at the AMQP messaging standard that underpins RabbitMQ.
RabbitMQ is built on top of the AMQP. This is a network protocol that enables client applications to communicate with a compatible messaging system.
A message broker works by receiving messages from a client (publisher) and that broker routes the message to a receiving application (consumer).
RabbitMQ currently supports version 0-9-1 of the AMQP protocol. With the AMQP protocol, a message is published by an application to an exchange. You can think of the exchange as a mailbox. The exchange then sends the message to a queue by using different rules called bindings. This is all within the message broker.
The message broker will then deliver the message from the queue to a consumer (the consumer pulls the message) that is subscribed to the queue. When a message is published to a queue, a publisher can specify various different message attributes. Some of these attributes will be used by the message broker but the rest is completely opaque to the broker and is only used by any applications that receive the message.
Due to network unreliability, applications could fail to correctly process messages, therefore the AMQP protocol has a mechanism for message acknowledgements. This means when a message is delivered to a consuming application, the consumer notifies the broker (either automatically or as soon as the application developer chooses to do so). When message acknowledgements are used, the message broker will only remove the message from the queue when it receives a notification for that message.
If you are using messages that are routed with a routingKey and the message cannot be routed anywhere, it can either be returned to the sender, dropped or, if configured, can be placed into a dead letter queue which is monitored. A publishing application will choose how to handle this situation by publishing messages by using certain parameters.
Exchanges are AMQP entities where messages are sent to the message broker. Exchanges take a message and then route it to one or more queues. The type of routing depends upon the exchange type used and different exchange rules called bindings. RabbitMQ supports AMQP 0-9-1 brokers which provides four exchange types. These are direct exchanges, fanout exchanges, topic exchanges, and headers exchanges.
Each exchange is also declared with a number of different attributes. The most important attributes to us are:
- Name: The name of the exchange.
- Durability: This flag determines whether or not messages sent to the exchange survive a broker/server restart by persisting the messages to disk.
- Auto-delete: This flag determines if the exchange is deleted when all of the queues have finished using it.
- Arguments: These are message broker-dependent.
AMQP message brokers contain a default exchange that is a direct exchange with no name (i.e., empty string) that has been predeclared. It has one special property that makes it useful for simple applications: every queue that is created is automatically bound to it with a routing key which is the same as the queue name.
For example, when you declare a queue with the name of “payment-requests,” the message broker will bind it to the default exchange by using “payment-requests” as the routing key. In other words, the default exchange makes it seem as if it is possible to directly deliver messages to queues even though that is not technically what is happening.
A direct exchange delivers messages to queues that are based on a message routing key. A direct exchange is ideal if you need to publish a message onto just one queue (like with a more traditional MSMQ setup) but you can also route messages onto multiple queues as well.
A direct exchange works as follows: a queue binds to the exchange with a routing key (payment-message, for example). Then, when a new message with a routing key of payment-message arrives at the direct exchange, the exchange routes the message to the queue if both routing keys match.
Direct queues are commonly used to distribute messages between multiple worker processes in a round robin manner. Later in the series, we’ll develop a sample that does exactly this.
A fanout exchange routes messages to all of the queues that are bound to it and the routing key is ignored. If 10 queues are bound to a fanout exchange, when a new message is published to that exchange, a copy of the message is delivered to all 10 queues. Fanout exchanges are ideal for the broadcast routing of messages.
This is in contrast to the direct exchange where the message is targeted to the relevant queue by using the routing key. So, if you want to broadcast a message to all of the queue consumers, the fanout exchange is what you want to use.
Some example uses of this might be:
- Sending online game scores to all players.
- Sending weather updates to all interested systems.
- Chat sessions between groups of people.
Topic exchanges route messages to one or many queues based upon matching between a message routing key and the pattern that was used to bind a queue to an exchange. The topic exchange type is often used to implement various publish/subscribe pattern variations. Topic exchanges are commonly used for the multicast routing of messages to different queues.
Topic exchanges have a broad set of use cases. Whenever a problem involves multiple consumers/applications that selectively choose which type of messages they want to receive, the use of topic exchanges should be considered.
A topic exchange works by using a wildcarded routing key. For example, in the context of a company departmental hierarchy, if you send a message to “all.*.*”, the message will be sent to all of the departments. If you sent a message to “all.payroll.*”, the message will only be sent to consumers who are interested in the payroll department.
The topic exchange is powerful and can behave like other exchanges already discussed.
When a queue is bound with the “#” (hash) binding key, it will receive all of the messages regardless of the routing key, in exactly the same manner as a fanout exchange.
When special characters “*” (star) and “#” (hash) aren’t used in bindings, the topic exchange will behave just like a regular direct exchange.
Some typical examples of this exchange might be:
- Sending messages to relevant departments in an organization.
- Stock prices for certain types of companies.
- Categorized news updates (e.g, business, technology, entertainment).
The headers exchange is for routing on multiple attributes that are expressed in headers. This means that the routing key is ignored for this type of exchange due to the fact that it can only express one piece of information. You can assign multiple header values that pass into the exchange and are routed to the queues.
In the diagram above, the first queue takes messages that are routed with the header values of ‘material = wood’ and ‘type = cupboard.’ The second queue takes messages that are routed with ‘material = metal’ and ‘type = filing cabinet.’
Header exchanges can be looked at as supercharged direct exchanges as they route based on header values, and they can be used as direct exchanges where the routing key does not have to be a string.
Queues in AMQP are similar to queues in any other messaging system. Messages are placed onto a queue and they work on a first in, first out (FIFO) basis. Queues have the following additional properties over exchanges:
- Name: The name of the queue.
- Durable: The queue and messages will survive a broker or server restart.
- Exclusive: The queue is used by only one connection and the queue will be deleted when that collection closes.
- Auto Delete: The queue is deleted when the consumer or subscriber unsubscribes.
Before you can use a queue, it must be declared. If the queue doesn’t already exist, it will be created. If the queue already exists, then redeclaring the queue will have no additional effect on the queue that already exists.
Queue names can be picked by the application or they can be automatically named by the broker that generates them. We will see examples of this later on in the series. Queue names can be up to 255 characters in length. If you want the broker to pick the queue name for you, then you don’t pass a name at the point where you declare the queue.
Queues can be made durable which means the queue is persisted to the disk. This only makes the queue persistent and not the messages. Queue durability means the queue will be redeclared once the broker is restarted. If you want the messages to be also persisted, then you have to post persistent messages. This is useful if you need the messages to remain intact if the broker or server is restarted. Making queues durable does come with additional overhead so you need to decide if you need this enabling but, if your application can’t be in a position where it can lose messages, then you need queue durability.
When you want to define rules that specify how messages are routed from exchanges to queues, you need to define bindings. Bindings may have an optional routing key attribute that is used by some exchange types to route messages from the exchange to the queue. The purpose of the routing key is to select certain messages published to an exchange to be routed to the bound queue. This means that the routing key acts like a filter. If an AMQP message cannot be routed to any queue because it does not have a valid binding from the exchange to the queue, then it is either dropped or returned to the publisher depending upon message attributes the publisher has set.
Storing messages in queues is all well and good but you need applications on the other side of the queues to consume those messages. Typically, applications will register as a consumer or subscribe to a queue. You can have more than one application registered as a subscriber to the queue, and this is a common usage scenario when you want to balance the load of applications feeding from the queue in high-volume scenarios.
When a consuming application acts on a message from the queue, it is possible that a problem could occur in that application which means that message is lost. Generally, when an application acts on a message, that message is removed from the queue but you may not want this to happen until you have successfully processed the message. The AMQP protocol gives you a couple of ways of defining when a message is removed from the queue.
- The message is removed once the broker has sent the message to the application.
- The message is removed once the application has sent an acknowledgement message back to the broker.
The first example uses an automatic acknowledgement from the application to remove the message whereas the second example uses an explicit acknowledgement. With the explicit acknowledgement, it is up to the application to decide when to remove the message from the queue. This could be when you have just received the message or after you have processed it.
If the consuming application crashes before the acknowledgement has been sent, then the message broker will try to redeliver the message to another consumer.
When an application processes a message, that processing may or may not succeed. If the processing fails for any reason (for example, there is a database time-out), then the consuming application can reject the message. When this happens, the application can ask the broker to discard the message or re-queue it. If there is only one consumer application subscribed to a queue, you need to make sure you do not create an infinite message delivery loop by rejecting and re-queuing a message from the same consumer over and over again.
This has been a whistle stop tour of the AMQP protocol but it has not been exhaustive. If you wish to read up on the full specification, then you can do so on the AMQP Working Group website.
In the next article we will look at installing and configuring RabbitMQ.