, have worked with machine learning or large-scale data pipelines, likelihood is you’ve used some form of queueing system.
Queues let services seek advice from one another asynchronously: you send off work, don’t wait around, and let one other system pick it up when ready. This is crucial when your tasks aren’t quick — think long-running model training jobs, batch ETL pipelines, and even processing requests for LLMs that take minutes per query.
So why am I writing this? I recently migrated a production queueing setup to RabbitMQ, bumped into a bunch of bugs, and located that documentation was thin on the trickier parts. After a good little bit of trial and error, I assumed it’d be value sharing what I learned.
Hope you can see this handy!
A fast primer: queues vs request-response model
Microservices typically communicate in two styles — the classic request–response model, or the more flexible queue-based model.
Imagine ordering pizza. In a request–response model, you tell the waiter your order after which wait. He disappears, and thirty minutes later your pizza shows up — but you’ve been left in the dead of night the entire time.
In a queue-based model, the waiter repeats your order, gives you a number, and drops it into the kitchen’s queue. Now you realize it’s being handled, and also you’re free to do something else till the chef gets to it.
That’s the difference: request–response keeps you blocked until the work is finished, while queues confirm immediately and let the work occur within the background.
What’s Rabbit MQ?
RabbitMQ is a preferred open-source message broker that ensures messages are reliably delivered from producers (senders) to consumers (receivers). First released in 2007 and written in Erlang, it implements AMQP (Advanced Message Queuing Protocol), an open standard for structuring, routing, and acknowledging messages.
Consider it like a post office for distributed systems: applications drop off messages, RabbitMQ sorts them into queues, and consumers pick them up when ready.
A typical pairing within the Python world is Celery + RabbitMQ: RabbitMQ brokers the tasks, while Celery staff execute them within the background.
In containerised setups, RabbitMQ typically runs in its own container, while Celery staff run in separate containers that you could scale independently.
How it really works at a high level
Your app desires to run some work asynchronously. Since this task might take some time, you don’t want the app to take a seat idle waiting. As a substitute, it creates a message describing the duty and sends it to RabbitMQ.
- Exchange: This lives inside RabbitMQ. It doesn’t store messages but just decides where each message should go based on rules you set (routing keys and bindings).
Producers publish messages to an exchange, which acts as a routing intermediary. - Queues: They’re like mailboxes. Once the exchange decides which queue(s) a message should go to, it sits there till it’s picked up.
- Consumer: The service that reads and processes messages from a queue. In a Celery setup, the Celery employee is the patron — it pulls tasks off the queue and does the actual work.
Once the message is routed right into a queue, the RabbitMQ broker pushes it out to a consumer (if one is out there) over a TCP connection.
Core components in Rabbit MQ
1. Routing and binding keys
Routing and binding keys work together to make a decision where a message finally ends up.
- A routing secret is attached to a message by the producer.
- A binding secret is the rule a queue declares when it connects (binds) to an exchange.
A binding defines the link between an exchange and a queue.
When a message is distributed, the exchange looks on the message’s routing key. If that routing key matches the binding key of a queue, the message is delivered to that queue.
A message can only have one routing key.
A queue can have one or multiple binding keys, meaning it could actually listen for several different routing keys or patterns.
2. Exchanges
An exchange in RabbitMQ is sort of a traffic controller. It receives messages, doesn’t store messages, and it’s key job is to make a decision which queue(s) the message should go to, based on rules.
There are several forms of exchanges, each with its own routing style.
2a) Direct exchange
Consider a direct exchange like a precise address delivery. The exchange looks for queues with binding keys that exactly match the routing key.
- If just one queue matches, the message will only be sent there (1:1).
- If multiple queues have the identical binding key, the message can be copied to all of them (1:many).
2b) Fanout exchange
A fanout exchange is like shouting through a loudspeaker.
Every message is copied to all queues sure to the exchange. The routing keys are ignored, and it’s at all times a 1:many broadcast.
Fanout exchanges will be useful when the identical message must be sent to 1 or more queues with consumers who may process the identical message in alternative ways.
2c) Topic exchange
A subject exchange works like a subscription system with categories.
Every message has a routing key, for instance "order.accomplished”. Queues can then subscribe to patterns corresponding to "order.*”. Which means that each time a message is expounded to an order, it should be delivered to any queues which have subscribed to that category.
Depending on the patterns, a message might find yourself in only one queue or in several at the identical time.
There are two essential special cases for binding keys:
*(star) matches exactly one word within the routing key.#(hash) matches zero or more words.
Let’s illustrate this to make the syntax alot more intuitive.
2nd) Headers exchange
A headers exchange is like sorting mail by labels as an alternative of addresses.
As a substitute of taking a look at the routing key (like "order.accomplished"), the exchange inspects the headers of a message: These are key–value pairs attached as metadata. As an example:
x-match: all, priority: high, type: email→ the queue will only get messages which have eachpriority=highandtype=email.x-match: any, region: us, region: eu→ the queue will get messages where at the very least one of the conditions is true (region=usorregion=eu).
The x-match field is what determines whether all rules must match or anyone rule is enough.
Because multiple queues can each declare their very own header rules, a single message might find yourself in only one queue (1:1) or in several queues without delay (1:many).
Headers exchanges are less common in practice, but they’re useful when routing will depend on more complex business logic. For instance, it is advisable to deliver a message provided that customer_tier=premium, message_format=json, or region=apac .
2e) Dead letter exchange
A dead letter exchange is a security net for undeliverable messages.
3. A push delivery model
Which means that as soon as a message enters a queue, the broker will push it out to a consumer that’s subscribed and prepared. The consumer doesn’t request messages and as an alternative just listens on the queue.
This push approach is great for low-latency delivery — messages get to consumers as soon as possible.
Useful features in Rabbit MQ
Rabbit MQ’s architecture enables you to shape message flow to suit your workload. Listed below are some useful patterns.
Work queues — competing consumers pattern
You publish tasks into one queue, and many consumers (eg. celery staff) all hearken to that queue. The broker delivers each message to precisely one consumer, so staff “compete” for work. This implicitly translates to easy load-balancing.
In the event you’re on celery, you’ll need to keep worker_prefetch_multiplier=1 . What this implies is that a employee will only fetch one message at a time, avoiding slow staff from hoarding tasks.
Pub/sub pattern
Multiple queues sure to an exchange and every queue gets a copy of the message (fanout or topic exchanges). Since each queue gets its own message copy, so different consumers can process the identical event in alternative ways.
Explicit acknowledgements
RabbitMQ uses explicit acknowledgements (ACKs) to ensure reliable delivery. An ACK is a confirmation sent from the patron back to the broker once a message has been successfully processed.
When a consumer sends an ACK, the broker removes that message from the queue. If the patron NACKs or dies before ACKing, RabbitMQ can redeliver (requeue) the message or route it to a dead letter queue for inspection or retry.
There may be, nonetheless, a crucial nuance when using Celery. Celery does send acknowledgements by default, nevertheless it sends them early — right after a employee receives the duty, before it actually executes it. This behaviour (acks_late=False, which is the default) implies that if a employee crashes midway through running the duty, the broker has already been told the message was handled and won’t redeliver it.
Priority queues
RabbitMQ has a out of the box priority queueing feature which lets higher priority messages jump the road. Under the hood, the broker creates an internal sub-queue for every priority level defined on a queue.
For instance, if you happen to configure five priority levels, RabbitMQ maintains five internal sub-queues. Inside each level, messages are still consumed in FIFO order, but when consumers are ready, RabbitMQ will at all times attempt to deliver messages from higher-priority sub-queues first.
Doing so implicitly would mean an increasing amount of overhead if there have been many priority levels. Rabbit MQ’s docs note that though priorities between 1 and 255 are supported, values between 1 and 5 are highly beneficial.
Message TTL & scheduled deliveries
Message TTL (per-message or per-queue) mechanically expires stale messages; and delayed delivery is out there via plugins (e.g., delayed-message exchange) if you need scheduled execution.
Easy methods to optimise your Rabbit MQ and Celery setup
Whenever you deploy Celery with RabbitMQ, you’ll notice a number of “mystery” queues and exchanges appearing within the RabbitMQ management dashboard. These aren’t mistakes — they’re a part of Celery’s internals.
After a number of painful rounds of trial and error, here’s what I learned about how Celery really uses RabbitMQ under the hood — and easy methods to tune it properly.
Kombu
Celery relies on Kombu, a Python messaging framework. Kombu abstracts away the low-level AMQP operations, giving Celery a high-level API to:
- Declare queues and exchanges
- Publish messages (tasks)
- Devour messages in staff
It also handles serialisation (JSON, Pickle, YAML, or custom formats) so tasks will be encoded and decoded across the wire.
Celery events and the celeryev Exchange
Celery includes an event system that tracks employee and task state. Internally, events are published to a special topic exchange called celeryev.
There are two such event types:
- Employee events eg.
employee.online,employee.heartbeat,employee.offlineare at all times on and are lightweight liveliness signals. - Task events, eg.
task-received,task-started,task-succeeded,task-failedthat are disabled by default unless the-Eflag is added.
You’ve gotten wonderful grain control over each forms of events. You’ll be able to turn off employee events (by turning off gossip, more on that below) while turning on task events.
Gossip
Gossip is Celery’s mechanism for staff to “chat” about cluster state — who’s alive, who just joined, who dropped out, and sometimes elect a pacesetter for coordination. It’s useful for debugging or ad-hoc cluster coordination.
By default, Gossip is enabled. When a employee starts:
- It creates an exclusive, auto-delete queue only for itself.
- That queue is sure to the
celeryevtopic exchange with the routing key patternemployee.#.
Because every employee subscribes to each employee.* event, the traffic grows quickly because the cluster scales.
With N staff, each publishes its own heartbeat, and RabbitMQ fans that message out to the opposite N-1 gossip queues. In effect, you get an N × (N-1) fan-out pattern.
In my setup with 100 staff, that meant a single heartbeat was duplicated 99 times. During deployments — when staff were spinning up and shutting down, generating a burst of join, leave, and heartbeat events — the pattern spiraled uncontrolled. The celeryev exchange was suddenly handling 7–8k messages per second, pushing RabbitMQ past its memory watermark and leaving the cluster in a degraded state.
When this memory limit is exceeded, RabbitMQ blocks publishers until usage drops. Once memory falls back under the edge, RabbitMQ resumes normal operation.
Nonetheless, because of this in the course of the memory spike the broker becomes unusable — effectively causing downtime. You won’t want that in production!
The answer is to disable Gossip so staff don’t bind to employee.#. You’ll be able to do that within the docker compose where the employees are spun up.
celery -A myapp employee --without-gossip
Mingle
Mingle is a employee startup step where the brand new employee contacts other staff to synchronise state — things like revoked tasks and logical clocks. This happens just once, during employee boot. In the event you don’t need this coordination, it’s also possible to disable it with --without-mingle
Occasional connection drops
In production, connections between Celery and RabbitMQ can occasionally drop — for instance, attributable to a transient network blip. If you may have monitoring in place, you might see these as transient errors.
The excellent news is that these drops are often recoverable. Celery relies on Kombu, which incorporates automatic connection retry logic. When a connection fails, the employee will try to reconnect and resume consuming tasks.
So long as your queues are configured appropriately, messages are not lost:
durable=True(queue survives broker restart)delivery_mode=2(persistent messages)- Consumers send explicit ACKs to verify successful processing
If a connection drops before a task is acknowledged, RabbitMQ will safely requeue it for delivery once the employee reconnects.
Once the connection is re-established, the employee continues normal operation. In practice, occasional drops are wonderful, so long as they continue to be infrequent and queue depth doesn’t construct up.
To finish off
That’s all folks, these are among the key lessons I’ve learned running RabbitMQ + Celery in production. I hope this deep dive has helped you higher understand how things work under the hood. If you may have more suggestions, I’d love to listen to them within the comments and do reach out!!
