A simple but effective RabbitMQ broker migration strategy

RabbitMQ

RabbitMQ is an amazing piece of software that has become an industry standard across many high profile companies. We use it for asynchronous service-to-service communication. Messages are first pushed to the queue by web services and are then processed by backend workers. This lets us decouple event creation and processing. These events include things like payments, user registrations, document generation, etc.

Here at Nord Security, I've had the fortune to work with this technology for over 4 years with multiple programming languages and many projects, so I've come to know it and love it. Recently, me and my team were tasked with migrating our main RabbitMQ broker instance to a new and updated one. I'd like to share how we did it.

Out of Date

The RabbitMQ message broker has proven to be a reliable and resilient piece of software. But as time goes on, software must be updated to remain performant and secure. Unfortunately, our main RabbitMQ broker server got left un-updated for too long and it became impossible to apply new and important upgrades to the system.

Here were the main issues with the old RabbitMQ version:

• New Debian OS versions are incompatible with the old RabbitMQ;

• The RabbitMQ software version is not supported anymore;

• New bug fixes and security updates cannot be applied;

• Cannot install important new features and improvements.

Furthermore, a lot of our services depend on this RabbitMQ broker instance to work, so any outage could mean pretty serious downtime for our business.

Therefore, we needed to migrate all the queues from the old broker to a new and shiny one. But this is easier said than done. There are over 200 queues in this broker with just as many consumers from different services.

We needed a solution that would allow for a seamless transition with zero message loss, as our business-critical systems depend on the message queues to perform their tasks.

Federated Queues

Fortunately for us, RabbitMQ has a feature called federated queues. It's a way to pass messages from one broker to another without duplicating or losing messages in the process. It manages to do this by moving messages from the upstream broker queue to the downstream when the upstream queue has no consumers.

To have access to the queue federation feature, we needed to install the federation plugin on the RabbitMQ instance. You can find information on how to do that on the official rabbitmq documentation.

The Strategy

We have separated our queues into logical domains by using RabbitMQ vhosts, so it made sense to migrate each vhost separately. We came up with this queue migration strategy:

1. Set up queue federation from the old broker to the new one.

2. Migrate the consumer to the new broker.

3. Migrate the publisher to the new broker.

4. Disable the queue federation and remove leftover queues.

In theory, this process would allow us to migrate all the queues separately and without any downtime because all the messages would simply be moved to the new broker once the consumer was migrated to it.

Migration Process

First, we ran an audit of all the queues and their consumers. After that, we decided which queues to migrate first and started to set up the queue federation. I'll walk you through the steps we took on one of these queues.

Let's say we want to migrate a queue called order.events.nordvpn-affiliate.process-orders. It is placed under the nordvpn_affiliate vhost and bound to the order exchange with a routing key order.event.created. We're using the topic exchange type. This queue is consumed by the nordvpn-affiliate microservice consumer, which handles purchases made through affiliate channels.

We set up a new RabbitMQ broker at new-rabbit.nordvpn.com. To set queue federation to the old downstream broker old-rabbit.nordvpn.com, we have to ssh into the new broker server and run this command in the terminal:

1
rabbitmqctl set_parameter --vhost "nordvpn_affiliate" federation-upstream old-nordvpn-affiliate
2
'{"uri":"amqp://user:password@old-rabbit.nordvpn.com:5673/nordvpn_affiliate"}'

This will set up a federation upstream for the nordvpn_affiliate vhost. You can check all configured federation upstreams using the RabbitMQ management panel at https://new-rabbit.nordvpn.com/#/federation. Our new federated vhost looks like this:

Next, we must set the queue federation policy and override the default one. We run this command:

1
rabbitmqctl set_policy --vhost "nordvpn_affiliate" old-nordvpn-affiliate-federation '.*' 
2
'{"federation-upstream-set":"all"}' --apply-to queues --priority 1

This sets up queue federation for all the queues in the nordvpn_affiliate vhost with a priority of 1, which takes precedence before other policies. We check that the policy was set correctly at https://new-rabbit.nordvpn.com/#/policies:

As the queue federation is set up, we are now ready to migrate the consumer. All our consumers automatically create and bind queues on startup. If they didn’t, we'd need to create the queue in the new broker by hand or by exporting the configuration from the old broker and importing it into the new one.

We change the RabbitMQ broker URL in the consumer and deploy the new configuration. After deployment, it immediately starts consuming messages from the newly federated queue. No messages are lost and they keep their order in the queue:

We repeat this process for all the queues in the nordvpn_affiliate vhost. After all the consumers are migrated, we migrate the publisher as well.

All that is left now is to remove the queue federation and delete the queues from the old RabbitMQ broker. That's it!

Summary

Migrating software infrastructure is rarely fun. But we've found an easy and reliable way to migrate queues from one RabbitMQ broker to another using federated queues. We've already migrated a lot of queues this way to our new RabbitMQ broker with no major problems and soon we'll have fully migrated all of them.