How To Deploy RabbitMQ With The Kubernetes Operators In 1 Hour

Leverage the power of Kubernetes Operators to deploy your own RabbitMQ Cluster

RabbitMQ is the most widely deployed open-source message broker. Before talking about RabbitMQ, let's define what a message broker is. It allows applications and services to communicate with each other and exchange data. The diagram below presents the use case of an event bus using a message broker:

Message brokers are very useful to decouple communications between your components. It gives you the opportunity to create fully distributed and modern architectures. In the message broker family, you can find other popular names like Apache Kafka, Apache ActiveMQ.

RabbitMQ is lightweight and easy to deploy on-premise or in the cloud. It offers interesting features:

• It supports multiple messaging protocols like AMQP, STOMP, MQTT, etc...
• It can be deployed in distributed and federated configurations to meet high-scale and high availability requirements.
• It provides a UI for management and an HTTP API for monitoring.
• It can be extended with many available plugins.
• There are implemented clients in many programming languages: Python, Javascript, Java, Go, Ruby, etc.

In this tutorial, we will focus on how to deploy a high-available RabbitMQ cluster in a Kubernetes cluster. The choice will be to use Kubernetes Operators to achieve this goal efficiently. We will see the advantages of using them and how to deploy RabbitMQ resources step by step.

Why Use The Kubernetes Operators?

"Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components." https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

Operators in Kubernetes allow you to extend the cluster's behavior without modifying the code of Kubernetes itself. In our case, the behavior of RabbitMQ will be delegated to the operators. This will save us a lot of time on our side.

The RabbitMQ team develops and maintains two Kubernetes operators :

• RabbitMQ Cluster Kubernetes Operator to automate provisioning, management, and operations of RabbitMQ clusters running on Kubernetes.
• RabbitMQ Messaging Topology Operator to manage the topology of the clusters (Permissions, Users, etc...)

The Operators are installed through CRDs (Custom Resource Definition) in the Kubernetes cluster. Once installed, new resources are known in the cluster such as classical kinds (Pod, Deployment, etc...). You just have to create the YAML manifests to invoke them.

Deploying The RabbitMQ Cluster Kubernetes Operator

The CRD is available in the repository releases. Install it with the kubectl command:

$ kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"

namespace/rabbitmq-system created
customresourcedefinition.apiextensions.k8s.io/rabbitmqclusters.rabbitmq.com created
serviceaccount/rabbitmq-cluster-operator created
role.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-role created
clusterrole.rbac.authorization.k8s.io/rabbitmq-cluster-operator-role created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-operator-rolebinding created
deployment.apps/rabbitmq-cluster-operator created

The new resources are inside the rabbitmq-system namespace:

$ kubectl get all -n rabbitmq-system
NAME                                             READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-operator-7cbf865f89-9pq6n   1/1     Running   4          2d21h

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rabbitmq-cluster-operator   1/1     1            1           2d21h

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/rabbitmq-cluster-operator-7cbf865f89   1         1         1       2d21h

Ensure the resources are all ready to go next.

Create The RabbitMQ Cluster

The RabbitMQ Cluster Operator provides a RabbitmqCluster kind. You can now create as many RabbitMQ clusters as you want in your Kubernetes cluster. You have just to define them in a YAML manifest like this:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  labels:
    app: rabbitmq
  name: rabbitmq
spec:
  replicas: 3
  image: rabbitmq:3.9.11
  service:
    type: ClusterIP
  persistence:
    storageClassName: hostpath
    storage: 10
  resources:
    requests:
      cpu: 256m
      memory: 1Gi
    limits:
      cpu: 256m
      memory: 1Gi
  rabbitmq:
    additionalPlugins:
      - rabbitmq_management
      - rabbitmq_peer_discovery_k8s
    additionalConfig: |
      cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      cluster_formation.k8s.address_type = hostname
      vm_memory_high_watermark_paging_ratio = 0.85
      cluster_formation.node_cleanup.interval = 10
      cluster_partition_handling = autoheal
      queue_master_locator = min-masters
      loopback_users.guest = false
      default_user = guest
      default_pass = guest
    advancedConfig: ""

Let's recap the content of the manifest above:

• We create a resource of kind  Rabbitmqcluster. It is defined by the CRD we installed in the previous part.
• The cluster has 3 replicas for high availability and fault tolerance.
• Each pod has clearly defined resources.
• Added plugins: rabbitmq_management for management UI and rabbitmq_peer_discovery_k8s for cluster formation. You can find other plugins here: https://www.rabbitmq.com/plugins.html.
• The cluster configuration is done in the additionalConfig field: https://www.rabbitmq.com/configure.html#config-file.
• The type of service used is ClusterIP. You can configure a LoadBalancer type if your application is outside the cluster.
• The persistence is configured. The given example runs on a local Kubernetes cluster so the storage class is hostpath. You can look at the classes available in Kubernetes that will fit your needs: https://kubernetes.io/docs/concepts/storage/storage-classes/

$ kubectl create -f rabbitmq.yaml
$ kubectl get pods -l app.kubernetes.io/name=rabbitmq
NAME                READY   STATUS    RESTARTS   AGE
rabbitmq-server-0   1/1     Running   0          4m41s
rabbitmq-server-1   1/1     Running   0          4m41s
rabbitmq-server-2   1/1     Running   0          4m41s

Deploying the RabbitMQ Topology Operator

Let's place to RabbitMQ Topology Operator now. It allows you to manage RabbitMQ specific resources like users and permissions for example: https://www.rabbitmq.com/kubernetes/operator/using-topology-operator.html.

This Operator needs cert-manager to work. Deploy it and ensure all pods are healthy:

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml
$ kubectl get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7dd5854bb4-vt5jn              1/1     Running   0          47s
cert-manager-cainjector-64c949654c-j2npq   1/1     Running   0          47s
cert-manager-webhook-6b57b9b886-f6gwt      1/1     Running   0          47s

Install the CRD:

kubectl apply -f https://github.com/rabbitmq/messaging-topology-operator/releases/latest/download/messaging-topology-operator-with-certmanager.yaml

Here is an example of a manifest to configure the permissions for the guest user to do tests with the authentication:

apiVersion: rabbitmq.com/v1beta1
kind: Permission
metadata:
  name: rabbitmq-guest-permission
spec:
  vhost: /
  user: guest
  permissions:
    write: ".*"
    configure: ".*"
    read: ".*"
  rabbitmqClusterReference:
    name: rabbitmq

Install The RabbitMQ Cluster Operator Kubectl Plugin

The RabbitMQ team also provides a kubectl plugin to use the RabbitMQ Cluster Operator. This plugin will automate many interactions with the Kubernetes API and the RabbitMQ cluster operator.

The plugin needs krew plugin manager installed (Installation here is for MacOSX, check the documentation and adapt for your OS)

(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

Update your PATH environment variable with the location of krew:

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

Check the installation is ok:

$ kubectl krew

Install the RabbitMQ plugin:

$ kubectl krew install rabbitmq

Check the RabbitMQ plugin:

$ kubectl rabbitmq help

You can retrieve your cluster in the list:

$ kubectl rabbitmq list                          
NAME       ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmq   True               True               4d20h

You can also open directly the management UI:

$ kubectl rabbitmq manage rabbitmq

Conclusion

Through this tutorial, we saw the advantage of using Operators to deploy Kubernetes. They allow transparency and provisioning of RabbitMQ clusters easily. It automates a large part of the management and allows to define the RabbitMQ topology through YAML manifests. Finally, we have seen how to set up the kubectl plugin to interact with the operator.