Running Rails with Sidekiq on Kubernetes is a powerful way to decouple background jobs and take full advantage of Kubernetes' scalability. Sidekiq’s multi-threading works great, but when the workload spikes, horizontal scaling becomes essential to keep jobs flowing smoothly.

While Kubernetes can scale workloads based on CPU or memory usage, that’s often not enough for Sidekiq. Imagine a job is delayed because it’s waiting for a third-party service to respond to an HTTP request—not because it’s exhausting CPU or memory. In this case, scaling based on resource usage won't help, and jobs can pile up even though your resources are underutilised.

That’s where KEDA shines. By scaling based on Redis queue length, KEDA allows you to autoscale your Sidekiq workers based on the real workload, such as the number of pending jobs. This ensures efficient scaling without unnecessary complexity.

Step 1: Quick Setup

First you will need to install Keda, you can get the up-to-date methods here: Keda Documentation
My recommendation is that unless you are using OpenShift to install Keda using the Helm Chart:

lang shell

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace

Step 2: Create ScaledObject

Here's a basic configuration for setting up KEDA to scale your Rails Sidekiq workers based on the size of the Redis queue.

ScaledObject YAML

lang yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sidekiq  # The name of the KEDA ScaledObject for scaling Sidekiq
spec:
  pollingInterval: 30  # How often KEDA checks the Redis queue (in seconds)
  cooldownPeriod: 300  # Time to wait (in seconds) before scaling down after scaling up
  minReplicaCount: 1  # Minimum number of Sidekiq worker replicas to run
  maxReplicaCount: 3  # Maximum number of Sidekiq worker replicas to run
  scaleTargetRef:
    apiVersion: apps/v1  # The API version of the resource being scaled
    kind: Deployment  # The kind of resource being scaled (in this case, a Deployment)
    name: sidekiq  # The name of the Deployment that runs Sidekiq
  triggers:
    - type: redis  # The type of event trigger (here, Redis is being used)
      metadata:
        address: redis-master.default.svc.cluster.local:6379  # The address of the Redis instance
        listName: queue:default  # The name of the Redis list (queue) for Sidekiq jobs
        listLength: "100"  # The length of the list that triggers scaling (scale up if jobs > 100)

Step 3: Adding Authentication

If your Redis instance requires authentication, you can use TriggerAuthentication to provide a password for connecting to Redis securely.

lang yaml

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: redis-auth  # The name of the TriggerAuthentication resource used for Redis authentication
spec:
  secretTargetRef:
    - parameter: password  # The parameter to authenticate with, here it's the Redis password
      name: redis-secret  # The name of the Kubernetes secret where the Redis credentials are stored
      key: REDIS_PASSWORD  # The specific key in the secret that holds the Redis password

Updated ScaledObject with Authentication

lang yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sidekiq
spec:
  pollingInterval: 30
  cooldownPeriod: 300
  minReplicaCount: 1
  maxReplicaCount: 3
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sidekiq
  triggers:
    - type: redis
      metadata:
        address: redis-master.default.svc.cluster.local:6379
        listName: queue:default
        listLength: "100"
      authenticationRef:
        name: redis-auth # Configure the reference to the TriggerAuthentication

Maintenance

When everything is set up correctly, KEDA will automatically create a HorizontalPodAutoscaler (HPA) to scale your deployment. To avoid conflicts, ensure you do not set the replicas field on your deployment, especially if you're using GitOps. Also, avoid setting up another HPA for the same deployment, as this could lead to scaling conflicts with KEDA.

If you need to troubleshoot the scaling, you can check the Events log of the ScaledObject or view the operator's logs. If the HPA hasn’t been created automatically, it's likely due to a misconfiguration in your ScaledObject.