Sign in Subscribe

On this page

MongoDB

MongoDB on Kubernetes: An Engineering Decision Guide to Backup, PITR, and the Percona Operator

Learn how to run MongoDB on Kubernetes with the Percona Operator using replica sets, physical backups, and Point-in-Time Recovery (PITR). Explore production architecture, backup strategies, and recovery best practices.

Rithvik Kalive

• 7 min read

Introduction

Running databases on Kubernetes still carries a reputation for being unreliable—and that skepticism is notoriously stronger for non-relational systems. Stateful workloads require durable storage, careful failover, and a recovery story that goes far beyond simply restarting a pod.

For MongoDB in particular, many teams stop at a basic StatefulSet and a scheduled mongodump. Unfortunately, this covers neither high availability (HA) nor point-in-time recovery (PITR).

Popular approaches often avoid this problem rather than solve it by relying on managed services like Atlas or keeping MongoDB outside the cluster entirely. But when your requirements dictate a self-managed MongoDB on Kubernetes—with durability, ACID semantics, and true recoverability—the operational gap becomes painfully clear.

Looking for a stack that treats backup and PITR as first-class concerns led us to the Percona Operator for MongoDB and Percona Backup for MongoDB (PBM). Here is our engineering decision record on why we chose this stack, the technical mechanics of how it works, and the topology we deployed.

1. The Production-Readiness Scorecard

Major database platforms (Atlas, Aurora, Cloud SQL) describe production readiness using a shared vocabulary. They distinctly separate guarantees that teams often mistakenly treat as one:

  • Durability: A committed write survives a crash (journal + durable storage).
  • Availability: The system stays up when a node or zone fails (replicas + automatic failover via Raft-like elections).
  • Recoverability: You can restore after corruption, accidental deletes, or cluster loss (base backup + continuous oplog archival for PITR).

Atlas is explicit: high availability is not disaster recovery. Failover handles infrastructure failure; backups and PITR handle data integrity failure. Multi-document ACID consistency also requires replica-set semantics, not a bare standalone mongod.
| RPO / RTO | Data loss tolerance and recovery time limits | Define & Monitor |

2. Why Not a Plain StatefulSet?

Many teams begin with a mongod image inside a StatefulSet—one pod, one PVC, and maybe a CronJob wrapping mongodump.
With a plain StatefulSet, you own the hard parts yourself. This model usually breaks down in three specific areas:

  1. Replica Set Initialization: StatefulSets do not run MongoDB commands. You are responsible for generating keyfiles, executing rs.initiate(), and dynamically adding/removing members via rs.reconfig() when scaling.
  2. Split-Brain & Networking: If a pod restarts with a new IP and you haven't perfectly configured headless services and DNS, MongoDB nodes lose track of each other, leading to election loops or split-brain scenarios.
  3. Backups & True PITR: A CronJob doing a mongodump only provides logical snapshots at specific intervals. True PITR requires archiving the MongoDB oplog.rs collection continuously, which requires custom scripts and tracking upload cursors to object storage.

The gap between "StatefulSet + CronJob" and "Operator + PBM" is the difference between hope and a tested recovery path.

3. The Solution: Percona Operator + PBM Architecture

The Percona Operator runs Percona Server for MongoDB (PSMDB) via Kubernetes Custom Resources (CRs). The operator is a Go-based controller loop that constantly watches the K8s API. If a node fails, the operator reconciles the desired state, orchestrating the safe replacement of the pod and ensuring the new member joins the replica set.
Crucially, Percona Backup for MongoDB (PBM) runs as a sidecar (pbm-agent) next to every mongod container. The agents coordinate via a control collection inside MongoDB itself to elect a backup leader (preferring secondary nodes to avoid hitting the primary with backup I/O).

Note: For a 1-member replica set, only the Primary Pod exists. PBM still seamlessly handles backup and oplog uploads from that single node.

4. How Point-in-Time Recovery (PITR) Actually Works

PITR is easier to understand if you stop thinking of a "backup" as a single file. With PBM, you have two distinct layers:

  1. A Base Backup (Physical): Instead of a logical mongodump (which reads data into BSON files via the query engine), PBM performs a physical backup. It utilizes the $backupCursor command to safely copy the underlying WiredTiger data files directly to object storage. This is significantly faster for large datasets.
  2. Oplog Slices: Small, continuous uploads of everything that changed after that snapshot. The MongoDB oplog is an idempotent changelog of every write.

Restore means landing on the nearest good base backup, then having the PBM agent fetch and apply oplog slices forward until the exact moment you care about.

RPO (how much data you might lose) is bounded by your oplogSpanMin setting (typically 5–10 minutes). RTO (how long a restore takes) is dominated by copying the base backup back into the cluster and replaying the oplog. This is why frequent full snapshots still matter—compute-heavy oplog replay takes time, so keeping the replay window short is essential.

5. Topologies & Trade-offs: 1-Node vs. 3-Node

Our workload features important data but currently has low traffic. We had a hard requirement to stay self-managed on Kubernetes. Against our scorecard, this meant buying durability and recoverability immediately, but deferring live multi-node availability until uptime becomes the constraint.
We opted for a 1-member replica set. It is critical to use a 1-member replica set and not a standalone mongod. Standalone nodes do not have a local database with an oplog.rs collection, meaning PBM cannot function, and they do not support multi-document transactions.

When our uptime requirements tighten, the migration is just an incremental config change: increase replsets[].size from 1 to 3, and change application connection strings to demand w: majority. The operator handles the node provisioning, network routing, and replica set reconfiguration automatically.

6. What We Deployed (The Declarative YAML)

Here is a simplified look at the declarative state that achieves full physical backups and 5-minute PITR on a single node:

```yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB

metadata:
  name: mongo-prod

spec:
  crVersion: 1.16.0

  replsets:
    - name: rs0
      size: 1

      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 50Gi

  backup:
    enabled: true

    pitr:
      enabled: true
      oplogSpanMin: 5

    storages:
      s3-main:
        type: s3
        s3:
          bucket: mongo-prod-backups
          region: ap-south-1
          credentialsSecret: mongo-backup-s3

    tasks:
      - name: daily-physical
        enabled: true
        schedule: "0 2 * * *"
        keep: 14
        storageName: s3-main
        type: physical
```

7. Day-2 Rules for Survival

A workable topology is not enough on its own. A few habits keep the recovery story honest:

  1. Monitor the Metrics: The Percona Operator deploys an integrated Prometheus exporter. You must set up alerts on specific metrics: alert on pbm_backup_status if a base backup fails, and monitor pbm_pitr_status to ensure oplog uploads are not falling behind. A silent backup pipeline is vastly more dangerous than no pipeline at all.
  2. Treat object storage as part of the database: Use immutable or versioned buckets, set lifecycle retention policies you can explain, and utilize cross-region copies if region loss matters.
  3. Drill your restores: Restore on a schedule into a scratch cluster. Keep a short runbook outlining who performs the restore, how apps reconnect, and how PITR is re-enabled afterward.
  4. Define your scaling triggers: Decide in advance what triggers the move to three nodes (uptime, compliance, or traffic) so the upgrade is a planned size change, not a panicked incident response.

Frequently Asked Questions

What is the Percona Operator for MongoDB?

The Percona Operator automates MongoDB deployment, scaling, replica set management, backups, and upgrades on Kubernetes. It also integrates with Percona Backup for MongoDB (PBM) to support point-in-time recovery (PITR).

Can MongoDB run reliably on Kubernetes?

Yes. With a replica set, persistent storage, and the Percona Operator, MongoDB can run reliably on Kubernetes while supporting automated failover, backups, and recovery.

What is Point-in-Time Recovery (PITR) in MongoDB?

Point-in-Time Recovery (PITR) restores MongoDB to a specific timestamp by combining a physical backup with continuously archived oplog entries.

Why is a MongoDB replica set required for PITR?

PBM relies on the MongoDB oplog.rs collection, which exists only in replica sets. A standalone MongoDB deployment does not support point-in-time recovery.

What is the difference between mongodump and PBM?

mongodump creates logical backups, while PBM creates physical backups and continuously archives oplog changes. PBM supports faster restores and point-in-time recovery.

Can I start with one MongoDB node and scale later?

Yes. You can start with a single-member replica set and later increase it to three members. The Percona Operator handles replica set reconfiguration automatically.

Where should MongoDB backups be stored?

Store backups in Amazon S3, Google Cloud Storage, or Azure Blob Storage. Object storage provides durable backup retention and supports point-in-time recovery.

Is the Percona Operator suitable for production?

Yes. It provides replica set automation, rolling upgrades, physical backups, monitoring integration, and point-in-time recovery for production Kubernetes deployments.

Conclusion

Running important data on MongoDB in Kubernetes is less about picking a container image and more about deciding which guarantees you need—and proving you can restore when something goes wrong.
For us, the Percona Operator for MongoDB + PBM closed the operational gap. It offers declarative lifecycle management, physical backups, continuous oplog PITR to object storage, and a restore path we could actually execute.
Start small or go multi-node, tighten RPO with denser oplog archival, or add geographic DR when the business asks for it—all without throwing away your backup model.
Backups are inventory. Restores are proof. PITR is the difference between losing an hour and losing the wrong hour entirely.

Read More

Recent