Installing & Configuring a MongoDB Replica Set: Production-Ready Automatic Failover - ITFROMZERO - Share tobe shared! A hands-on guide to deploying a production-grade 3-node MongoDB Replica Set: installation on Ubuntu 22.04, keyfile security, automated failover drills, and resilient connection strings.

Table of Contents

1. A Costly Lesson: When a Standalone MongoDB Crashes at 3 AM

It is 3 AM, and your phone will not stop buzzing—the MongoDB server just went down. Every API is paralyzed, and customers cannot check out. If you have ever been in this situation, you know the dread of relying on a single standalone node in production.

A faulty RAM stick or a 100% full disk is all it takes to bring your entire service down (Single Point of Failure – SPOF). The only real solution is to distribute your data across multiple independent nodes.

A minimal production-grade architecture requires 3 nodes:

1 Primary Node: Handles all write operations and, by default, serves reads as well. Every write operation is recorded sequentially in an internal log called the oplog (operations log).
2 Secondary Nodes: Continuously pull the oplog from the Primary and apply those changes to their local datasets. If the Primary suddenly fails, the cluster takes just 2 to 12 seconds to elect the Secondary with the most up-to-date data as the new Primary.

Why must it always be an odd number, such as 3, 5, or 7? If a network partition occurs between datacenters, the cluster must achieve a strict majority (n/2 + 1) of votes to elect a Primary. With only 2 nodes, a network split leaves each side with exactly 50% of the vote. Consequently, neither side can reach a quorum, and the cluster completely locks up.

2. Infrastructure Preparation & Installation

In this lab, we use 3 Ubuntu 22.04 LTS VPS instances (recommended specs: at least 2 vCPUs and 4GB RAM) within the same private network:

node1.db.internal – 10.0.0.11 (Primary Node candidate)
node2.db.internal – 10.0.0.12 (Secondary Node 1)
node3.db.internal – 10.0.0.13 (Secondary Node 2)

Step 2.1: Synchronize Hostname Resolution

Avoid hardcoding IP addresses directly into your database configuration. Instead, map hostnames in /etc/hosts across all three servers to simplify maintenance during infrastructure migrations:

sudo bash -c 'cat <<EOF >> /etc/hosts
10.0.0.11 node1.db.internal node1
10.0.0.12 node2.db.internal node2
10.0.0.13 node3.db.internal node3
EOF'

Step 2.2: Install MongoDB 7.0

Run the following commands on all 3 servers to install the official packages directly from MongoDB’s repository:

# Import the official GPG key
curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \
   sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor

# Add the repository for Ubuntu 22.04 (Jammy)
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | \
   sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list

# Update and install
sudo apt update && sudo apt install -y mongodb-org

# Enable the service to start on boot
sudo systemctl enable mongod

3. Configuring Keyfile Security and Initializing the Cluster

Never run a Replica Set without authentication enabled. The nodes require a shared secret (Keyfile) to mutually authenticate before exchanging replication data.

Step 3.1: Generate the Internal Authentication Keyfile

On node1, generate a 756-byte random string and apply strict file permissions:

sudo mkdir -p /var/lib/mongodb-key
openssl rand -base64 756 | sudo tee /var/lib/mongodb-key/mongo-keyfile
sudo chown -R mongodb:mongodb /var/lib/mongodb-key
sudo chmod 400 /var/lib/mongodb-key/mongo-keyfile

Permission 400 (read-only for the mongodb user) is mandatory. If permissions are too open (like 777), the mongod daemon will refuse to start.

Use scp to distribute this file to node2 and node3:

# Copy to node2 and node3
scp /var/lib/mongodb-key/mongo-keyfile [email protected]:/tmp/
scp /var/lib/mongodb-key/mongo-keyfile [email protected]:/tmp/

Then, on both node2 and node3, move the file into place and restore proper ownership and permissions:

sudo mkdir -p /var/lib/mongodb-key
sudo mv /tmp/mongo-keyfile /var/lib/mongodb-key/
sudo chown -R mongodb:mongodb /var/lib/mongodb-key
sudo chmod 400 /var/lib/mongodb-key/mongo-keyfile

Step 3.2: Update mongod.conf

Open /etc/mongod.conf on each server. Update the following three configuration blocks:

# /etc/mongod.conf

net:
  port: 27017
  bindIp: 127.0.0.1,node1.db.internal # Replace with the respective hostname for each node

security:
  keyFile: /var/lib/mongodb-key/mongo-keyfile
  authorization: enabled

replication:
  replSetName: "rs0"

Make sure to keep 127.0.0.1 in bindIp so local CLI tools can still connect. Once configured, restart the daemon across all 3 nodes:

sudo systemctl restart mongod

Step 3.3: Initialize the Cluster and Create the Root User

Thanks to MongoDB’s Localhost Exception, you can connect locally from node1 to create the initial admin user without credentials:

mongosh --port 27017

Initiate the replica set using rs.initiate():

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "node1.db.internal:27017", priority: 2 },
    { _id: 1, host: "node2.db.internal:27017", priority: 1 },
    { _id: 2, host: "node3.db.internal:27017", priority: 1 }
  ]
});

The priority: 2 setting tells the replica set to prefer electing node1 as Primary whenever it is healthy. Within 2 to 3 seconds, your prompt will change from rs0 [direct: other] test> to rs0 [direct: primary] test>.

Now, immediately create the root administrator account:

use admin;
db.createUser({
  user: "siteRootAdmin",
  pwd: "MatKhauSieuKho123!@#",
  roles: [ { role: "root", db: "admin" } ]
});

From this point onward, every shell session must be authenticated:

mongosh -u siteRootAdmin -p --authenticationDatabase admin

Step 3.4: Seed Sample Data to Verify Synchronization

When provisioning staging environments or stress-testing, engineering teams often need to import tens of thousands of records from customer Excel or CSV files. If you need a fast way to convert CSV to JSON, check out toolcraft.app/en/tools/data/csv-to-json — it parses files entirely client-side in your browser via JavaScript, so no sensitive data is leaked over the internet.

Once you have your JSON file ready, run mongoimport to quickly load it into the analytics database (and remember to practice optimizing compound indexes to keep queries performant):

mongoimport --uri="mongodb://siteRootAdmin:MatKhauSieuKho123!%40%[email protected]:27017/analytics?authSource=admin" \
  --collection=users --file=data.json --jsonArray

4. Health Checks and Failover Drill

Step 4.1: Checking Replication Lag

The definitive command for cluster diagnostics is rs.status(). However, to quickly inspect replication lag across nodes, run:

rs.printSecondaryReplicationInfo();

This displays the time difference in seconds between each Secondary and the Primary. If this lag spikes beyond 5 to 10 seconds, investigate network throughput or verify if disks are hitting an I/O bottleneck (IOPS saturation).

Also, watch the stateStr field in the output of rs.status(). If any node gets stuck in STARTUP2 or RECOVERING, it is still performing initial data synchronization and cannot yet vote or serve queries.

Step 4.2: Simulating a Primary Outage

Can the cluster truly recover on its own? The only way to be certain is to pull the plug yourself.

Stop the MongoDB service on node1:

# Run on node1
sudo systemctl stop mongod

Immediately open a terminal on node2 and launch the shell:

mongosh -u siteRootAdmin -p --authenticationDatabase admin

Within just 2 to 4 seconds, the prompt on node2 will switch to rs0 [direct: primary] test>. The election completes completely automatically, and your application continues writing data without requiring manual engineering intervention.

Now, bring node1 back online:

# Run on node1
sudo systemctl start mongod

Upon restarting, node1 joins as a SECONDARY and catches up on missing oplog entries. Because it has priority: 2, once it is fully synchronized, it will trigger a graceful stepdown on node2 to reclaim its role as the Primary.

Step 4.3: Crafting a Production-Grade Application Connection String

The most common mistake developers make is pointing application drivers to a single IP address. Your connection string (in Node.js, Go, Python, Spring Boot, etc.) must list all replica set members alongside cluster options:

mongodb://siteRootAdmin:[email protected]:27017,node2.db.internal:27017,node3.db.internal:27017/analytics?replicaSet=rs0&authSource=admin&w=majority&readPreference=primaryPreferred

Two essential flags to remember:

w=majority: Requires a quorum of nodes (at least 2 out of 3) to acknowledge writing the data to disk before the driver returns success. This guarantees zero data loss if the Primary crashes mid-operation.
readPreference=primaryPreferred: Directs reads to the Primary by default for maximum consistency, but automatically fails over read queries to Secondaries if the Primary temporarily goes offline, avoiding 500 errors for end users.