Table of contents

Learning Redis (Part 4)

When the primary goes down, you would otherwise need to manually select a replica, promote it to primary, and point the other replicas to the new primary.

Sentinel performs these operations automatically, without manual intervention.

Sentinel

Responsibilities

1. Monitor whether the primary and replicas are working normally.
2. Automatically promote a replica to primary when the primary fails.
3. Sentinels monitor not only primaries and replicas: multiple Sentinels also monitor one another.
4. Automatically discover child nodes.

Configuring sentinel.conf

Syntax: sentinel monitor . quorum is the minimum number of votes and the number of Sentinels required to consider the primary objectively down.

sentinel monitor mymaster 127.0.0.1 6379 1

Syntax: sentinel down-after-milliseconds .

If the primary does not respond within 15 seconds, the Sentinel considers it subjectively down.

sentinel down-after-milliseconds mymaster 15000

Starting the Primary and Replicas

Master:127.0.0.1:6379

Slave1: 127.0.0.1:16379

Slave2: 127.0.0.1:26379 (Sentinel’s default port is 26379. Because it is occupied here, change Sentinel to 36379 in sentinel.conf.)

Starting Sentinel

# Specify the Sentinel configuration file.
redis-sentinel ./redis/redis-5.0.3/sentinel.conf

Interpreting the Output

1. On startup, Sentinel loads the configuration above and begins monitoring the primary.

2. While monitoring, Sentinel sends INFO to the primary every 10 seconds. INFO retrieves the monitored Redis instance’s run ID, replication information, and child-node information. This allows Sentinel to discover child nodes automatically.

3. Every second, Sentinel sends PING to the primary and replica instances to check their health. If no reply arrives within down-after-milliseconds, this Sentinel considers mymaster subjectively down.

4. After startup, Sentinel uses another connection to subscribe to the sentinel:hello channel on the primary and replicas. It sends its own information to the channel every 2 seconds. When a new Sentinel joins, the others quickly discover it through the channel they all subscribe to.

   Why use another connection to subscribe to the channel?

   Because after subscribing, that connection can no longer execute other commands.

   5. Kill the primary on 6379.

   6. sdown (subjectively down): Sentinel considers the primary on 6379 subjectively down.

   7. odown (objectively down): when at least quorum Sentinels consider the primary on 6379 down, it is considered objectively down.

   8. try-failover: begin failover.

   9. selected-slave: begin selecting among the replicas.

   10. failover-end: failover is complete.

   11. switch-master mymaster 127.0.0.1 6379 127.0.0.1 16379: move the primary from 6379 to 16379.

   12. If 6379 recovers after the new primary is selected, it becomes a replica of the new primary.

   13. -sdown means 6379 is healthy again.

   14. convert-to-slave: make 6379 a replica of 16379.

How Is a Leader Elected among Multiple Sentinels?

Why select a leader Sentinel?

A leader is needed to choose the new primary, so the first step is to elect a leader Sentinel.

What happens if there is no guarantee of a single leader?

In an extreme case, every Sentinel might act as leader. Each may have a different view of the primary and replicas, leading to multiple primaries being selected. Those primaries could instruct other databases to become their replicas, causing confusion.

• A Sentinel that detects the primary is down sends a command to the other Sentinels asking them to elect it leader.

• If another Sentinel has not voted for anyone else, it agrees.

• If more than quorum Sentinels agree to make A the leader, A becomes the leader.

• When several Sentinels run for leader, none may be elected. In that case, they wait a random amount of time before restarting the election, until a leader is chosen.

How Does Sentinel Choose the Best Replica to Become Primary?

Choose the replica with the highest priority. The replica priority setting is in the configuration file:

Configure replica priority. A smaller number means higher priority.

replica-priority 100

If priorities are equal, select the replica with the largest replication offset, because it has replicated more of the old primary’s data.

If offsets are equal, select the replica with the smallest run ID.

How Does Recovery Proceed after Choosing a Replica?

Use SLAVEOF NO ONE to promote the selected replica to primary.

Send SLAVEOF to the other replicas.

These are the same two commands used for manual failover.

How Should Sentinels Be Deployed?

A single Sentinel creates considerable risk. If it fails, the Redis primary-replica setup will not recover without manual intervention.

Sentinels therefore also need to monitor one another.

Recommendations

1. Deploy a Sentinel for every node, whether primary or replica. This provides comprehensive monitoring, but many Sentinels also produce many request connections.
2. Set the voting threshold quorum to N/2+1.