The most secure bank vault in the world requires two different keys, held by two different people, turned simultaneously. Neither person alone can open it. Now try coordinating this when the key holders are in different buildings, communicating by phone, and sometimes the phone lines go down. That’s distributed locking - ensuring exclusive access to shared resources when the parties involved can’t simply tap each other on the shoulder.
The Single Bank Vault Scenario
Let’s start simple. The First National Bank has one vault requiring two keys:
Morning at the Bank
The Manager arrives at 8 AM with Key A. The Security Chief arrives at 8:05 AM with Key B. They meet at the vault. “Ready?” asks the Manager. “Ready,” confirms the Security Chief. They insert their keys. “On three. One… two… three!” Click - The vault opens.
Perfect coordination. Easy when you’re standing side by side.
The Communication Challenge
Now the bank expands. For security, they separate the key holders:
- Manager works from the Downtown Office
- Security Chief works from the Uptown Office
- They coordinate by phone
Manager calls: “Ready to open vault #3?” Security Chief: “Give me 30 seconds to get to my control panel.” Manager: “Okay, are you ready now?” Security Chief: “Ready. On your count.” Manager: “Three… two… one… turn!”
But what if the phone line drops after “two”? The Manager turns their key, the Security Chief doesn’t. The vault stays locked, and worse, the Manager’s key is now stuck in the “turned” position.
The Multiple Vaults Problem
The bank grows. Now there are fifty vaults:
This diagram requires JavaScript.
Enable JavaScript in your browser to use this feature.
The Coordination Nightmare
Monday morning. Three customers need their vaults:
Customer 1: “I need Vault 7 opened.” Customer 2: “I need Vault 7 opened.” (Same vault!) Customer 3: “I need Vault 22 opened.”
Without careful coordination:
- Both Customer 1 and 2 might be told “yes”
- They arrive to find the vault already open
- Or worse, they interfere with each other’s access
The Reservation System
The bank implements a reservation system:
Customer 1: “I’d like to reserve Vault 7 from 9:00-9:30 AM” Bank System: “Confirmed. You have exclusive access.” Customer 2: “I’d like to reserve Vault 7 from 9:15-9:45 AM” Bank System: “Denied. Vault already reserved until 9:30 AM.”
But now there’s a new problem: What if Customer 1’s phone dies at 9:01 AM? The vault stays locked (and reserved) until 9:30, wasting 29 minutes.
Distributed Locking Patterns
Pattern 1: The Timeout Lock
Like a parking meter, locks expire:
Customer 1: “I need Vault 7 for approximately 30 minutes” Bank: “You have exclusive access until 9:30 AM. If you need more time, renew before 9:25 AM.”
If Customer 1 disappears:
- At 9:30 AM, the lock expires
- Vault 7 becomes available
- Customer 2 can now reserve it
The challenge: What if Customer 1 needs 31 minutes? They might lose the lock while still using the vault.
Pattern 2: The Heartbeat System
Like proving you’re still alive:
Customer 1 gets Vault 7 Every 5 minutes: “I’m still here, still need the vault” Bank: “Confirmed, lock extended”
If heartbeats stop:
- Bank waits one interval (grace period)
- No heartbeat? Lock released
- Vault available for others
The network problem: What if heartbeat messages get delayed in traffic? Customer 1 is actively using the vault, frantically sending “I’m still here!” messages, but the bank doesn’t receive them in time.
Pattern 3: The Token System
Physical possession proves ownership:
Bank issues a unique token: “VAULT-7-TOKEN-12345” Customer must present token to access vault One token exists per vault If you have the token, you have access
The Byzantine problem: What if someone copies the token? Now two parties think they have exclusive access.
Common Locking Pitfalls
The Deadlock Diner
Customer A has Vault 1, needs Vault 2 Customer B has Vault 2, needs Vault 1 Both wait forever
Solution: Lock ordering protocol Always acquire locks in numerical order:
- Need Vaults 7 and 3? Lock 3 first, then 7
- Everyone follows same order = no deadlocks
The Phantom Lock
System shows Vault 7 as “available” Customer requests lock System grants lock Customer arrives: Vault already open!
Cause: Stale data in distributed system
Solution: Version numbers or timestamps on lock status
The Thundering Herd
Popular vault becomes available 100 customers simultaneously request access System overwhelmed processing requests Nobody gets access during the chaos
Solution: Queue with rate limiting, lottery system, or exponential backoff
The Priority Inversion
VIP customer needs Vault 7 urgently Regular customer already has it for routine task VIP waits while regular customer counts pennies
Solution: Preemptible locks, priority queues, or separate VIP vaults
Lock Implementation Strategies
Database Locks
Use a central database as the source of truth:
- Lock table with vault_id, customer_id, expiry_time
- Atomic operations ensure consistency
- Database handles concurrent access
Pros: Simple, reliable Cons: Database becomes bottleneck, single point of failure
Redis/Cache Locks
Use in-memory cache for speed:
- SET vault:7:lock customer123 EX 300 NX
- Atomic operation with automatic expiry
- Blazing fast
Pros: Very fast, automatic expiry Cons: Lock lost if cache fails
Zookeeper/etcd Locks
Use coordination service:
- Create ephemeral node for lock
- Node automatically deleted if client disconnects
- Sequential nodes for queue fairness
Pros: Designed for this, handles failures well Cons: Additional infrastructure complexity
Decision Rules
Use database locks when:
- Simplicity matters more than performance
- You already have a database
- Lock contention is low
Use Redis locks when:
- You need high performance
- You can tolerate brief inconsistency on failure
- Lock duration is short
Use Zookeeper/etcd when:
- You need strong consistency guarantees
- You’re already running coordination infrastructure
- Lock failure is unacceptable
Distributed locking is fundamentally about coordination and trust. Two parties must agree, must act in concert, and must handle failures gracefully. It’s not just about preventing simultaneous access - it’s about creating a system where exclusive access is meaningful, enforceable, and recoverable.
The goal isn’t to build an impenetrable vault. It’s to build a vault that serves its customers reliably, even when the key holders are in different time zones, the phones are unreliable, and Murphy’s Law is in full effect.
