When your system hits 20–30 million daily visits, memory consumption on Redis quickly becomes a glaring bottleneck. Developers typically throw user IDs or IP addresses straight into a Set to track unique visitors (UV). The result? Tens of gigabytes of RAM evaporate after just a week-long campaign.
The reason is straightforward: a Set stores every raw string in its entirety to guarantee 100% precision. However, for traffic analytics or spam URL filtering, trading off a 1% margin of error to compress tens of gigabytes down to a few megabytes is a no-brainer. This is where probabilistic data structures like HyperLogLog and Bloom Filter shine as ideal alternatives.
1. Quick Start: A 5-Minute Test Drive
Open your terminal and test these directly with redis-cli to see them in action.
Testing HyperLogLog (Built into Redis Core)
HyperLogLog comes built into every standard Redis distribution without requiring any external modules:
# Connect to redis-cli
redis-cli
# Record visitor IPs for the day
PFADD uv:2026-10-26 "192.168.1.1" "192.168.1.2" "192.168.1.3"
# Returns: (integer) 1 (when estimated cardinality changes)
# Re-add an existing IP
PFADD uv:2026-10-26 "192.168.1.1"
# Returns: (integer) 0 (does not increase the count)
# Get the total unique IP count
PFCOUNT uv:2026-10-26
# Returns: (integer) 3
Testing Bloom Filter (Requires Redis Stack)
Bloom Filter requires the RedisBloom module. The quickest way to run it locally is with a Redis Stack container:
docker run -d --name redis-stack -p 6379:6379 redis/redis-stack-server:latest
Once the container is running, test its deduplication capabilities:
redis-cli
# Initialize filter: 1% error rate (0.01), expected capacity of 1 million items
BF.RESERVE blacklisted_emails 0.01 1000000
# Add an offending email to the blacklist
BF.ADD blacklisted_emails "[email protected]"
# Check if email exists
BF.EXISTS blacklisted_emails "[email protected]"
# Returns: (integer) 1 (very likely in the list)
BF.EXISTS blacklisted_emails "[email protected]"
# Returns: (integer) 0 (100% certain it has never appeared)
2. How Do They Work Under the Hood?
Redis HyperLogLog: Cardinality Counting Capped at 12KB RAM
HyperLogLog solves the count-distinct problem. Its killer feature is a fixed size: whether tracking 1,000 or 1 billion records, a single HyperLogLog key in Redis consumes at most 12KB of memory. In return, it carries a standard error rate of approximately 0.81%.
Under the hood, the algorithm hashes input values into random 64-bit strings. The probability that a random binary string begins with k consecutive zeros is (1/2)^k. By dividing data across 16,384 buckets and recording the longest run of leading zeros in each bucket, Redis estimates the total unique cardinality remarkably close to reality.
The Python script below illustrates how to merge metrics across multiple days:
import redis
r = redis.Redis(host='localhost', port=6379, db=0)
# Record individual visits for each day
r.pfadd('uv:day1', 'user_A', 'user_B', 'user_C')
r.pfadd('uv:day2', 'user_B', 'user_C', 'user_D')
# Merge both daily keys into a weekly key directly in Redis memory
r.pfmerge('uv:week_total', 'uv:day1', 'uv:day2')
total_uv = r.pfcount('uv:week_total')
print(f"Estimated weekly total UV: {total_uv}") # Result: 4
Bloom Filter: Set Membership Testing with O(k) Complexity
A Bloom Filter addresses the set membership problem: has element X been seen before?
This structure consists of a long bit array and k independent hash functions. When an element is added, it passes through the k hash functions to compute k array indexes, and all bits at those positions are set to 1.
- Result is 0: The element definitely does not exist (100% guaranteed). If even a single bit is 0, the element cannot possibly be in the dataset.
- Result is 1: The element probably exists. There is still a small chance of a false positive due to hash collisions with other elements. False negatives, however, are strictly impossible.
A key technical constraint to keep in mind: standard Bloom Filters do not support element deletion. Setting a bit back to 0 could inadvertently corrupt the membership tests for numerous other elements.
3. Real-World Applications in Backend Architecture
Preventing Cache Penetration
A common attack scenario: An attacker fires millions of requests with bogus IDs (e.g., product_id = -1 or 99999999). Since these IDs do not exist in cache, every query punches straight through to PostgreSQL. The database connection pool is quickly exhausted, leading to downtime.
Placing a Bloom Filter in front of the caching layer cleanly eliminates this problem:
import redis
r = redis.Redis(host='localhost', port=6379, decode_responses=True)
def get_product(product_id: int):
# 1. Intercept immediately with Bloom Filter
if not r.execute_command('BF.EXISTS', 'bf:products', product_id):
# 100% certain ID does not exist in the system; abort early
return None
# 2. Read standard Redis cache
cache_key = f"product:{product_id}"
data = r.get(cache_key)
if data:
return data
# 3. Query the database only when the ID is genuinely valid
data = query_from_postgres(product_id)
if data:
r.setex(cache_key, 3600, data)
return data
URL Deduplication for Web Crawlers
When running a crawler bot that processes millions of links daily, querying a database to check if a URL has already been fetched quickly creates a severe Disk I/O bottleneck. Storing 10 million raw URL strings in RAM can devour several gigabytes. Compressed into a Bloom Filter with a 1% error rate, the entire bit array takes up only ~11.5MB of RAM, comfortably handling 50,000 ops/sec with sub-millisecond latency.
4. Production Lessons and Operational Tips
My team once faced a critical production incident while tracking click events for an e-commerce platform. Initially, we played it safe by dumping all user IDs directly into Redis Sets. When the Flash Sale hit and traffic spiked, Redis chewed through all 32GB of RAM in just four days.
The primary server ran completely out of memory, triggering an emergency failover via Redis Sentinel. That very night, our team refactored the unique view-counting logic using HyperLogLog. RAM usage plummeted from over 30GB to under 400MB across the entire Redis cluster, running rock-solid throughout the sale campaign.
Here are several key lessons we learned:
- Always initialize filters explicitly with BF.RESERVE: Avoid calling
BF.ADDon an undeclared key. Doing so causes RedisBloom to fall back on an auto-created filter with a tiny default capacity (often just 100 items). As data scales, Redis is forced to chain stacked sub-filters, which wastes memory and inflates query latency. - Avoid HyperLogLog if you need raw IDs: This structure is designed purely to return a single cardinality count. There is no way to retrieve or decode original IDs. If business requirements call for features like “Show the 10 most recent viewers,” you still need a
Setor a database table. - Carefully evaluate the false positive threshold: Reducing the error rate from 1% (0.01) to 0.01% (0.0001) nearly doubles the required bit array size and increases hash computations per request, raising CPU utilization. For most web applications, a 1% error rate is the sweet spot balancing accuracy and resource consumption.
- Need deletions? Use a Cuckoo Filter: When your use case strictly demands removing stored elements while maintaining low memory overhead, switch to a Cuckoo Filter (via
CF.ADDandCF.DELin Redis Stack).
