Posted inDatabase

Manticore Search: The Lifesaver Alternative to Elasticsearch When Server RAM Runs Out

Database tutorial - IT technology blog
Database tutorial - IT technology blog

If you’re struggling with Elasticsearch’s high RAM consumption…

About a year ago, I was tasked with optimizing a system for a news site with around 50 million articles. At that time, the team was running Elasticsearch on a 16GB RAM VPS. Every time we re-indexed data or faced a traffic spike, the Java Virtual Machine (JVM) would crash due to overload, requiring real-time monitoring to manage the crisis.

After testing several options, including building high-speed instant search for the application, I decided to migrate everything to Manticore Search. The results were shocking: RAM consumption dropped from 14GB to less than 2GB. Search speed remained just as fast, and complex queries were even smoother than before. Manticore is a fork of Sphinx Search but modernized, supporting real-time indexing and being extremely lightweight.

Install Manticore in 5 Minutes

I’ll be using Ubuntu 22.04/24.04. Manticore supports most popular Linux distributions today.

Step 1: Add repo and install

# Install official repo
wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update

# Install manticore and extra packages
sudo apt install manticore manticore-extra -y

Step 2: Start the service

sudo systemctl start manticore
sudo systemctl enable manticore

Step 3: Query via MySQL Protocol

Manticore’s biggest advantage is its support for the MySQL protocol. You don’t need to learn complex JSON syntax like in Elasticsearch. Just use familiar SQL commands, which is much easier than analyzing MySQL slow query log entries to find performance issues.

mysql -h 127.0.0.1 -P 9306

Try creating a table and inserting test data:

CREATE TABLE products(title text, price float) realtime;
INSERT INTO products(title, price) VALUES ('iPhone 15 Pro Max', 1200.50);
SELECT * FROM products WHERE MATCH('iphone');

Why Choose Manticore Over Elasticsearch?

When dealing with big data, infrastructure costs are a critical factor, especially when processing billions of records in real-time. Elasticsearch is powerful but runs on the JVM, making it resource-hungry. In contrast, Manticore is written in C++, maximizing hardware performance without an intermediate virtualization layer.

  • RAM Efficiency: For the same dataset, Manticore uses about 1/10th the RAM of ES. A 4GB RAM server can easily handle tens of millions of records.
  • SQL Friendly: Connect via port 9306 using MySQL libraries for PHP, Python, or Node.js. Very convenient for web developers.
  • Instant Startup: Restarting Manticore takes only 1-2 seconds. Meanwhile, ES often takes minutes for its status to turn “green”.
  • Advanced Full-text Search: Supports stemming, weighting, and highly accurate fuzzy searching.

Real-world Production Configuration

The configuration file is located at /etc/manticoresearch/manticore.conf. Instead of using the old static file style, I recommend using Real-time (RT) index for better flexibility.

If you need to periodically load data from MySQL/PostgreSQL, perhaps by mastering Airbyte to automate your ELT pipelines, refer to the source configuration template below:

source src_products {
    type = mysql
    sql_host = localhost
    sql_user = your_user
    sql_pass = your_password
    sql_db = your_database
    sql_port = 3306
    sql_query = SELECT id, title, content, created_at FROM articles
}

index idx_articles {
    source = src_products
    path = /var/lib/manticore/idx_articles
    morphology = stem_en # English stemming
    min_word_len = 1
}

After saving the config, run the following command to start indexing:

indexer --all --rotate

Handling Vietnamese and Optimizing Performance

1. Searching Accented and Unaccented Vietnamese

Manticore requires a charset_table to understand special characters. For proper Vietnamese search, you need to define a character map so the system can perform “folding” (automatically converting accented characters to unaccented during queries).

In my experience, using a comprehensive charset_table for Vietnamese in the config file is best. This allows users to type “dien thoai” and still get “điện thoại” in the results.

2. Distributed Index (Data Sharding)

If your data grows to hundreds of millions of rows, a single server might not handle it. In this case, use a Distributed Index to aggregate results from multiple nodes.

CREATE TABLE remote_index TYPE='distributed' local='idx_part1' remote='192.168.1.10:9312:idx_part2';

Hard-won Lessons from the Field

  1. Always back up your data: With Real-time indexes, use the FREEZE command before copying data files in /var/lib/manticore to ensure integrity.
  2. Use Columnar Library: If you need to perform sum or group by calculations on millions of rows, install manticore-columnar-lib. Its analytical speed is comparable to ClickHouse.
  3. System Monitoring: Don’t forget to use the SHOW STATUS command. I usually push QPS (Queries Per Second) and Latency metrics to Grafana to monitor system health.
  4. Use HTTP JSON API for Apps: While the SQL protocol is great for debugging, when writing code (Node.js, Go…), using the HTTP API (port 9308) provides better load balancing.

Manticore Search might not be as famous as Elasticsearch, but in terms of price-to-performance (P/P), it’s truly unbeatable. If you want to save server resources while maintaining lightning-fast search speeds, try migrating today.

If you encounter errors with Vietnamese configuration or get stuck anywhere, feel free to comment below!

Share:
Tags: