Python JSON Serialization: 10x Speed Boost with orjson and msgspec – ITFROMZERO

Table of Contents

Why the default json library is no longer enough

JSON is the lifeblood of modern Microservices and Big Data systems. The built-in json library in Python works fine for small scripts. However, when your application needs to handle thousands of requests per second or JSON files weighing hundreds of MBs, it quickly becomes a “bottleneck”.

This standard library is written primarily in Python. It lacks optimization for modern data types like datetime, UUID, or numpy arrays. To encode a complex object, you often have to write a manual default function. This approach is both a maintenance headache and a performance drag.

Through many Backend optimization projects, I’ve found orjson and msgspec to be the two most worthwhile replacements:

orjson: Written in Rust, supports SIMD (Single Instruction, Multiple Data). It’s one of the fastest encoders/decoders available today.
msgspec: A modern serialization framework. It doesn’t just handle JSON; it validates data extremely fast, far outperforming even Pydantic.

Quick Installation

You can install both with a single command. Don’t forget to use a virtualenv to keep your environment clean.

pip install orjson msgspec

On Linux or macOS, make sure you have build-essential installed. On Windows, available wheels will make installation happen in a flash.

Real-world Implementation

1. Optimizing speed with orjson

orjson smoothly handles data types that usually make the standard library “give up”. For example, you no longer need to manually convert datetime to string.

import orjson
from datetime import datetime
import uuid

data = {
    "id": uuid.uuid4(),
    "timestamp": datetime.now(),
    "tags": ["python", "high-performance"]
}

# orjson returns bytes for maximum performance
json_bytes = orjson.dumps(
    data, 
    option=orjson.OPT_INDENT_2 | orjson.OPT_SERIALIZE_DATETIME
)

print(json_bytes.decode('utf-8'))

Using bitwise option constants makes the code look much more professional and clean. If you need to quickly check complex data strings before parsing, I often use the Regex Tester at Toolcraft. This tool runs directly in the browser, making it very convenient to test patterns without having to rerun your entire Python script.

2. Controlling structure with msgspec

When you need high-speed data validation, msgspec is unbeatable. It skips redundant checks by defining the Schema upfront.

import msgspec

class User(msgspec.Struct):
    id: int
    name: str
    email: str

raw_json = b'{"id": 101, "name": "Thanh", "email": "[email protected]"}'

# Decode and validate in a single step
try:
    user = msgspec.json.decode(raw_json, type=User)
    print(f"Hello {user.name}!")
except msgspec.ValidationError as e:
    print(f"Data error: {e}")

This mechanism significantly reduces CPU load. In tasks repeated millions of times, msgspec is often 10-20 times faster than Pydantic.

The numbers don’t lie: Real-world Benchmark

I tested a dictionary containing 100,000 records. The results on my machine (Core i7, 16GB RAM) are as follows:

Standard JSON: ~0.95s (Slowest)
orjson: ~0.12s (8x faster)
msgspec: ~0.08s (Nearly 12x faster)

Hard-earned lessons for Production:

First, pay attention to data types. orjson returns bytes, not str. If your legacy system requires a string, you must call .decode(). However, if sending via Sockets or HTTP, keeping it as bytes saves a processing step.

Second is the memory issue. For massive JSON files (several GBs), never load the whole thing into RAM. Use streaming mechanisms to avoid crashing the server.

Finally, both of these libraries are extremely strict. Just one extra comma, and they will throw an error immediately. This ensures your data is always clean and compliant.

If you are using FastAPI or building a Data Pipeline, switching to orjson or msgspec is the lowest-effort performance upgrade. With just a few lines of code changed, your system can handle much higher loads. Good luck with your optimization!