Mastering PySpark on Linux: Processing Hundreds of GBs of Data with Python – ITFROMZERO

Table of Contents

Why Pandas Isn’t Enough for Big Data?

If you frequently use Pandas for data processing, you’ve likely experienced your computer “freezing” when a CSV file exceeds your RAM capacity. I once tried loading a 20GB log file on a laptop with 16GB of RAM, and the system locked up within seconds. This is the physical limitation of pure Python: it runs on a single CPU core and is constrained by the RAM of a single machine.

Apache Spark was created to solve this problem. Instead of trying to cram all the data into one place, Spark breaks it down to process it in parallel across multiple CPU cores or even hundreds of servers in a cluster.

PySpark acts as the Python library that lets you control that distributed power. You don’t need to learn complex Java or Scala. Using familiar Python syntax, you can smoothly process billions of rows of data.

Installing PySpark on Linux

For PySpark to operate stably on Ubuntu or CentOS, we need to set up a standard Java environment. Spark doesn’t run directly on Python; it operates via the Java Virtual Machine (JVM).

1. Installing Java (JRE/JDK)

Practical experience shows that Spark is most stable with Java 8 or 11. Avoid rushing to install the latest versions like Java 21, as library compatibility issues are common.

sudo apt update
sudo apt install openjdk-11-jdk -y
# Check the version again
java -version

2. Downloading and Setting Up Apache Spark

You should choose a Spark distribution pre-built for Hadoop. In this example, I’m using version 3.5.0, which is currently quite stable:

wget https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xvzf spark-3.5.0-bin-hadoop3.tgz
sudo mv spark-3.5.0-bin-hadoop3 /opt/spark

3. Installing the PySpark Library via pip

Even though you have the Spark source on your machine, you still need to install the package so Python can call Spark functions:

pip install pyspark

Configuring Environment Variables: The Most Important Step

Many users finish the installation only to find that typing the pyspark command results in “command not found.” This error occurs because the operating system doesn’t know where Spark and Java are located.

Open your shell configuration file (.bashrc or .zshrc):

nano ~/.bashrc

Add the following lines to the end of the file to define the paths:

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
export PYSPARK_PYTHON=python3

Save and activate the new configuration with the command:

source ~/.bashrc

Writing Your First PySpark Script: Real-world Data Analysis

Let’s try to solve a word count problem in a large dataset. This is the foundation of search engines and user behavior analysis systems.

Create the pyspark_demo.py file:

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode, split, col

# Initialize SparkSession - The "heart" of the application
spark = SparkSession.builder \
    .appName("LogAnalysisApp") \
    .getOrCreate()

# In practice, you would use spark.read.csv("path/to/bigfile.csv")
data = [("PySpark processes data very quickly",), 
        ("Linux is a great environment for Big Data",), 
        ("Python and Spark are a perfect match",)]

df = spark.createDataFrame(data, ["content"])

# Split sentences into individual words
words_df = df.select(explode(split(col("content"), " ")).alias("word"))

# Calculate frequency of occurrence
word_counts = words_df.groupby("word").count().orderBy("count", ascending=False)

word_counts.show()
spark.stop()

Execute the script using Spark’s dedicated command:

spark-submit pyspark_demo.py

Monitoring Performance with Spark Web UI

Working with terabytes of data without knowing how the system is performing is a major mistake. While the script is running, open your browser and navigate to http://localhost:4040.

Here, pay special attention to the Stages tab. If you see one task taking 20 minutes while others take only seconds, it’s a sign of “Data Skew.” This phenomenon occurs when one CPU core handles 90% of the workload, dragging down the entire system.

Additionally, if you want to experiment quickly, just type pyspark into your terminal. This opens an interactive environment (REPL) similar to Jupyter Notebook but specifically for Big Data—extremely useful for debugging small code snippets before adding them to your main script.

Big Data processing isn’t out of reach if you master how Spark distributes tasks. I hope this guide helps you confidently deploy large-scale data projects on Linux.