16-core server but the script takes all afternoon?
Have you ever sat waiting for hours just to compress thousands of images or parse dozens of gigabytes of server logs? If the answer is “yes,” this article is for you. When I first started, I used simple for loops in Bash. Once, I needed to convert 20,000 images on a 16-core server. The result was the script running on exactly 1 core, while the other 15 cores sat… idle, wasting my entire afternoon just watching the terminal crawl along.
Everything changed when I discovered GNU Parallel. It acts as a coordinator, automatically distributing tasks across all available CPU cores. Instead of running commands sequentially, we force the server to work at full capacity, completing tasks 5 to 10 times faster depending on the machine’s configuration.
Install GNU Parallel in a Flash
To get started, install it on your system. GNU Parallel is available in the repositories of most popular Linux distributions.
# Ubuntu / Debian
sudo apt update && sudo apt install parallel -y
# CentOS / RHEL / Fedora
sudo dnf install parallel -y
# Arch Linux
sudo pacman -S parallelTry a simple command to see how it works:
parallel echo ::: 1 2 3 4 5Here, the ::: symbol is used to separate the command from the inputs. Instead of printing 1, 2, 3, 4, 5 sequentially at a snail’s pace, Parallel throws them into different processes for simultaneous processing.
Let’s look at a more practical example: You have 50 compressed .gz log files totaling about 10GB and need to extract them urgently.
ls *.gz | parallel gunzipInstead of running gunzip on each file one by one, the command above triggers multiple processes to run at once. You will see CPU load spike, but in return, the waiting time will drop to just a few minutes.
Why GNU Parallel Outperforms xargs and for Loops
You might wonder: “Can’t I just use xargs -P for parallel execution?”. In reality, Parallel is much smarter and safer. After years of managing VPS systems, I prefer Parallel for these three “killer” features:
- Preserves output order: With
xargs, terminal output can be completely chaotic depending on which process finishes first. Parallel groups the results and returns them in the original input order, which is crucial when exporting data to reports. - Handles filenames with spaces: Spaces in filenames are a nightmare for Bash scripting. Parallel handles this smoothly without needing complex
-0orIFStricks. - Distributes work via SSH: This is the ultimate feature. You can push commands to 3-4 other servers via SSH to leverage the resources of an entire cluster simultaneously.
3 Real-World Application Scenarios
1. Batch Image Resizing for Web
If you need to resize a 5GB image library in a folder to 800px using ImageMagick, using Parallel will save you a ton of time:
ls *.jpg | parallel convert {} -resize 800x800 {.}__resized.jpgWhere:
{}: A variable representing the current filename.{.}: The filename without the extension (making it easy to create new files).
2. High-Speed Data Downloads
Have a list of 100 URLs in a links.txt file? Instead of using wget to download them one by one, try this:
cat links.txt | parallel -j 10 wget {}The -j 10 parameter tells Parallel to run a maximum of 10 download threads simultaneously. The download speed will be significantly faster, but remember not to set the number too high or the target server might block your IP for looking like a DDoS attack!
3. Ultra-Fast Log Error Scanning
When searching for the ‘ERROR’ keyword in hundreds of old compressed log files, I usually use:
ls access.log.*.gz | parallel "zgrep 'ERROR' {} >> errors.txt"Resource Control: Don’t Let Your Server Crash
When working on a production server, you can’t just throw commands out and let them hog 100% of the CPU. A small tip is to limit the number of jobs based on the core ratio:
parallel --jobs 50% ... # Only use 50% of available coresOr if your task is heavy and prone to failure, use the --joblog feature to track the status:
parallel --joblog my_tasks.log ./my_script.py ::: input1 input2 input3If the server happens to crash or you want to pause, just add the --resume flag next time. Parallel will only run the remaining unfinished tasks from my_tasks.log, which is extremely convenient for tasks running overnight.
Hard-Won Lessons for Parallel Execution
Although Parallel is very powerful, based on real experience, you need to keep three things in mind:
- Watch out for Disk I/O bottlenecks: If your task involves heavy data writing (like copying large files), running multiple processes in parallel might actually be slower than single-tasking. The disk heads have to seek continuously, leading to high I/O Wait.
- Manage RAM: Every spawned process consumes a certain amount of RAM. If you run 50 threads and each takes 1GB of RAM on a 16GB server, the system will definitely freeze.
- Always test with a Dry-run: Before hitting Enter to run for real, add the
--dry-runflag to see how the commands will be executed. Avoid cases where a small mistake wipes out your data.
parallel --dry-run echo {} ::: test1 test2In conclusion, GNU Parallel is a must-know tool if you want to work professionally on Linux. It turns boring, time-consuming tasks into something much faster and more efficient. Try applying it to your next project; I’m sure you’ll be surprised by the performance boost it brings.
