Optimizing Big Data Workflows: A Comparative Analysis of Spark Compression Codecs

Abstract

As business, science and user activity exploded in the past few years, demand for efficient data processing frameworks such as Apache Spark rose. Although Spark allows large computations over networks of cheap hardware, efficient storage and communication is the key challenge. Data compression is the most popular approach to mitigate this issue. By minimising the size of data on disk and in motion, compression speeds up I/O, reduces network traffic and lowers storage costs. But with so many different compression codecs available, with their own trade-offs in terms of speed, compression ratio, and resource overhead, practitioners and researchers find it very difficult to make a informed decision for certain use cases.

This paper explores four popular Spark compression codecs (Snappy, LZ4, ZSTD and Gzip) and analyzes their storage and computation performance. Our comprehensive comparative analysis combines two real-world datasets: an airline flight dataset, and a web logs dataset. Our test workloads include aggregation, multi-column joins, and iterative machine learning computations. We examine compression ratio, compression/decompression time, job completion time and resource consumption and provide feedback that can help developers make decisions regarding the tradeoffs between storage and computation speed. We further discuss how the underlying nature of the datasets (structural regularity, repetition of values, irregular text) may affect the choice of codec. This indicates that Gzip typically has the best compression ratio at the cost of speed, while Snappy and LZ4 perform better at speed. ZSTD offers a hybrid approach, integrating both speed and ratio in many situations. We present our results as a detailed roadmap for researchers and engineers to help their Big Data pipelines run more efficiently.