Amazon S3 Data Compression: Formats and Best Practices

Amazon S3 stores the bytes your application uploads. Compression happens in the application or a data-processing job before upload, and decompression happens in a reader. Compressing text such as JSON and logs can reduce object size and the bytes sent over a network. The result depends on the data, codec, compression settings, storage class, and tools that must read the object.

There is no compression switch that makes S3 compress every object for you. Choose a format your consumers can read, then measure the complete workflow: stored bytes, transfer or query time, and the compute spent compressing and decompressing.

Key points

S3 stores compressed bytes as uploaded; your producer or processing pipeline applies compression.
Gzip works across many tools for text, while Snappy and Zstandard are common choices inside analytics file formats such as Parquet. Check the exact reader and format.
A gzip stream is usually read as one sequential stream. Compression inside Parquet row groups is different and can still support parallel reads.
Compression can reduce storage and transfer bytes, but it does not remove request charges or the CPU cost of compression. Measure before standardizing a codec.

Choose a format your application can read

“Compression format” can refer to a codec such as Gzip or Snappy, or it can be confused with a file format such as Parquet. A codec encodes data; a file format defines how records, columns, and metadata are arranged. For analytics, that distinction affects which parts of a file a query engine can read efficiently.

Start with the data and the reader

Workload	Good starting point	What to check
Text logs or JSON files exchanged between different tools	Gzip-compressed files, such as `.json.gz`	Confirm that each reader recognizes the extension and can process a compressed stream. Whole-file Gzip can limit parallel scans.
Repeated analytical queries over structured records	Parquet or ORC with a codec supported by the query engine, such as Snappy or Zstandard where available	Check format and codec support for the service and version. Columnar formats can avoid reading unrelated columns; avoid wrapping the entire file in another compression stream.
Images, video, compressed archives, or already-compressed datasets	Usually keep the existing representation	Try a sample first. Recompressing may use CPU without meaningfully reducing bytes.

These are starting points, not guarantees about compression ratios. The same codec can produce different results for different content and settings.

Common codec tradeoffs

Gzip is widely supported and convenient for text interchange. A conventional .gz stream must generally be decompressed from its beginning, so one large file may not feed many distributed readers efficiently.
Snappy favors quick compression and decompression. It is often used inside formats such as Parquet, where the file's internal structure matters as much as the codec.
Zstandard (Zstd) offers configurable compression and can provide strong size reduction at useful speeds. Availability and read/write support depend on the consumer and container format.
Bzip2 is available in some text-data workflows and readers. Check compatibility and scan behavior before choosing it for a shared dataset.

Do not select by codec name alone. Check whether the codec applies to the whole object or to independently readable blocks inside a file format.

Check support in the query service

Amazon Athena supports multiple codecs, but its support varies by file format, engine version, and whether the operation reads or writes data. For CSV, TSV, JSON, and other text data, Athena uses the file extension to identify compression; use a supported lower-case suffix such as .json.gz. Review the current Athena compression guide and its format and codec support matrix for the table format and engine you run.

Redshift Spectrum recognizes whole-file compression from the filename extension. Its current support table distinguishes codecs that allow parallel reads from those that do not. Parquet is different: its row groups can use codecs such as Snappy while remaining readable in parallel, because the entire Parquet object is not wrapped in one compressed stream. See the Redshift Spectrum data file guide before standardizing the output of a pipeline.

AWS Glue also has format-specific options. For example, its Parquet writer lets you configure a supported codec for Parquet data; those settings do not mean that every S3 object or every reader supports the same codec. Check the AWS Glue Parquet options for the job you use.

Compress files before uploading

For a text file, a local command-line workflow keeps the original intact and writes a compressed copy:

gzip -c events.jsonl > events.jsonl.gz
gzip -t events.jsonl.gz
aws s3 cp events.jsonl.gz s3://my-bucket/analytics/events.jsonl.gz

The gzip -t command checks that the compressed stream is readable. In a production pipeline, also compare a decompressed sample or checksum with the source before replacing or deleting original data.

Use consistent names and metadata

Use filenames that tell consumers what the data and compression are, such as events.jsonl.gz. The analytics example above leaves HTTP-specific metadata unset; analytics readers generally follow the filename, table, or format configuration instead. S3's Content-Encoding metadata describes an encoding already applied to the object body; setting that metadata does not compress the object. For HTTP delivery, set Content-Type and Content-Encoding only when they match the representation expected by clients, and verify that the clients handle the compressed response as intended. AWS documents these fields in its guide to S3 object metadata.

Plan for parallel reads and file sizes

Parallel processing depends on both the compression layout and the file format. A single Gzip stream is generally not splittable by ordinary distributed readers: a reader assigned the middle of the object cannot independently decompress from that byte. Bzip2 splitting and standalone Snappy behavior vary by engine and implementation, so check the consumer's documentation rather than treating a codec as universally split-friendly.

Columnar formats such as Parquet divide data into row groups and can compress those groups independently. A query engine can read selected columns or row groups without decompressing one outer Gzip stream. This is why a compressed Parquet file can still support parallel work even when its internal codec is Snappy.

A large number of tiny objects also adds listing, scheduling, and per-object overhead to analytics jobs. Write files sized for the query engine and workload, and compact small files when measurement shows that overhead is material. There is no single target size for every service, partitioning strategy, or query. Avoid turning a dataset into one huge object if readers need to parallelize scans or retrieve only part of it.

Automate with separate input and output paths

An S3 event notification can route new objects to a Lambda function for a small, bounded transformation. Scope the notification to an input prefix and write results to a different prefix or bucket. If the function writes another object under the same triggering scope, that upload can invoke the function again. AWS describes this recursive-trigger risk in its guide to processing S3 events with Lambda.

Do not read an unbounded object entirely into Lambda memory just to compress it. Check object sizes, function memory, temporary storage, runtime, and retry behavior against measured workload limits. For large files or batch datasets, use a distributed processing job such as AWS Glue or Amazon EMR, and write a supported output format directly. Configure its file grouping or compaction based on observed query performance instead of a fixed threshold.

Measure the storage and performance effect

Compare stored bytes with total cost

Compression may lower byte-based storage costs when it reduces an object's stored size. The actual bill also depends on the storage class, Region, retention period, and any minimum billable size or duration. Requests still occur, and the compression job itself consumes compute. Check the current Amazon S3 pricing for the classes in use. This S3 storage-class guide explains other access and retention costs that can affect the comparison.

Estimate savings with representative data: compare original and compressed object sizes, time to compress and decompress, and the bytes and time used by the real readers. Include the cost of any temporary copies and the storage class's billing rules. No codec provides a guaranteed percentage reduction for an unknown dataset.

Check whether fewer bytes improve the workflow

When network bandwidth or scanned bytes are the bottleneck, smaller files may transfer or scan faster. Compression can make a job slower when decompression consumes more CPU, or when one unsplittable file prevents readers from working in parallel. Benchmark end-to-end upload, download, and query paths before and after changing the format. For wider S3 performance tuning, see the guide to Amazon S3 performance optimization.

Account for compression work

Higher compression settings often trade more CPU and time for a smaller output, but the best setting depends on the codec, data, and machine. Test a few settings on representative files and include CPU time in the cost and latency comparison. Do not assume a “hot,” “warm,” or “cold” dataset maps to one compression level; choose from the measured access and processing needs.

A practical rollout checklist

List every producer and consumer of the object, including query engines, downloads, and recovery tools.
Pick a file format and codec supported by those consumers, including their read or write path.
Test representative data for output size, encode/decode time, integrity, query behavior, and parallel reads.
Compare total costs for the relevant retention and storage class; include compute and request costs.
Roll out to a scoped prefix, monitor real workloads, and retain a recovery path until consumers are verified.

Conclusion

Amazon S3 stores the compressed object you send; your applications determine how it is encoded and read. Gzip is a practical choice for interoperable text files, while Parquet or ORC with a reader-supported codec is often a better fit for repeated analytics. Confirm codec and file-format support, preserve opportunities for parallel reads, and measure storage, compute, and query effects with your own data before applying a change broadly.