Skip to content

Use zlib.compressobj instead of GzipFile in GZipMiddleware - #3411

Merged
Kludex merged 3 commits into
mainfrom
gzip-compressobj
Aug 5, 2026
Merged

Kludex merged 3 commits into
mainfrom
gzip-compressobj

Conversation

@Kludex

@Kludex Kludex commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Summary

Replace the gzip.GzipFile + io.BytesIO machinery in GZipResponder with zlib.compressobj using wbits=16 + zlib.MAX_WBITS, which produces the same gzip wire format (header, CRC32 and size trailer).

Rationale

The GzipFile approach dates back to the original middleware commit (#111, 2018) and requires supporting machinery that compressobj makes unnecessary:

  • the io.BytesIO buffer and per-chunk getvalue()/seek(0)/truncate() dance
  • the ExitStack lifecycle and __call__ override added in Lazily allocate GZipMiddleware compression resources #3407 (a compressobj holds no resources that need closing)
  • GzipFile's internal BufferedWriter, which hides when compression work actually happens

Unlike GzipFile.write(), compressobj.compress() consumes and deflates its entire input during the call, making the compression cost explicit and self-contained. This also simplifies any future work offloading large-chunk compression to a worker thread (#3410), which currently needs a Z_NO_FLUSH workaround to drain GzipFile's buffer inside the worker. This is the same primitive aiohttp uses in its ZLibCompressor.

Net: 17 insertions, 34 deletions.

Behavior

  • Standard responses: byte-identical semantics; output is valid gzip with correct Content-Encoding/Content-Length.
  • Streaming responses: each chunk is flushed with Z_SYNC_FLUSH, matching the previous GzipFile.flush() behavior, so every streamed chunk remains immediately decodable by clients.
  • The gzip header no longer embeds a timestamp (mtime=0 equivalent), making compressed output deterministic for identical input.

Performance

No measurable change: 10 MiB level-9 response ~153 ms before and after; 160×64 KiB streamed chunks at level 6: 19.7 ms vs 19.1 ms (noise).

Validation

  • tests/middleware/test_gzip.py: 16 passed
  • scripts/check: mypy and ruff clean

Review in cubic

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

@Kludex Kludex changed the title Use zlib.compressobj instead of GzipFile in GZipMiddleware Use zlib.compressobj instead of GzipFile in GZipMiddleware Aug 5, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7da8b6f125

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread starlette/middleware/gzip.py Outdated
@codspeed

codspeed Bot commented Aug 5, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 77.19%

⚡ 70 improved benchmarks
✅ 64 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Memory test_gzip[json-32KiB-level-1] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[json-32KiB-level-6] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[json-32KiB-level-9] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-1MiB-level-1] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-1MiB-level-2] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-1MiB-level-3] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-1MiB-level-4] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-1MiB-level-5] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-1MiB-level-6] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-1MiB-level-7] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-1MiB-level-8] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-1MiB-level-9] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-256KiB-level-1] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-256KiB-level-6] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-256KiB-level-9] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-32KiB-level-1] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-32KiB-level-6] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[text-32KiB-level-9] 806.4 KB 293.9 KB ×2.7
Memory test_gzip[incompressible-32KiB-level-6] 822.5 KB 309.9 KB ×2.7
Memory test_gzip[incompressible-32KiB-level-9] 822.5 KB 309.9 KB ×2.7
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing gzip-compressobj (e178b1c) with main (a375798)

Open in CodSpeed

@Kludex
Kludex merged commit 7ed6b26 into main Aug 5, 2026
15 checks passed
@Kludex
Kludex deleted the gzip-compressobj branch August 5, 2026 07:31
This was referenced Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant