Use zlib.compressobj instead of GzipFile in GZipMiddleware - #3411
Conversation
|
Docs preview: https://d50e9243-starlette.marcelotryle.workers.dev |
zlib.compressobj instead of GzipFile in GZipMiddleware
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7da8b6f125
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Merging this PR will improve performance by 77.19%
Performance Changes
Tip Curious why this is faster? Comment Comparing |
Summary
Replace the
gzip.GzipFile+io.BytesIOmachinery inGZipResponderwithzlib.compressobjusingwbits=16 + zlib.MAX_WBITS, which produces the same gzip wire format (header, CRC32 and size trailer).Rationale
The
GzipFileapproach dates back to the original middleware commit (#111, 2018) and requires supporting machinery thatcompressobjmakes unnecessary:io.BytesIObuffer and per-chunkgetvalue()/seek(0)/truncate()danceExitStacklifecycle and__call__override added in Lazily allocate GZipMiddleware compression resources #3407 (acompressobjholds no resources that need closing)GzipFile's internalBufferedWriter, which hides when compression work actually happensUnlike
GzipFile.write(),compressobj.compress()consumes and deflates its entire input during the call, making the compression cost explicit and self-contained. This also simplifies any future work offloading large-chunk compression to a worker thread (#3410), which currently needs aZ_NO_FLUSHworkaround to drainGzipFile's buffer inside the worker. This is the same primitive aiohttp uses in itsZLibCompressor.Net: 17 insertions, 34 deletions.
Behavior
Content-Encoding/Content-Length.Z_SYNC_FLUSH, matching the previousGzipFile.flush()behavior, so every streamed chunk remains immediately decodable by clients.mtime=0equivalent), making compressed output deterministic for identical input.Performance
No measurable change: 10 MiB level-9 response ~153 ms before and after; 160×64 KiB streamed chunks at level 6: 19.7 ms vs 19.1 ms (noise).
Validation
tests/middleware/test_gzip.py: 16 passedscripts/check: mypy and ruff clean