Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zlib: add zstd support #52100

Closed
wants to merge 2 commits into from
Closed

zlib: add zstd support #52100

wants to merge 2 commits into from

Conversation

jkrems
Copy link
Contributor

@jkrems jkrems commented Mar 15, 2024 •
edited
Loading

Adds ZstdCompress and ZsdDecompress to the zlib module which can be used to compress/decompress with the Zstandard ("zstd") algorithm.

Notable omissions:

  • Providing dictionaries isn't implemented.
  • The docs in zlib.md don't call out any params beyond the basic compression level.

The code follows similar patterns to the PR that added Brotli support. Just that instead of brotli, it adds the equivalent zstd APIs. Just like Brotli, this required separate compression/decompression context objects.

Zstd itself has been around and stable for multiple years but this PR is early in terms of web support: It only just starts shipping by default in Chrome 123. On the other hand, by shipping in Chrome it will soon be supported quite widely on the web. Firefox also signaled support (https://bugzilla.mozilla.org/show_bug.cgi?id=1301878#c65). Zstd is now enabled on the web and supported by both Chrome and Firefox: https://caniuse.com/zstd

Official support in node.js would allow passing additional WPTs around fetch (nodejs/undici#2847).

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/security-wg

@nodejs-github-bot nodejs-github-bot added dependencies Pull requests that update a dependency file. needs-ci PRs that need a full CI run. labels Mar 15, 2024
@jkrems jkrems force-pushed the zstd branch 4 times, most recently from ec19b6e to 28914c7 Compare March 17, 2024 00:07
@jkrems
Copy link
Contributor Author

jkrems commented Mar 17, 2024

Alright, got up to making the basics (seemingly) work:

$ out/Debug/node -p 'zlib.zstdDecompressSync(zlib.zstdCompressSync("Hello World")).toString()'
Hello World

@jkrems jkrems force-pushed the zstd branch 6 times, most recently from 383ff46 to 47c7ab8 Compare March 17, 2024 20:59
@jkrems jkrems marked this pull request as ready for review March 17, 2024 21:00
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
@jkrems
deps,tools: add zstd 1.5.5
3aaf9df 
PR-URL: nodejs#52100
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
@jkrems
zlib: add zstd support
9d3b94a 
Fixes: nodejs#48412
PR-URL: nodejs#52100
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
@jkrems
zlib: add zstd support
cefdec3 
Fixes: nodejs#48412
PR-URL: nodejs#52100
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
@jkrems
zlib: add zstd support
d75f43f 
Fixes: nodejs#48412
PR-URL: nodejs#52100
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
@jkrems
zlib: add zstd support
db32a52 
Fixes: nodejs#48412
PR-URL: nodejs#52100
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
@jkrems
zlib: add zstd support
401c399 
Fixes: nodejs#48412
PR-URL: nodejs#52100
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
@jkrems
zlib: add zstd support
f7b4fb3 
Fixes: nodejs#48412
PR-URL: nodejs#52100
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
@jkrems
zlib: add zstd support
066e76b 
Fixes: nodejs#48412
PR-URL: nodejs#52100
@anonrig anonrig added request-ci Add this label to start a Jenkins CI on a PR. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. and removed tsc-agenda Issues and PRs to discuss during the meetings of the TSC. labels Feb 8, 2025
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 8, 2025
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/65051/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/65052/

@anonrig anonrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Feb 8, 2025
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Feb 8, 2025
@nodejs-github-bot
Copy link
Collaborator

Landed in 0782c91...bf12d72

nodejs-github-bot pushed a commit that referenced this pull request Feb 8, 2025
@jkrems @nodejs-github-bot
deps,tools: add zstd 1.5.6
f535310 
PR-URL: #52100
Reviewed-By: Yagiz Nizipli <[email protected]>
nodejs-github-bot pushed a commit that referenced this pull request Feb 8, 2025
@jkrems @nodejs-github-bot
zlib: add zstd support
bf12d72 
Fixes: #48412
PR-URL: #52100
Reviewed-By: Yagiz Nizipli <[email protected]>
@anonrig anonrig mentioned this pull request Feb 8, 2025
Merged
targos pushed a commit that referenced this pull request Feb 10, 2025
@jkrems @targos
deps,tools: add zstd 1.5.6
380a8d8 
PR-URL: #52100
Reviewed-By: Yagiz Nizipli <[email protected]>
targos pushed a commit that referenced this pull request Feb 10, 2025
@jkrems @targos
zlib: add zstd support
4cc7907 
Fixes: #48412
PR-URL: #52100
Reviewed-By: Yagiz Nizipli <[email protected]>
@Phillip9587 Phillip9587 mentioned this pull request Feb 10, 2025
Open
@jkrems jkrems deleted the zstd branch February 10, 2025 15:00
nodejs-github-bot pushed a commit that referenced this pull request Feb 10, 2025
@anonrig
zlib: make all zstd functions experimental
92dae7d 
PR-URL: #56964
Refs: #52100
Reviewed-By: Jan Krems <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Chemi Atlow <[email protected]>
targos pushed a commit that referenced this pull request Feb 11, 2025
@anonrig @targos
zlib: make all zstd functions experimental
a3ca7f3 
PR-URL: #56964
Refs: #52100
Reviewed-By: Jan Krems <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Chemi Atlow <[email protected]>
targos added a commit that referenced this pull request Feb 11, 2025
@targos
2025-02-12, Version 23.8.0 (Current)
aea2631 
Notable changes:

crypto:
  * (SEMVER-MINOR) support --use-system-ca on Windows (Joyee Cheung) #56833
  * (SEMVER-MINOR) added support for reading certificates from macOS system store (Tim Jacomb) #56599
deps:
  * update timezone to 2025a (Node.js GitHub Bot) #56876
  * (SEMVER-MINOR) update ada to v3.0.1 (Yagiz Nizipli) #56452
deps,tools:
  * (SEMVER-MINOR) add zstd 1.5.6 (Jan Krems) #52100
sqlite:
  * (SEMVER-MINOR) allow returning `ArrayBufferView`s from user-defined functions (René) #56790
src:
  * set signal inspector io thread name (RafaelGSS) #56416
  * set thread name for main thread and v8 worker (RafaelGSS) #56416
  * set worker thread name using worker.name (RafaelGSS) #56416
  * use a default thread name for inspector (RafaelGSS) #56416
src, quic:
  * (SEMVER-MINOR) refine more of the quic implementation (James M Snell) #56328
test:
  * (SEMVER-MINOR) add WPT for URLPattern (Yagiz Nizipli) #56452
url:
  * (SEMVER-MINOR) add URLPattern implementation (Yagiz Nizipli) #56452
zlib:
  * (SEMVER-MINOR) add zstd support (Jan Krems) #52100

PR-URL: TODO
@github-actions github-actions bot mentioned this pull request Feb 11, 2025
Merged
nodejs-github-bot added a commit that referenced this pull request Feb 11, 2025
@nodejs-github-bot
2025-02-12, Version 23.8.0 (Current)
4fe8da4 
Notable changes:

crypto:
  * (SEMVER-MINOR) support --use-system-ca on Windows (Joyee Cheung) #56833
  * (SEMVER-MINOR) added support for reading certificates from macOS system store (Tim Jacomb) #56599
deps:
  * update timezone to 2025a (Node.js GitHub Bot) #56876
  * (SEMVER-MINOR) update ada to v3.0.1 (Yagiz Nizipli) #56452
deps,tools:
  * (SEMVER-MINOR) add zstd 1.5.6 (Jan Krems) #52100
sqlite:
  * (SEMVER-MINOR) allow returning `ArrayBufferView`s from user-defined functions (René) #56790
src:
  * set signal inspector io thread name (RafaelGSS) #56416
  * set thread name for main thread and v8 worker (RafaelGSS) #56416
  * set worker thread name using worker.name (RafaelGSS) #56416
  * use a default thread name for inspector (RafaelGSS) #56416
src, quic:
  * (SEMVER-MINOR) refine more of the quic implementation (James M Snell) #56328
test:
  * (SEMVER-MINOR) add WPT for URLPattern (Yagiz Nizipli) #56452
url:
  * (SEMVER-MINOR) add URLPattern implementation (Yagiz Nizipli) #56452
zlib:
  * (SEMVER-MINOR) add zstd support (Jan Krems) #52100

PR-URL: #57005
targos pushed a commit that referenced this pull request Feb 12, 2025
@nodejs-github-bot @targos
2025-02-12, Version 23.8.0 (Current)
1ddf20c 
Notable changes:

crypto:
  * (SEMVER-MINOR) support --use-system-ca on Windows (Joyee Cheung) #56833
  * (SEMVER-MINOR) added support for reading certificates from macOS system store (Tim Jacomb) #56599
deps:
  * update timezone to 2025a (Node.js GitHub Bot) #56876
sqlite:
  * (SEMVER-MINOR) allow returning `ArrayBufferView`s from user-defined functions (René) #56790
src:
  * set signal inspector io thread name (RafaelGSS) #56416
  * set thread name for main thread and v8 worker (RafaelGSS) #56416
  * set worker thread name using worker.name (RafaelGSS) #56416
  * use a default thread name for inspector (RafaelGSS) #56416
url:
  * (SEMVER-MINOR) add URLPattern implementation (Yagiz Nizipli) #56452
zlib:
  * (SEMVER-MINOR) add zstd support (Jan Krems) #52100

PR-URL: #57005
targos pushed a commit that referenced this pull request Feb 13, 2025
@nodejs-github-bot @targos
2025-02-13, Version 23.8.0 (Current)
196efd7 
Notable changes:

crypto:
  * (SEMVER-MINOR) support --use-system-ca on Windows (Joyee Cheung) #56833
  * (SEMVER-MINOR) added support for reading certificates from macOS system store (Tim Jacomb) #56599
deps:
  * update timezone to 2025a (Node.js GitHub Bot) #56876
sqlite:
  * (SEMVER-MINOR) allow returning `ArrayBufferView`s from user-defined functions (René) #56790
src:
  * set signal inspector io thread name (RafaelGSS) #56416
  * set thread name for main thread and v8 worker (RafaelGSS) #56416
  * set worker thread name using worker.name (RafaelGSS) #56416
  * use a default thread name for inspector (RafaelGSS) #56416
url:
  * (SEMVER-MINOR) add URLPattern implementation (Yagiz Nizipli) #56452
zlib:
  * (SEMVER-MINOR) add zstd support (Jan Krems) #52100

PR-URL: #57005
targos pushed a commit that referenced this pull request Feb 13, 2025
@nodejs-github-bot @targos
2025-02-13, Version 23.8.0 (Current)
69d32d1 
Notable changes:

crypto:
  * (SEMVER-MINOR) support --use-system-ca on Windows (Joyee Cheung) #56833
  * (SEMVER-MINOR) added support for reading certificates from macOS system store (Tim Jacomb) #56599
deps:
  * update timezone to 2025a (Node.js GitHub Bot) #56876
sqlite:
  * (SEMVER-MINOR) allow returning `ArrayBufferView`s from user-defined functions (René) #56790
src:
  * set signal inspector io thread name (RafaelGSS) #56416
  * set thread name for main thread and v8 worker (RafaelGSS) #56416
  * set worker thread name using worker.name (RafaelGSS) #56416
  * use a default thread name for inspector (RafaelGSS) #56416
url:
  * (SEMVER-MINOR) add URLPattern implementation (Yagiz Nizipli) #56452
zlib:
  * (SEMVER-MINOR) add zstd support (Jan Krems) #52100

PR-URL: #57005
@J3m5 J3m5 mentioned this pull request Feb 17, 2025
Open
@0434863032 0434863032 mentioned this pull request Feb 19, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@terrelln terrelln terrelln left review comments

@anonrig anonrig anonrig approved these changes

@jasnell jasnell Awaiting requested review from jasnell

Assignees
No one assigned
Labels
c++ Issues and PRs that require attention from people who are familiar with C++. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. dependencies Pull requests that update a dependency file. needs-ci PRs that need a full CI run. notable-change PRs with changes that should be highlighted in changelogs. review wanted PRs that need reviews. semver-minor PRs that contain new features and should be released in the next minor version. zlib Issues and PRs related to the zlib subsystem.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.