Commit Graph

9629 Commits

Author SHA1 Message Date
Crypto City
d5fbbe4c18 bump min cmake version to 3.5 2025-08-03 12:45:57 +00:00
Felix Handte
02ef78be58
Merge pull request #3184 from htnhan/features/list_verbose_to_show_dictionary_id
zstd -lv <file> to show dictID
2022-07-08 16:04:39 -04:00
htnhan
d7eb829af5 Detect multiple dictIDs in one file 2022-07-08 12:20:50 -05:00
htnhan
cc8c98485a zstd -lv <file> to show dictID 2022-07-05 21:28:33 -05:00
Elliot Gorokhovsky
3ef92cfcd4
Merge pull request #3180 from nocnokneo/MSVCBuildTests
Fix ZSTD_BUILD_TESTS=ON with MSVC
2022-07-05 13:13:34 -04:00
Taylor Braun-Jones
cd9d0a7e6e Fix ZSTD_BUILD_TESTS=ON build with MSVC
Fixes:

    Command line error D8021 : invalid numeric argument '/Wno-deprecated-declarations'
2022-06-30 13:20:42 -04:00
Elliot Gorokhovsky
5d2fb4288f
Merge pull request #3179 from embg/1.5.3_bump
Prepare v1.5.3
2022-06-29 13:03:52 -07:00
Elliot Gorokhovsky
bb3839a78c make -C programs zstd.1 2022-06-29 14:55:14 -04:00
Elliot Gorokhovsky
5c382bf110 1.5.3 version bump 2022-06-29 14:45:53 -04:00
Elliot Gorokhovsky
e9d6fc867a
Merge pull request #3177 from embg/dms_prefetch2
Add prefetchCDictTables CCtxParam (+10-20% cold dict compression speed)
2022-06-24 08:24:43 -07:00
Elliot Gorokhovsky
cb9e341129 Nits 2022-06-23 16:59:21 -04:00
Elliot Gorokhovsky
bb4a3c71ef
Update README.md for fuzzers (#3174)
* Update README.md for fuzzers

* Add ls corpora/*crash command

* nit

* Clarify wording and add Nick's command

* Minor clarification
2022-06-22 21:02:07 -04:00
Elliot Gorokhovsky
747e06f4f6 Add tests 2022-06-22 17:05:23 -04:00
Elliot Gorokhovsky
6bd5ac6713 add prefetchCDictTables to largeNbDicts 2022-06-22 16:13:07 -04:00
Elliot Gorokhovsky
93b89fb24b Add docs 2022-06-22 16:13:07 -04:00
Elliot Gorokhovsky
2a128110d0 Add prefetchCDictTables CCtxParam 2022-06-22 16:13:07 -04:00
Yann Collet
f5c4ec4658
Merge pull request #3175 from facebook/fix3169
Streaming decompression can detect incorrect header ID sooner
2022-06-22 11:21:09 -07:00
Yann Collet
91aeade735 Streaming decompression can detect incorrect header ID sooner
Streaming decompression used to wait for a minimum of 5 bytes before attempting decoding.
This meant that, in the case that only a few bytes (<5) were provided,
and assuming these bytes are incorrect,
there would be no error reported.
The streaming API would simply request more data, waiting for at least 5 bytes.

This PR makes it possible to detect incorrect Frame IDs as soon as the first byte is provided.

Fix #3169
2022-06-21 23:09:03 -07:00
Elliot Gorokhovsky
f6ef14329f
"Short cache" optimization for level 1-4 DMS (+5-30% compression speed) (#3152)
* first attempt at fast DMS short cache

* significant wins for some scenarios

* fix all clang regressions

* nits

* fix 1.5% gcc11 regression on hot 110Kdict scenario

* fix CI

* nit

* Add tags to doublefast hash table

* use tags in doublefast DMS

* Fix CI

* Clean up some hardcoded logic / constants

* Switch forCCtx to an enum

* nit

* add short cache to ip+1 long search

* Move tag size into hashLog

* Minor nits

* Truncate dictionaries greater than 16MB in short cache mode

* Helper function for tag comparison

* Cap short cache hashLog at 24 to prevent overflow

* size_t dictTagsMatch -> int dictTagsMatch

* nit

* Clean up and comment dictionary truncation

* Move ZSTD_tableFillPurpose_e next to ZSTD_dictTableLoadMethod_e

* Comment and expand helper functions

* Asserts and documentation

* nit
2022-06-21 17:27:19 -04:00
Yann Collet
eb842a2260
Merge pull request #3170 from facebook/mesongnu99
removed gnu99 statement from meson recipe
2022-06-21 10:17:36 -07:00
Yann Collet
15f3605135 removed gnu99 statement from meson recipe 2022-06-20 18:18:40 -07:00
Yann Collet
3367e6d414
Merge pull request #3167 from facebook/cmake_std
remove explicit standard setting from cmake script
2022-06-19 16:49:21 -07:00
Yann Collet
eceecc5b2c removed explicit compilation standard from cmake script
it's not expected to be useful
and can actually lead to subtle side effects
such as #3163.
2022-06-19 14:52:32 -07:00
Yann Collet
f15dd6420c
Merge pull request #3166 from facebook/warning_clockt
display a warning message when using C90 clock_t
2022-06-19 14:45:49 -07:00
Yann Collet
574ecbb0fc display a warning message when using C90 clock_t for MT speed measurements. 2022-06-19 11:38:06 -07:00
Yann Collet
b33ef91694 updated documentation regarding build systems 2022-06-19 11:12:16 -07:00
Elliot Gorokhovsky
b7b7edb3a3
Merge pull request #3161 from embg/largeNbDictsImprovements
[contrib] largeNbDicts bugfix + improvements
2022-06-15 07:39:50 -07:00
Elliot Gorokhovsky
24364057bc
fix typo
Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
2022-06-14 19:18:49 -04:00
Elliot Gorokhovsky
2bbdc9f40e Fix FILE handle leak 2022-06-14 14:57:54 -07:00
Elliot Gorokhovsky
f7ebbcd0cc Support advanced API so forceCopy/forceAttach works properly 2022-06-14 14:52:51 -07:00
Elliot Gorokhovsky
e0c4863c5c largeNbDicts bugfix + improvements 2022-06-13 17:26:44 -07:00
Elliot Gorokhovsky
b944db0c45
Merge pull request #3160 from danlark1/patch-1
Fix big endian ARM NEON path
2022-06-13 14:01:43 -04:00
Daniel Kutenin
05f3f415ce
Fix big endian ARM NEON path
It is not using the NEON acceleration but the bit grouping was applied
2022-06-13 09:16:24 +01:00
Nick Terrell
3b1bd91852
Merge pull request #3141 from JunHe77/seqDec
dec: adjust seqSymbol load on aarch64
2022-06-09 13:40:51 -07:00
Nick Terrell
3b915cd94b
Merge pull request #3145 from JunHe77/wildcopy
common: apply two stage copy to aarch64
2022-06-09 13:38:30 -07:00
Elliot Gorokhovsky
f313a773a4
Merge pull request #3157 from embg/huge_dict_bugfix
Bugfix for huge dictionaries
2022-06-09 15:35:29 -04:00
Elliot Gorokhovsky
31bd6402c6 Bugfix for huge dictionaries 2022-06-09 11:39:30 -04:00
Yann Collet
27bf96e72b updated --single-thread man 2022-06-07 17:45:15 -07:00
Nick Terrell
802ad778cc
Merge pull request #3154 from terrelln/rsyncable-speed-fix
Remove expensive assert in --rsyncable hot loop
2022-06-06 16:07:20 -07:00
Nick Terrell
7c05b9aec3 Remove expensive assert in --rsyncable hot loop
This assert slows the loop down by 10x. We can get similar
coverage by asserting at the beginning & end of the loop.

We need this fix because Debian compiles zstd with asserts
enabled. Separately, we should ask them why, and if they would
consider disabling asserts in their builds. Since we don't
optimize for assert enabled builds.

Fixes Issue #3150.
2022-06-06 11:56:13 -07:00
Nick Terrell
9f346dbe45
Merge pull request #3147 from animalize/dev
fix leaking thread handles on Windows
2022-06-02 10:04:55 -07:00
Yann Collet
b06d10adbc
Merge pull request #3148 from ihsinme/patch-1
simple fix
2022-06-02 09:58:45 -07:00
Jun He
2491c65937 dec: adjust seqSymbol load on aarch64
ZSTD_seqSymbol is a structure with total of 64 bits
wide. So it can be loaded in one operation and
extract its fields by simply shifting or extracting
on aarch64.
GCC doesn't recognize this and generates more
unnecessary ldr/ldrb/ldrh operations that cause
performance drop.
With this change it is observed 2~4% uplift of
silesia and 2.5~6% of cantrbry @L8 on Arm N1.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I7748909204cf78a17eb9d4f2333692d53239daa8
2022-05-30 22:01:38 +08:00
ihsinme
5081ccb056
Update zstd_compress.c 2022-05-30 14:08:19 +03:00
Ma Lin
95073b1af1 fix leaking thread handles on Windows
On Windows, thread handle should be closed explicitly.

Co-authored-by: luben karavelov <luben@users.noreply.github.com>
2022-05-30 16:35:44 +08:00
Jun He
d7249dafb4 common: apply two stage copy to aarch64
On aarch64 ZSTD_wildcopy uses a simple loop to do
16B based memory copy. There is existing optimized
two stage copy that can achieve better performance.
By applying this to aarch64 it is also observed ~1%
uplift in silesia corpus.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: Ic1253308e7a8a7df2d08963ba544e086c81ce8be
2022-05-26 14:40:21 +08:00
Yann Collet
9a5e73c74e
Merge pull request #3143 from facebook/fixdoc_3142
fix small error in format documentation example
2022-05-24 10:19:14 -07:00
Nick Terrell
1c8a6974c7
Merge pull request #3139 from danlark1/dev
[lazy] Optimize ZSTD_row_getMatchMask for levels 8-10 for ARM
2022-05-24 11:10:26 -04:00
Yann Collet
f33ccd2d1b fix small error in format documentation example
reported by @dkcasset
fix #3142
2022-05-24 04:47:49 -07:00
Danila Kutenin
9166c6ae20 Again unused error warning. Fixed 2022-05-23 14:51:47 +00:00