Commit Graph

1736 Commits

Author SHA1 Message Date
Elliot Gorokhovsky
e9d6fc867a
Merge pull request #3177 from embg/dms_prefetch2
Add prefetchCDictTables CCtxParam (+10-20% cold dict compression speed)
2022-06-24 08:24:43 -07:00
Elliot Gorokhovsky
bb4a3c71ef
Update README.md for fuzzers (#3174)
* Update README.md for fuzzers

* Add ls corpora/*crash command

* nit

* Clarify wording and add Nick's command

* Minor clarification
2022-06-22 21:02:07 -04:00
Elliot Gorokhovsky
747e06f4f6 Add tests 2022-06-22 17:05:23 -04:00
Yann Collet
91aeade735 Streaming decompression can detect incorrect header ID sooner
Streaming decompression used to wait for a minimum of 5 bytes before attempting decoding.
This meant that, in the case that only a few bytes (<5) were provided,
and assuming these bytes are incorrect,
there would be no error reported.
The streaming API would simply request more data, waiting for at least 5 bytes.

This PR makes it possible to detect incorrect Frame IDs as soon as the first byte is provided.

Fix #3169
2022-06-21 23:09:03 -07:00
W. Felix Handte
1bc8019e10 Update results.csv 2022-05-11 10:27:35 -07:00
Elliot Gorokhovsky
3be9a81e46 Update results.csv 2022-05-04 16:05:37 -04:00
Yann Collet
460780f804
Merge pull request #3094 from dirkmueller/usage_cleanup
Split help in long and short version, cleanup formatting
2022-04-05 07:09:54 -07:00
Yann Collet
678bfff4fe fix minor bug in sequence_compression_api tester
margin was merely slightly too short for extra splitting.
2022-03-29 16:45:09 -07:00
Dirk Müller
7fbe60d577
Split help in long and short version, cleanup formatting
Adopt the more standard Usage: formatting style
List short and long options alongside where available
Print lists as a table
Use command style description
2022-03-29 12:57:47 +02:00
Nick Terrell
f229daaf42
Merge pull request #3052 from dirkmueller/gzip_keep
Keep original file if -c or --stdout is given
2022-03-28 10:35:21 -07:00
Dominique Pelle
b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
Dirk Müller
7a3997c21a
Handle newer less versions in zstdless testing
Newer less versions appear to have changed how stderr
and stdout are showing error messages. hardcode the
expected behavior to make the tests pass with any less version.

Also set locale to C so that the strings are matching.
2022-03-10 09:47:33 +01:00
Dirk Müller
3f4f8b04ed
Keep original file if -c or --stdout is given
Set removeSrcFile back to false when -c or --stdout is used to improve
compatibility with gzip(1) behavior.

gzip(1) is removing the original file on compression unless --stdout or
/-c is used. zstd is defaulting to keep the file unless --rm is used or
when it is called via a gzip symlink, in which it is removing by
default. Specifying -c/--stdout turns this behavior off.
2022-03-09 23:38:59 +01:00
Felix Handte
8814aa5bfa
Merge pull request #3059 from dirkmueller/gzip_fast_best
Implement more gzip compatibility (#3037)
2022-03-08 15:32:06 -05:00
Dimitris Apostolou
cf1894b324
Fix typos 2022-03-05 23:47:25 +02:00
Xi Ruoyao
d109cef201
fix the assertion in readLinesFromFile (#3084)
* fix the assertion in readLinesFromFile

When the file is not terminated by endline, readLineFromFile will append
a '\0' for the last line.  In this case pos + lineLength == dstCapacity.

* test: don't print very long text garbage
2022-03-04 14:56:44 -05:00
Nick Terrell
da737c7ab8 [programs] Fix infinite loop when empty input is passed to trainer
When an empty input file was passed to the dictionary trainer, it would infinite loop.
The added test case exposes the bug, and is fixed with this PR.
2022-03-02 11:05:26 -08:00
Elliot Gorokhovsky
71d9dab76f Replace XOR with subtraction for readability 2022-02-16 16:49:42 -05:00
Elliot Gorokhovsky
856c7dc51d Fix fuzzer.c nits and replace CLZ fallback 2022-02-16 11:40:05 -05:00
Elliot Gorokhovsky
7c674a0919 Add tests for bitwise intrinsics 2022-02-14 18:36:33 -05:00
Elliot Gorokhovsky
db2f4a6532 Move bitwise builtins into bits.h 2022-02-14 11:16:03 -05:00
Dirk Müller
e653e97f77
Implement more gzip compatibility (#3037)
-n --no-name is the current behavior already, so we can implement
this as a noop.

--best is an alias for -9 in gzip

add basic cli tests.
2022-02-09 23:09:18 +01:00
binhdvo
936ae8a3a0
Move zstdgrep and zstdless tests to cli-tests (#3057)
* Move zstdgrep and zstdless tests to cli-tests

Co-authored-by: Binh Vo <binhvo@fb.com>
2022-02-09 11:21:33 -05:00
Nick Terrell
169f8c11ff [cli-tests] Fix zstd symlinks
The zstd symlinks, notably `zstdcat`, weren't working as expected
because only the `tests/cli-tests/bin/zstd` wrapper was symlinked. We
still invoked `zstd` with the name `zstd`. The fix is to create a
directory of zstd symlinks in `tests/cli-tests/bin/symlinks` for each
name that zstd recognizes. And when `tets/cli-tests/bin/zstd` is
invoked, it selects the correct symlink to call.

See the test `zstd-cli/zstdcat.sh` for an example of how it would work.
2022-02-07 15:20:42 -08:00
Yann Collet
be0a4b4df6
Merge pull request #3046 from rex4539/typos
Fix typos
2022-02-04 07:50:25 -08:00
Yonatan Komornik
4bba97b4cb
Macos playtest envvars fix (#3035)
* playtests.sh: fix for a bug in macos' /bin/sh that persists temporary env vars when introduced before function calls
* cli-tests/run.py: Do not use existing ZSTD* envvars
2022-02-03 18:42:20 -08:00
Dimitris Apostolou
4c4d403ecb
Fix typos 2022-02-02 19:32:31 +02:00
Yann Collet
cdee6a7dbd Merge branch 'dev' into fix44168 2022-01-31 17:31:55 -08:00
Yann Collet
9a758ce520 update sequence_compression_api fuzzer test
to check for under-sized dstCapacity.
2022-01-31 16:17:11 -08:00
Yonatan Komornik
cc0657f27d
AsyncIO compression part 2 - added async read and asyncio to compression code (#3022)
* Compression asyncio:
- Added asyncio functionality for compression flow
- Added ReadPool for async reads, implemented in both comp and decomp flows
2022-01-31 15:43:41 -08:00
Nick Terrell
0b70da6277
Merge pull request #3020 from terrelln/cli-tests
Add new CLI testing platform
2022-01-31 10:02:27 -08:00
Yann Collet
637b2d7a24 fixed bug 44168
discovered by oss-fuzz

It's a bug in the test itself :
ZSTD_compressBound() as an upper bound of the compress size
only works for data compressed "normally".
But in situations where many flushes are forcefully introduced,
this creates many more blocks,
each of which has a potential to increase the size by 3 bytes.
In extreme cases (lots of small incompressible blocks), the expansion can go beyond ZSTD_compressBound().

This situation is similar when using the CompressSequences() API
with Explicit Block Delimiters.
In which case, each explicit block acts like a deliberate flush.
When employed by a fuzzer, it's possible to generate scenarios like the one described above,
with tons of incompressible blocks of small sizes,
thus going beyond ZSTD_compressBound().

fix : when using Explicit Block Delimiters, use a larger bound, to account for this scenario.
2022-01-29 16:36:20 -08:00
Yann Collet
c9072dd60a
Merge pull request #3027 from brailovich/dev
fix for -r on empty directory
2022-01-29 15:40:43 -08:00
Yann Collet
85a13250c7
Merge pull request #3030 from terrelln/verbose-version
Print zlib/lz4/lzma library versions in verbose version output
2022-01-29 15:40:09 -08:00
Yann Collet
d64d5ddc57 fix 44122 test error
It's a bug in the test itself, in exceptional circumstances (no more space for additional sequence).

There should be enough room for all cases to work fine from now on,
and if not, we have an additional `assert()` to catch that situation.
2022-01-27 14:54:18 -08:00
Nick Terrell
1fc42de86a [CI] Hook cli-tests up to CI
Add cli-tests to `make test`. This adds a `python3` dependency to `make
test`, but not `make check`. We could make this dependency optional by
skipping the tests if `python3` is not present.
2022-01-27 13:56:59 -08:00
Nick Terrell
f3096ff6d1 [test] Add new CLI testing platform
Adds the new CLI testing platform that I'm proposing.
See the added `README.md` for details.
2022-01-27 13:56:59 -08:00
Nick Terrell
f088c430e3 [datagen] Remove extra newline printed
`datagen` was printing a `\n` even when it had no other output. Raise
the output level for the final `\n` to the minimum output level used.

This minor bug was caught by the new testing framework.
2022-01-27 13:56:59 -08:00
brailovich
9f37d1fede
Update playTests.sh
combination of -r with empty folder simplified to comply with sh compatibility tests
2022-01-27 08:22:05 -08:00
Yann Collet
bad7f82300
Merge pull request #2974 from facebook/fix2966_part3
Lazy parameters adaptation (part 1 - ZSTD_c_stableInBuffer)
2022-01-27 06:14:04 -08:00
brailovich
501a353b91
Update playTests.sh 2022-01-26 18:56:52 -08:00
Yann Collet
f2d9652ad8 more usage of new error code stabilityCondition_notRespected
as suggested by @terrelln
2022-01-26 18:30:55 -08:00
Nick Terrell
e60eba58bf Print zlib/lz4/lzma library versions in verbose version output
Knowing the version of zlib/lz4/lzma we're linking against is very
useful for debugging issues with those libraries, so print it out in the
verbosity 4 version output.

Also print this information at the top of `playTests.sh`.
2022-01-26 18:25:58 -08:00
brailovich
5e7523385b
Update playTests.sh 2022-01-26 16:53:11 -08:00
Yann Collet
dda4c10f07 added ZSTD_compressStream2() + ZSTD_c_stableInBuffer test 2022-01-26 13:33:04 -08:00
Yann Collet
af3d9c506e added streaming test starting from non-0 pos 2022-01-26 10:31:25 -08:00
Yann Collet
c1668a00d2 fix extended case combining stableInBuffer with continue() and flush() modes 2022-01-26 10:31:25 -08:00
Yann Collet
270f9bf005 better consistency in accessing @input
as suggested by @terrelln.

Also : commented zstreamtest more
to ensure ZSTD_stableInBuffer is tested/
2022-01-26 10:31:24 -08:00
Yann Collet
27d336b099 minor behavior refinements
specifically, there is no obligation to start streaming compression with pos=0.
stableSrc mode is now compatible with this setup.
2022-01-26 10:31:24 -08:00
Yann Collet
37b87add7a make stableSrc compatible with regular streaming API
including flushStream().

Now the only condition is for `input.size` to continuously grow.
2022-01-26 10:31:24 -08:00