townforge/zstd - zstd - Townforge git

Author	SHA1	Message	Date
Yann Collet	33d643124d	fixed fullbench-lib target	2018-10-04 18:12:36 -07:00
Yann Collet	efbc3e823d	fixed paramgrill wrong assert() conditions and slightly refactored affected function. Honestly, the formula calculating variance should get a second reviewing round, it's not clear if it's correct.	2018-10-04 14:27:13 -07:00
Yann Collet	c9843ec232	Merge pull request #1348 from facebook/donotdelete Fix #1082	2018-10-02 16:37:58 -07:00
Yann Collet	9012b6cba0	./zstd -f do no longer overwrite destination file if source file does not exist (#1082)	2018-10-01 17:16:34 -07:00
Yann Collet	c7bd6a41ab	zstd -d -f do no longer erase destination file when source file does not exist (#1082)	2018-10-01 14:04:00 -07:00
Nick Terrell	eb4423e7ed	Fix another warning	2018-09-28 14:24:38 -07:00
Nick Terrell	0e7a7f1def	Fix warnings	2018-09-28 12:14:24 -07:00
Nick Terrell	146049a1ea	[zstreamtest] Add failing test case	2018-09-28 12:09:14 -07:00
Nick Terrell	e06f91a169	Merge pull request #1343 from terrelln/faster-ci [zstreamtest] Reduce memory of newapi tests	2018-09-27 19:29:02 -07:00
Yann Collet	913a0365b6	Merge branch 'dev' into donotdelete	2018-09-27 19:23:55 -07:00
Yann Collet	ff36513556	fixed longmatch test too	2018-09-27 18:24:41 -07:00
Yann Collet	73773c6b6a	fixed legacy compilation tests for some reason, these tests started failing recently on CircleCI	2018-09-27 18:15:14 -07:00
Yann Collet	9b45db7fa6	minor refactoring of --list trying to reduce recurrent patterns.	2018-09-27 16:49:08 -07:00
Nick Terrell	d8c73cd607	Reset number of threads less often	2018-09-27 15:49:31 -07:00
Nick Terrell	f2d6db45cd	[zstd] Add -Wmissing-prototypes	2018-09-27 15:24:48 -07:00
Nick Terrell	7ee910e86b	More aggressive limitations	2018-09-27 13:55:24 -07:00
Nick Terrell	ca0cfa3dbd	[zstreamtest] Reduce memory of newapi tests We could allocate up to 2^28 bytes of memory when using 2 threads with window log = 24. Now, we limit it to 2^26 bytes of memory when not running big tests. I chose max window log = 22 since that is the maximum source size when big tests are disabled. Hopefully this will be enough to reduce or eliminate the test failures.	2018-09-27 12:48:29 -07:00
Yann Collet	2a5cd8535a	Merge pull request #1342 from facebook/fixcatyd fix : huge (>4GB) chain of blocks	2018-09-27 10:20:14 -07:00
Yann Collet	9bce916732	Merge pull request #1340 from terrelln/generate-artifacts Publish artifacts with CircleCI	2018-09-26 15:52:23 -07:00
Yann Collet	8883af6a1e	Merge pull request #1327 from facebook/adapt Adaptive compression	2018-09-26 14:39:08 -07:00
Yann Collet	f98c69d77c	fix : huge (>4GB) stream of blocks experimental function ZSTD_compressBlock() is designed for very small data in mind, for situation where saving the ~12 bytes of frame header can actually make a difference. Some systems though may have to deal with small and large data entangled. If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round. That's why it's possible to compress in multiple rounds. This is a chain of compressed blocks. Some users push this capability to the limit, encoding gigantic chain of blocks. On crossing the 4GB limit, some internal overflow occurs. This fix moves the overflow correction mechanism higher in the call chain, so that it's applied also to gigantic chains of blocks. Added a test case in fuzzer.c, which crashes before the fix, and pass now.	2018-09-26 14:24:28 -07:00
Yann Collet	65ed6eeefb	Merge pull request #1337 from facebook/test_failure fixed `!` tests	2018-09-26 13:40:20 -07:00
Yann Collet	8ff17a6a09	Merge pull request #1329 from facebook/v04isout Changed default legacy support to v0.5+	2018-09-26 13:39:05 -07:00
Nick Terrell	3ff6040848	Publish artifacts with CircleCI * Updates CircleCI to use workflows. We can now specify any number of test jobs to run in parallel. * Switch the image to `buildpack-deps:trusty` which is only 500 MB instead of 7 GB, so that saves 7 minutes to download it if it isn't already cached on the host. * Publish the source tarball and sha256sum as artifacts. * If the `GITHUB_TOKEN` environment variable is set, we will also add the tarball + sha256sum to the tagged release, after manual approval.	2018-09-26 13:23:28 -07:00
Yann Collet	6c51bf420c	bounds for --adapt mode can supply min and max compression level through advanced command : --adapt=min=#,max=#	2018-09-25 16:03:28 -07:00
Yann Collet	63abaf2171	fixed `!` tests Sometimes, it's necessary to test that a certain command fail, as expected. Such failure is actually a success, and must not stop the flow of tests. Several tests were prefixed with `!` to invert return code. This does not work : it effectively makes the tests pass no matter what. Use instead function die(), which is meant to trap successes, and transform them into errors.	2018-09-25 15:57:28 -07:00
Yann Collet	04f47bbdd2	Merge branch 'dev' into adapt	2018-09-24 16:56:45 -07:00
Yann Collet	0e211bdd18	fixed constant comparison on 32-bits systems	2018-09-21 18:23:32 -07:00
Yann Collet	484f40697b	fix constant redeclaration in paramgrill	2018-09-21 17:28:37 -07:00
Yann Collet	a54c86cfc6	defined a minimum negative level which can be probed using new function ZSTD_minCLevel(). Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency used the opportunity to bump version number to v1.3.6	2018-09-20 16:52:03 -07:00
Yann Collet	db97310ace	fixed versions-test to only test v0.5+ since zstd_devel is no longer compatible with v0.4+	2018-09-20 14:59:11 -07:00
Yann Collet	15519479ba	fixed minor gcc warning on a unused variable	2018-09-20 13:00:11 -07:00
Yann Collet	45010da074	updated man page and added `--adapt` test in `playTests.sh`	2018-09-19 17:37:22 -07:00
Yann Collet	2f78228f65	Merge branch 'dev' into adapt	2018-09-19 12:43:42 -07:00
Björn Ketelaars	06fd1e473d	'head -c BYTES' is non-portable. tests/playTests.sh uses 'head -c' in a couple of tests to truncate the last byte of a file. The '-c' option is non-portable (not in POSIX). Instead use a wrapper around dd (truncateLastByte).	2018-09-17 20:39:35 +02:00
Yann Collet	d195eec97e	fixed msan error cold dictionary is detected through a comparison with dictEnd, which was not initialized at the beginning of first DCtx usage.	2018-09-13 12:29:52 -07:00
Yann Collet	31ebb26945	Merge pull request #1301 from terrelln/lit-size [zstd] Fix seqStore growth	2018-08-28 17:10:25 -07:00
Nick Terrell	e3b5286197	Fix decodecorpus	2018-08-28 13:56:47 -07:00
Nick Terrell	e984d01912	Small test fixes	2018-08-28 13:42:01 -07:00
Nick Terrell	5a4e6c9f3d	[fuzzer] Test growing the seqStore_t	2018-08-28 13:20:37 -07:00
Yann Collet	b37a0a6bde	Merge pull request #1298 from facebook/bench Refactored bench.c	2018-08-28 12:25:02 -07:00
Yann Collet	55affc09de	timedFn : measurement delay is programmable instead of hard-coded 1 second per measurement	2018-08-28 11:26:27 -07:00
Yann Collet	0ff9b67552	paramgrill: removed useless tests designed to compensate iter_mode, but since only time_mode is available now, all tests are guaranteed to last a minimum amount of time.	2018-08-27 19:07:17 -07:00
Yann Collet	9e26893e07	paramgrill: fixed a bunch of div-by-zero they were pretty easy to trigger by the way, just start an extended paramgrill session to find a compression table based on any sample, it would necessarily happen at some point.	2018-08-27 18:47:09 -07:00
Yann Collet	0071e8348f	restored assert() in paramgrill assert() in paramgrill are not in the benchmark path. They should remain active, as they don't impact measurements, and their runtime is insignificant.	2018-08-27 17:52:04 -07:00
Yann Collet	01dcd0fd17	bench: minor api update, for consistency BMK_benchTimedFn() BMK_isCompleted_TimedFn() uses TimedFnState	2018-08-26 21:30:18 -07:00
Yann Collet	c3a4baaf6e	fixed minor warnings valgrind: memory leak of a few bytes in fullbench static analyzer: uninitialized data passed as result	2018-08-24 23:25:35 -07:00
Yann Collet	af23d39eb8	Merge pull request #1297 from felixhandte/check-offset-table Fix Missing Offset Table Check	2018-08-24 17:36:44 -07:00
Yann Collet	2279f3d127	bench: reduce nb of return type runOutcome is enough removed timedFnOutcome	2018-08-24 17:28:38 -07:00
W. Felix Handte	db4c8d05b3	Add Failing Test	2018-08-24 14:30:21 -07:00
Yann Collet	7b23cc4d1e	fixed fullbench behavior now same as v1.3.5	2018-08-24 12:40:10 -07:00
Yann Collet	4da5bdf482	fixed zstd -b speed result the benchmark was displaying the speed of last run instead of the best of all previous runs.	2018-08-23 18:13:49 -07:00
Nick Terrell	3b56bb1e4c	Fix decodecorpus	2018-08-23 17:48:06 -07:00
Yann Collet	b0e1f3982d	fixed paramgrill to work with new bench.c	2018-08-23 17:21:38 -07:00
Yann Collet	1f9ec13621	introduced MB_UNIT so that all benchmarking programs use the same speed scale	2018-08-23 16:03:30 -07:00
Yann Collet	d39a25c5ed	update fullbench.c to work with new bench.h	2018-08-23 15:00:09 -07:00
Jennifer Liu	9d6ed9def3	Merge fastCover into DictBuilder (#1274 ) * Minor fix * Run non-optimize FASTCOVER 5 times in benchmark * Merge fastCover into dictBuilder * Fix mixed declaration issue * Add fastcover to symbol.c * Add fastCover.c and cover.h to build * Change fastCover.c to fastcover.c * Update benchmark to run FASTCOVER in dictBuilder * Undo spliting fastcover_param into cover_param and f * Remove convert param functions * Assign f to parameter * Add zdict.h to Makefile in lib * Add cover.h to BUCK * Cast 1 to U64 before shifting * Remove trimming of zero freq head and tail in selectSegment and rebenchmark * Remove f as a separate parameter of tryParam * Read 8 bytes when d is 6 * Add trimming off zero frequency head and tail * Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed) * Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary * Change nbDmer to always read 8 bytes even when d=6 * Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER * Update comments and benchmarking result * Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover * Add dictType enum and fix bug about passing zParam when converting to coverParam * Combine finalize and skip into a single parameter * Update acceleration parameters and benchmark on 3 sample sets * Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets * Initialize variables outside of for loop in benchmark.c * Update benchmark result for hg-manifest * Remove cover.h from install-includes * Add explanation of f * Set default compression level for trainFromBuffer to 3 * Add assertion of fastCoverParams in DiB_trainFromFiles * Add checkTotalCompressedSize function + some minor fixes * Add test for multithreading fastCovr * Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish * Free segmentFreqs * Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment * Add FASTCOVER_MEMMULT * Minor fix * Update benchmarking result	2018-08-23 12:06:20 -07:00
Yann Collet	77e805e3db	bench: changed creation/reset function to timedFnState for consistency	2018-08-21 18:19:27 -07:00
Yann Collet	801e3bcd97	Merge pull request #1290 from edenzik/ezik/1119-safe-strcpy-in-fileio Fixed unsafe string copy and concat in `fileio.c`.	2018-08-21 13:18:44 -07:00
Eden Zik	78af534f82	Fixed unsafe string copy and concat in `fileio.c`. Per warnings from flawfinder: "Does not check for buffer overflows when copying to destination [MS-banned] (CWE-120). Consider using snprintf, strcpy_s, or strlcpy (warning: strncpy easily misused).". Replaced called to strcpy and strcat in `fileio.c` to calls with a specified size (`strncpy` and `strncat`). Tested the changes on OSX, Linux, Windows. On OSX + Linux, changes were tested with ASAN. The following flags were used: 'check_initialization_order=1:strict_init_order=1:detect_odr_violation=1:detect_stack_use_after_return=1' To reproduce warning: ./flawfinder.py ./programs/fileio.c	2018-08-20 22:15:24 -04:00
Yann Collet	ea0b5fc193	Merge pull request #1285 from facebook/scanbuild static analyzer tests	2018-08-17 16:38:41 -07:00
Yann Collet	b4e7f71055	Merge branch 'dev' into adapt	2018-08-17 15:54:13 -07:00
George Lu	3959ba15e6	Clarify README	2018-08-16 17:22:29 -07:00
George Lu	8175b28f03	Fix negative lvl display value Also fix synthetic benchmark parameter setting	2018-08-16 16:46:37 -07:00
George Lu	239e114d62	prune comments	2018-08-15 16:04:34 -07:00
George Lu	8a296d3e1f	Move Stuff around Group similar functions together, remove outdated comments	2018-08-15 16:04:34 -07:00
George Lu	3f8b10baa1	consts	2018-08-15 16:04:34 -07:00
George Lu	46be2ef5d8	Remove unused stuff	2018-08-15 16:04:34 -07:00
George Lu	b234870c33	clarify display README	2018-08-15 14:29:49 -07:00
George Lu	ee77ddc28d	Fix wraparound	2018-08-15 14:01:32 -07:00
George Lu	1e8d352930	silencing params	2018-08-15 14:01:32 -07:00
George Lu	2c5fdae0ae	Clean up repetitive display Add documentation	2018-08-15 14:01:32 -07:00
George Lu	4d9c6f51b8	-q -v options	2018-08-15 14:01:32 -07:00
George Lu	3dcfe5cc2c	begin display changes	2018-08-15 14:01:32 -07:00
Yann Collet	3692c31598	Merge branch 'dev' into scanbuild	2018-08-15 13:50:49 -07:00
George Lu	b1d9ca737a	Add memoTable options -hashing memotable -no memotable	2018-08-15 10:19:38 -07:00
George Lu	8c918edd3a	MAke it easier to add params Make memoTable size limited	2018-08-14 16:15:46 -07:00
George Lu	96725989ef	Temp fix perf regression	2018-08-14 16:14:37 -07:00
George Lu	3f2d024dca	forceAttachDict	2018-08-14 14:24:41 -07:00
George Lu	e3c679484a	Add Time Checks Fix double -> U64 display	2018-08-14 14:24:41 -07:00
George Lu	88dda92285	Reduce Duplication Change Defaults Asserts actually disabled in paramgrill + fullbench	2018-08-14 14:24:41 -07:00
George Lu	f581ccd267	Doc Updates Add option to pass in existing parameters in use	2018-08-14 14:24:41 -07:00
George Lu	76acba025d	scan-build	2018-08-14 12:13:05 -07:00
George Lu	614aaa3ae1	rebase clevel	2018-08-14 10:53:04 -07:00
George Lu	3b36fe5c68	strategy switching	2018-08-13 16:36:14 -07:00
George Lu	d4730a4f66	Update fulltable to use same interface Add seperateFiles flag	2018-08-13 16:15:52 -07:00
George Lu	43b4971ca8	Renames, Documentation Updates	2018-08-13 16:15:52 -07:00
George Lu	a884b76bc2	Style Changes Add single run dictionaries Change MB to be consistent 1 << 20 rather than 1,000,000	2018-08-13 16:15:52 -07:00
George Lu	b3544217b7	Cleanup	2018-08-13 16:15:52 -07:00
George Lu	8ff0de15e4	Generalize, macro magic numbers	2018-08-13 16:15:52 -07:00
George Lu	3a2e95eba4	Perf improvements try decay strategy selection skipping	2018-08-13 16:15:52 -07:00
George Lu	2bdfe6ca71	Better Display	2018-08-13 16:15:52 -07:00
George Lu	f67d040c39	Bugfixes, style changes Complete euclidean distance climb	2018-08-13 16:15:52 -07:00
George Lu	5f4502fc07	New climb feas part 2 uses euclidean metric	2018-08-13 16:15:52 -07:00
George Lu	13611249a5	Table Compiling +Euclidean Metric	2018-08-13 16:15:52 -07:00
George Lu	0cea754024	Revert "Reorder declaration" This reverts commit `3ac2c22485`.	2018-08-13 16:15:34 -07:00
George Lu	486e586eed	Revert "Default lvl 1" This reverts commit `0cc75d6ee0`.	2018-08-13 16:13:46 -07:00
George Lu	0cc75d6ee0	Default lvl 1 MB to 2^20	2018-08-13 14:55:56 -07:00
Yann Collet	09c9cf3f51	simplified rateLimiter resists better to changing in/out conditions limits risks of "catching up"	2018-08-13 12:13:47 -07:00
Yann Collet	e11f91b039	remove error message for Ctrl+C	2018-08-13 11:48:25 -07:00
Yann Collet	f3aa510738	rateLimiter does not "catch up" when input speed is slow	2018-08-13 11:38:55 -07:00
Yann Collet	a996b1fd2d	fixed rate limited for high speed	2018-08-10 17:39:00 -07:00
Yann Collet	681a382eea	added rateLimiter.py, by @felixhandte this rate limiter avoid the problem of `pv` which "catch up" after a blocked period instead of preserving a constant speed cap.	2018-08-10 12:25:52 -07:00
George Lu	3ac2c22485	Reorder declaration	2018-08-09 16:38:32 -07:00
George Lu	bfe8392e23	Remove ctx from benchMem	2018-08-09 12:07:57 -07:00
George Lu	0ece2e5cdc	Add consts + fix gcc-8 warnings	2018-08-09 11:38:09 -07:00
George Lu	6f480927af	argument parsing cleanup + clarifying comment	2018-08-09 10:42:58 -07:00
George Lu	ad16a69408	Readability improvements, renaming	2018-08-09 10:42:58 -07:00
George Lu	8278a49cb6	const srcPtrs	2018-08-09 10:42:58 -07:00
George Lu	3d230db853	Change speed representation from floating point to integral	2018-08-09 10:42:58 -07:00
George Lu	8faeb41679	Update Documentation Change comment // to /* */ Add more description of what functions do Remove outdated comments	2018-08-09 10:42:58 -07:00
George Lu	dd270b2f75	Renaming / Style fixes	2018-08-09 10:42:58 -07:00
George Lu	e148db366e	Separate capacity vs size Also: Make suggested fixes -varInds_t -reorder some arguments -remove code duplication -update README / -h -Fix memory leaks	2018-08-09 10:42:58 -07:00
George Lu	df026e159f	Fix windows implicit casting bugs	2018-08-09 10:42:58 -07:00
George Lu	0f91b039ff	Add Levels	2018-08-09 10:42:58 -07:00
George Lu	7b5b3d7ae3	BenchMem with block compressed sizes passed back up	2018-08-09 10:42:58 -07:00
George Lu	3adc217ea4	Total Changes: Add different constraint types (decompression speed, compression memory, parameter constraints) Separate search space by strategy + strategy selection Memoize results Real random restarts Support multiple files Support Dictionary inputs Debug Macro for extra printing	2018-08-09 10:42:58 -07:00
George Lu	fab4438801	Dictionary + Multiple file Loading	2018-08-09 10:42:58 -07:00
George Lu	eb21b7f482	Not crashing	2018-08-09 10:42:58 -07:00
George Lu	5f49034520	Working V1	2018-08-09 10:42:58 -07:00
George Lu	cffb6da339	Parses additional parameters Additional constraint checking Minor fixes more param parsing Add Memory Change paramVariation work on feasibility reformat bench Changed Paramgrill to use bench.c benchmarking customlevel macro Printing Flag Minor changes Explicit casting Makefile fix casting, type fix Printing Flag Minor Changes comments, helper fn's	2018-08-09 10:42:58 -07:00
Yann Collet	5808027abf	Merge branch 'dev' into fix1241	2018-08-03 16:08:33 -07:00
Nick Terrell	b9faaa1dc3	[FSE] Add division by zero test	2018-07-30 13:24:09 -07:00
cyan4973	3f535007e4	fix %zu support under minGW and relevant test on Appveyor	2018-07-30 16:56:18 +02:00
cyan4973	aade1e5904	Merge branch 'dev' into fix1241	2018-07-30 16:30:35 +02:00
cyan4973	3d4b09a5aa	support %zu under mingw	2018-07-30 16:29:20 +02:00
Yann Collet	effa84c8d1	Merge pull request #1230 from terrelln/train-out zstdcli: Allow -o before --train	2018-07-18 16:34:10 +02:00
Yann Collet	6e490a2f09	Merge pull request #1237 from terrelln/init-cstream-adv Set requestedParams in ZSTD_initCStream*()	2018-07-18 16:33:30 +02:00
cyan4973	7d1bc9cc8c	fix minor conversion warning	2018-07-18 16:10:23 +02:00
cyan4973	9597b438e9	fix #1241 Ensure that first input position is valid for a match even during first usage of context by starting reference at 1 (avoiding the problematic 0).	2018-07-17 18:52:57 +02:00
Nick Terrell	58b8219475	zstdcli: Allow -o before --train Only set the default value if `outFileName` is unset. Fixes #1227.	2018-07-16 12:45:34 -07:00
Nick Terrell	45821fac0c	Merge pull request #1225 from jennifermliu/dev Split samples when building dictionary for COVER	2018-07-13 13:26:15 -07:00
Nick Terrell	6d222c437c	Set requestedParams in ZSTD_initCStream() The correct parameters are used once, but once `ZSTD_resetCStream()` is called the default parameters (level 3) are used. Fix this by setting `requestedParams` in the `ZSTD_initCStream()` functions. The added tests both fail before this patch and pass after.	2018-07-12 18:35:55 -07:00
Dmitry V. Levin	7ca12a1455	tests: use /dev/zero instead of /dev/random on all systems except GNU/Hurd https://github.com/facebook/zstd/pull/1124 broke the test suite on Linux. Looks like GNU/Hurd is the only operating system where /dev/random is available for writing by unprivileged processes. https://github.com/facebook/zstd/pull/1222 reverted the change introduced by https://github.com/facebook/zstd/pull/1124 for OpenBSD only, other operating systems need that change to be reverted, too. Fixes: `2dde9d5aba` ("Write to /dev/random for test") Complements: `276988f794` ("OpenBSD is unable to write to /dev/zero")	2018-07-11 12:41:50 +00:00
Jennifer Liu	a085d1aae1	Allow splitPoint==1.0 (using all samples for both training and testing)	2018-07-05 10:38:45 -07:00
Yann Collet	1bf0784975	Merge pull request #1206 from GeorgeLu97/FullBenchParam Fullbench Param	2018-07-02 13:33:53 -07:00
Yann Collet	689bfecd48	Merge pull request #1188 from GeorgeLu97/BenchModule Bench module	2018-07-02 13:33:27 -07:00
Jennifer Liu	1a14f8639c	Update COVER dictionary builder tests	2018-07-02 11:37:04 -07:00
Björn Ketelaars	527dbf89ec	xz/lzma warning causes test to fail OpenBSD's port building infrastructure is able to build in a privilege separated mode. It uses a privilege drop model. Regression tests fail in this mode as xz/lzma is unable to set file group and errors out with: xz: tmp.xz: Cannot set the file group: Operation not permitted gmake[1]: *** [Makefile:307: zstd-playTests] Error 2 Actually it is not a xz/lzma error but a warning causing zstd's regression test to fail. Proposed fix is to have xz/lzma not set the exit status to 2 even if a condition worth a warning was detected (-Q flag).	2018-06-30 13:54:34 +02:00
Björn Ketelaars	276988f794	OpenBSD is unable to write to /dev/zero https://github.com/facebook/zstd/pull/1124 fixes an issue with GNU/Hurd being unable to write to /dev/zero. Implemented fix is writing to /dev/random instead. On OpenBSD a regular user is unable to write to /dev/random because of permissions set on this device. Result is failing a regression test. Proposed solution should work for all platforms.	2018-06-30 13:54:34 +02:00
Yann Collet	328ec7e34c	fix `test-zstd` can be run with parallel compilation fix #1221	2018-06-29 16:52:21 -07:00
W. Felix Handte	8e7bdc18d6	Fix Tests of `--list` Behavior with `stdin`	2018-06-29 16:31:22 -04:00
Yann Collet	a2c3a4cd0e	Merge pull request #1214 from jennifermliu/dev Make --fast=0 fail	2018-06-27 18:53:39 -07:00
Jennifer Liu	1ab57a7ce1	Redirect failed test result to INTOVOID and update comment about parsing fast command	2018-06-27 16:27:45 -07:00
W. Felix Handte	c10561c752	poolTests.c: Fix Interval Var Type	2018-06-27 19:15:38 -04:00
Jennifer Liu	aef8486fee	Make fast=0 fail	2018-06-27 14:27:27 -07:00
George Lu	ceb4b9e670	New fullbench args -l# gives some level --zstd= style parameters also supported	2018-06-25 18:43:55 -07:00
Nick Terrell	b426bcc097	[zstdmt] Fix jobsize bugs (#1205 ) [zstdmt] Fix jobsize bugs * `ZSTDMT_serialState_reset()` should use `targetSectionSize`, not `jobSize` when sizing the seqstore. Add an assert that checks that we sized the seqstore using the right job size. * `ZSTDMT_compressionJob()` should check if `rawSeqStore.seq == NULL`. * `ZSTDMT_initCStream_internal()` should not adjust `mtctx->params.jobSize` (clamping to MIN/MAX is okay).	2018-06-25 15:21:08 -07:00
Yann Collet	3b53bfe4f3	Merge pull request #1200 from felixhandte/zstd-attach-dict-pref Add CCtx Param Controlling Dict Attachment Behavior	2018-06-25 12:42:31 -07:00
George Lu	d6121ad0e1	Opaque State And minor fixups (comments/alignment/checks/fix memory leak)	2018-06-25 08:07:43 -07:00
Yann Collet	31769ce702	error on no forward progress streaming decoders, such as ZSTD_decompressStream() or ZSTD_decompress_generic(), may end up making no forward progress, (aka no byte read from input __and__ no byte written to output), due to unusual parameters conditions, such as providing an output buffer already full. In such case, the caller may be caught in an infinite loop, calling the streaming decompression function again and again, without making any progress. This version detects such situation, and generates an error instead : ZSTD_error_dstSize_tooSmall when output buffer is full, ZSTD_error_srcSize_wrong when input buffer is empty. The detection tolerates a number of attempts before triggering an error, controlled by ZSTD_NO_FORWARD_PROGRESS_MAX macro constant, which is set to 16 by default, and can be re-defined at compilation time. This behavior tolerates potentially existing implementations where such cases happen sporadically, like once or twice, which is not dangerous (only infinite loops are), without generating an error, hence without breaking these implementations.	2018-06-22 17:58:21 -07:00
Yann Collet	3934e010a2	Merge pull request #1197 from facebook/poolResize Thread Pool resize	2018-06-22 14:20:07 -07:00
Yann Collet	fbd5dfc1b1	changed POOL_resize() return type to int return is now just en error code. This guarantee that `ctx` remains valid after POOL_resize(). Gets rid of internal POOL_free() operation.	2018-06-22 12:14:59 -07:00
George Lu	a8eea99ebe	Incremental Display + Fn Separations Seperate syntheticTest and fileTableTest (now renamed as benchFiles) Add incremental display to benchMem Change to only iterMode for benchFunction Make Synthetic test's compressibility configurable from cli (using -P#)	2018-06-21 16:23:18 -07:00
Yann Collet	818e72b4d5	added extended POOL test abrupt end + downsizing with running jobs remaining in queue. also : POOL_resize() requires numThreads >= 1	2018-06-21 14:58:59 -07:00
W. Felix Handte	01bb1c1016	Add CCtx Param Controlling Dict Attachment Behavior	2018-06-21 17:29:25 -04:00
Yann Collet	7d80ada5ca	added a test for POOL (multithreading) ensuring all jobs in queue are nonetheless completed when POOL is instructed to end abruptly (POOL_free())	2018-06-21 12:24:36 -07:00
Yann Collet	6de249c1c6	fixed: bug when counting nb of active threads when queueSize > 1 also : added a test in testpool.c verifying resizing is effective.	2018-06-20 18:28:49 -07:00
Yann Collet	62469c9f41	fixed wrong size in pthread struct transfer	2018-06-19 20:14:03 -07:00
Yann Collet	c0b6ce95b1	Merge pull request #1179 from supertopher/dev Improves UX for --list command's lack of support for pipes	2018-06-19 14:36:30 -07:00
Topher Lubaway	5bac1db28f	Tests to verify piped input to `--list` exits 1 I'm following the pattern that i saw in the rest of the test file please tell me if i am using the wrong conventions	2018-06-19 09:56:37 -07:00
Yann Collet	c9e8ee93a7	removed specific --opaqueapi test from zstreamtest. This test is now integrated within --newapi, which dynamically switches between the 2 modes randomly. The main outcome is reduced testing time.	2018-06-18 19:20:37 -07:00
George Lu	e482e328cd	Reorder Arguments make initFn nullable	2018-06-18 13:21:42 -07:00
George Lu	0d1ee22990	Requested Changes Add Comment Simplify Interface (Remove resultSet) Reorder Arguments Remove customBench displayLevel Reorder bench.h Change benchFiles return type to match advanced Rename stuff	2018-06-18 12:01:12 -07:00
George Lu	8522346322	Make Fullbench use new function Rearrange Args Add nothing function Use new function, change locals to match New Display Comment cleanup Change builds	2018-06-15 11:37:49 -04:00
Yann Collet	9698d2fb72	Merge pull request #1189 from facebook/hist histogram module	2018-06-14 20:39:52 -04:00
Yann Collet	f70f829ff5	Merge pull request #1187 from facebook/fix1186 fix dctx initialization within ZSTD_decompress in stack mode	2018-06-14 16:22:22 -04:00
George Lu	20f4f32379	Add to bench -Remove global variables -Remove gv setting functions -Add advancedParams struct -Add defaultAdvancedParams(); -Change return type of bench Files -Change cli to use new interface -Changed error returns to own struct value -Change default compression benchmark to use decompress_generic -Add CustomBench function -Add Documentation for new functions	2018-06-14 14:23:24 -04:00
George Lu	cc6539f4b9	Requested changes Remove g_displaylevel/setNotificationLevel function Add extern "C" Remove averaging Reorder arguments	2018-06-14 14:18:46 -04:00
Yann Collet	7fee966f02	fix dctx initialization within ZSTD_decompress in stack mode when ZSTD_HEAPMODE=0 (which is not default). Also : added an associated test (test-fuzzer-stackmode) run on travis CI fix #1186	2018-06-14 10:22:24 -04:00
Yann Collet	2d76defbfe	grouped all histogram functions into hist.c renamed functions with HIST_* prefix	2018-06-13 19:49:31 -04:00
Yann Collet	fa41bcc2c2	grouped debug functions into debug.h There were 2 competing set of debug functions within zstd_internal.h and bitstream.h. They were mostly duplicate, and required care to avoid messing with each other. There is now a single implementation, shared by both. Significant change : The macro variable ZSTD_DEBUG does no longer exist, it has been replaced by DEBUGLEVEL, which required modifying several source files.	2018-06-13 15:43:09 -04:00
Yann Collet	c986dbf241	Merge pull request #1168 from GeorgeLu97/paramgrillfeatures Have paramgrill share bench.c benchmarking function	2018-06-13 11:38:29 -04:00
George Lu	01d940b670	Requested changes -Remove g_displaylevel/setNotificationLevel function -Add extern "C" -Remove averaging -Reorder arguments More fixes -Added BMK_return_t (result + possible error) -Correct comment' -Nullcheck ctx, dctx when allocated -Remove extra assert	2018-06-12 17:02:44 -04:00
Ryan Schmidt	b567ce9d68	Fix name of macOS	2018-06-09 14:31:17 -05:00
George Lu	0e808d608b	Make paramgrill use bench.c benchmarking	2018-06-08 12:01:05 -07:00
Yann Collet	56961e4ced	fixed minor conversion warning	2018-06-07 16:59:33 -07:00
Yann Collet	0b1833c2cb	fixed regressiontest ZSTD_TARGETLEN_MIN no longer exists since is would be tautological to check if an unsigned value is < 0.	2018-06-07 16:07:46 -07:00
Yann Collet	3050733042	Merge branch 'dev' into negLevels	2018-06-07 15:51:35 -07:00
Yann Collet	c2c47e24e0	support targetlen==0 with strategy==ZSTD_fast to mean "normal compression", targetlen >= 1 now means "disable huffman compression of literals"	2018-06-07 15:49:01 -07:00
Yann Collet	8537bfd85c	fuzzer: make negative compression level fail result of ZSTD_compress_advanced() is different from ZSTD_compress_generic() when using negative compression levels because the disabling of huffman compression is not passed in parameters.	2018-06-07 15:12:13 -07:00
Yann Collet	43a5697b24	negative compression level test compare results from simple and advanced AIP	2018-06-07 14:46:55 -07:00
Yann Collet	8ef75547ef	Merge pull request #1165 from facebook/ctxSizeDown Dynamic context downsize	2018-06-07 14:44:32 -07:00
Yann Collet	b27c7389e3	Merge pull request #1164 from GeorgeLu97/CustomMacros Partial Compilation Macros	2018-06-06 16:47:42 -07:00
George Lu	21ec9e9c30	Rename tests	2018-06-06 15:16:37 -07:00
Yann Collet	f1ea383f45	context can be sized down even with constant parameters when parameters are "equivalent", the context is re-used in continue mode, hence needed workspace size is not recalculated. This incidentally also evades the size-down check and action. This patch intercepts the "continue mode" so that the size-down check and action is actually triggered.	2018-06-06 15:04:12 -07:00
George Lu	eab626292c	More Tests	2018-06-06 11:33:39 -07:00
George Lu	8e6267503b	Move stuff around test execution -> travis logic -> partialTests.sh	2018-06-05 15:20:34 -07:00
Yann Collet	f7392f3dc9	added test case	2018-06-05 14:53:28 -07:00
George Lu	11d5bfdaa9	Revert "Partial compilation test?" This reverts commit `b2496ab606`.	2018-06-05 13:55:36 -07:00
George Lu	b2496ab606	Partial compilation test?	2018-06-05 13:24:00 -07:00
Yann Collet	3d523c741b	added workSpaceTooLarge and workSpaceWasteful also : slightly increased speed of test fuzzer.16	2018-06-05 11:42:48 -07:00
Yann Collet	bdb673666f	Merge pull request #1162 from facebook/corruptionFix Fixed a corruption bug	2018-06-05 06:50:29 -07:00
Yann Collet	54f05641e3	Merge pull request #1161 from GeorgeLu97/paramgrillformat Added zstdcli-style format for compression parameters in paramgrill	2018-06-04 14:56:23 -07:00
George Lu	3f054dceb4	forgot \n, ;	2018-06-04 13:38:37 -07:00
George Lu	ddf143ba6a	Update usage_advanced	2018-06-04 10:16:05 -07:00
Yann Collet	2108decb41	Fixed a nasty corruption bug recently introduce into the new dictionary mode. The bug could be reproduced with this command : ./zstreamtest -v --opaqueapi --no-big-tests -s4092 -t639 error was in function ZSTD_count_2segments() : the beginning of the 2nd segment corresponds to prefixStart and not the beginning of the current block (istart == src). This would result in comparing the wrong byte.	2018-06-01 18:54:34 -07:00
George Lu	5e586aa025	-O# with no file fails	2018-06-01 18:02:56 -07:00
George Lu	e355f0a580	Added Level Option	2018-06-01 14:27:53 -07:00
Yann Collet	d3615c28db	Merge pull request #1159 from GeorgeLu97/suffixlist Unknown Suffix Error	2018-06-01 14:00:10 -07:00
George Lu	ae6d1fd3fa	Add Error Print	2018-06-01 13:54:08 -07:00
George Lu	110ec9079d	Remove echo	2018-06-01 12:45:02 -07:00
George Lu	c9b4d20f02	Added new --zstd= format	2018-06-01 12:39:39 -07:00
George Lu	41249bf34b	Modified Tests Changed format as per suggestion and added second test	2018-06-01 10:54:51 -07:00
George Lu	53ea32cabe	Suffix list test	2018-06-01 10:43:06 -07:00
George Lu	cfc3451dcc	Added Test Case	2018-06-01 09:52:25 -07:00
George Lu	cec205c842	copy paste	2018-05-31 17:39:36 -07:00
Nick Terrell	fdd4d8510f	Improve compiler detection to work on Mac	2018-05-24 14:21:12 -07:00
Nick Terrell	ac852abb8b	Define BIT_DEBUG for --debug	2018-05-24 14:21:12 -07:00
Nick Terrell	2a9975f77b	Increase the maximum file size	2018-05-24 14:21:12 -07:00
Nick Terrell	e712a3a0a3	Small fixes to fuzz.py	2018-05-24 14:21:12 -07:00
Yann Collet	776128d16f	fix corner case when requiring cost of an FSE symbol ensure that, when frequency[symbol]==0, result is (tableLog + 1) bits with both upper-bit and fractional-bit estimates. Also : enable BIT_DEBUG in /tests	2018-05-24 13:59:11 -07:00
Yann Collet	d59cf02df0	decompress: changed error code when input is too large ZSTD_decompress() can decompress multiple frames sent as a single input. But the input size must be the exact sum of all compressed frames, no more. In the case of a mistake on srcSize, being larger than required, ZSTD_decompress() will try to decompress a new frame after current one, and fail. As a consequence, it will issue an error code, ERROR(prefix_unknown). While the error is technically correct (the decoder could not recognise the header of _next_ frame), it's confusing, as users will believe that the first header of the first frame is wrong, which is not the case (it's correct). It makes it more difficult to understand that the error is in the source size, which is too large. This patch changes the error code provided in such a scenario. If (at least) a first frame was successfully decoded, and then following bytes are garbage values, the decoder assumes the provided input size is wrong (too large), and issue the error code ERROR(srcSize_wrong).	2018-05-14 15:32:28 -07:00
Yann Collet	174bd3d4a7	Merge pull request #1131 from facebook/zstdcli minor: control numeric argument overflow	2018-05-14 11:53:58 -07:00
Yann Collet	2c392952f9	paramgrill: use NB_LEVELS_TRACKED in loop make it easier to generate/track more levels than ZSTD_maxClevel()	2018-05-13 17:25:53 -07:00
Yann Collet	9cd5c63771	cli: control numeric argument overflow exit on overflow backported from paramgrill added associated test case	2018-05-12 14:29:33 -07:00
Yann Collet	3f89cd1081	minor : factor out errorOut()	2018-05-12 14:09:32 -07:00
Yann Collet	50993901b2	paramgrill: subtle change in level spacing distance between levels is slightly increased to compensate for level 1 speed improvements and the will to have stronger level 19 extending the range of speed to cover.	2018-05-12 09:40:04 -07:00
Yann Collet	a3f2e84a37	added programmable constraints	2018-05-11 19:43:08 -07:00
Yann Collet	17c19fbbb5	generalized use of readU32FromChar() and check input overflow	2018-05-11 17:32:26 -07:00
Nick Terrell	2dde9d5aba	Write to /dev/random for test	2018-05-08 11:06:01 -07:00
Yann Collet	ad4524d605	fix ZSTD_compressBlock() associated with CDict reported by @let-def. It's actually a bug in ZSTD_compressBegin_usingCDict() which would pass a wrong pledgedSrcSize value (0 instead of ZSTD_CONTENTSIZE_UNKNOWN) resulting in wrong window size, resulting in downsized seqStore, resulting in segfault when writing into the seqStore later in the process. Added a test in fuzzer to cover this use case (fails before the patch).	2018-05-07 12:54:13 -07:00
Nick Terrell	263134ce66	Fix playTests.sh typo Thanks to @mestia for reporting and finding the bug! Fixes #1116.	2018-05-03 13:03:08 -07:00
Nick Terrell	ca77822ddf	Fix parameter adjustment with dictionary The new advanced API basically set `requestedParams = appliedParams` when using a dictionary. This halted all parameter adjustment, which can hurt compression ratio if, for example, the window log is small for the first call, but the rest of the files are large. This patch fixes the bug, and checks that the `requestedParams` don't change in the new advanced API when using a dictionary, and generally in the fuzzer.	2018-04-25 16:32:29 -07:00
Nick Terrell	9f76eebd17	Add ZSTD_CCtx_resetParameters() function * Fix docs for `ZSTD_CCtx_reset()`. * Add `ZSTD_CCtx_resetParameters()`. Fixes #1094.	2018-04-12 16:54:07 -07:00
Nick Terrell	3c3f59e68f	Enforce pledgeSrcSize whenever known (#1106 ) The test fails before the patch and passes after. Fixes #1095.	2018-04-12 16:02:03 -07:00
Nick Terrell	280a236e9e	Add ZSTD_CCtx(Param)?_getParameter() function Closes #1096.	2018-04-12 11:50:12 -07:00
bket	daf3ab0aad	Make sure that make check runs on OpenBSD. (#1091 ) OpenBSD uses md5 instead of md5sum, and has no device called full. With this patch, make check runs until #1088. With the assumption made in the issue make check runs succesfully.	2018-04-02 14:12:18 -07:00
Yann Collet	754a821f17	Merge pull request #1080 from facebook/getFrameHeader added ZSTD_getFrameHeader_advanced()	2018-04-01 14:27:46 -07:00
Yann Collet	e6e848bfe9	added ZSTD_getFrameHeader_advanced() makes it possible to request frame header from a magicless frame	2018-03-29 17:51:08 -06:00
Yann Collet	96b480a96d	added ovlog tests check that setting ovlog has a direct impact on compression ratio (hence is not silently ignored)	2018-03-29 13:08:09 -06:00
Yann Collet	f15e2fd55e	fixed fuzz test	2018-03-21 16:45:15 -07:00
Yann Collet	b7b80e1569	fixed paramgrill	2018-03-21 16:09:15 -07:00
Yann Collet	93b8262c02	added large negative level test case	2018-03-21 15:54:44 -07:00
Yann Collet	a99c4a3621	Merge branch 'dev' into advancedDecompress	2018-03-21 06:08:28 -07:00
Yann Collet	87b0cf05bd	Merge pull request #1057 from facebook/lrmSettings LRM parameters	2018-03-21 05:59:39 -07:00
Yann Collet	453d487807	fixed minor declaration warning	2018-03-20 18:03:56 -07:00
Yann Collet	e0cb8d19c6	fixed legacy test case	2018-03-20 17:48:22 -07:00
Yann Collet	b18cb7e0b7	zstreamtest : slightly decreased time to build dictionary size	2018-03-20 16:18:48 -07:00
Yann Collet	6cda8c932c	added test with ZSTD_decompress_generic() + ZSTD_DCtx_refPrefix() also : clarified stage condition to accept new parameters, fixed initializers correspondingly.	2018-03-20 16:16:13 -07:00
Yann Collet	2af41592ea	added a test with ZSTD_decompress_generic() + ZSTD_DCtx_loadDictionary_byReference()	2018-03-20 15:59:25 -07:00
Yann Collet	6873fec658	changed dictMore for dictContentType which seems clearer to describe what the variable/argument is about.	2018-03-20 15:13:14 -07:00
Yann Collet	31b54b6eea	updated ZSTD_initStaticDDict() prototype can also specify dictContentType.	2018-03-20 14:52:02 -07:00
Yann Collet	e5ddfaede8	add a test for unordered parameters	2018-03-19 16:02:51 -07:00
Nick Terrell	f15a17e19f	Use a single buffer in zstdmt Summary: Allocate a single input buffer large enough to house each job, as well as enough space for the IO thread to write 2 extra buffers. One goes in the `POOL` queue, and one to fill, and then block on a full `POOL` queue. Since we can't overlap with the prefix, we allocate space for 3 extra input buffers. Test Plan: * CI * With and without ASAN/UBSAN run zstdmt with different number of threads on two large binaries, and verify that their checksums match. * Test on the tip of the zstdmt ldm integration. Reviewers: cyan Differential Revision: https://phabricator.intern.facebook.com/D7284007 Tasks: T25664120	2018-03-15 16:21:33 -07:00
Nick Terrell	33fb966e56	Fix overflow protection with wlog=31 The overflow protection is broken when the window log is `> (3U << 29)`, so 31. It doesn't work when `current` isn't around `1U << windowLog` ahead of `lowLimit`, and the the assertion `current > newCurrent` fails. This happens when the same context is used many times over, but with a large window log, like in zstdmt. Fix it by triggering correction based on `nextSrc - base` instead of `lowLimit`. The added test fails before the patch, and passes after.	2018-03-14 11:45:44 -07:00
Yann Collet	6a9b41b731	create command --fast[=#] access negative compression levels from command line for both compression and benchmark modes. also : ensure proper propagation of parameters through ZSTD_compress_generic() interface. added relevant cli tests.	2018-03-11 20:01:23 -07:00
Yann Collet	a146ee04ae	added negative compression levels negative compression level trade compression ratio for more compression speed. They turn off huffman compression of literals, and use row 0 as baseline with a stepSize = -cLevel. added associated test in fuzzer also added : new advanced parameter ZSTD_p_literalCompression	2018-03-11 05:21:53 -07:00
Yann Collet	6e481504ee	fullbench includes assert.h as it is missing for Windows	2018-02-21 11:42:23 -08:00
Yann Collet	364ce19463	update fullbench measurement methodology to use less calls to time(), like bench.c. also upgraded accuracy to nanosecond.	2018-02-21 09:43:32 -08:00
Yann Collet	9b184359e2	pretify last unit test output	2018-02-13 10:09:01 -08:00
Yann Collet	75689838e4	specify new command --single-thread	2018-02-09 15:55:41 -08:00
Yann Collet	4b525af53a	zstdmt: applies new parameters on the fly when invoked from ZSTD_compress_generic()	2018-02-02 15:58:13 -08:00
Yann Collet	209df52ba2	Changed nbThreads for nbWorkers This makes it easier to explain that nbWorkers=0 --> single-threaded mode, while nbWorkers=1 --> asynchronous mode (one mode thread on top of the "main" caller thread). No need for an additional asynchronous mode flag. nbWorkers>=2 works the same as nbThreads>=2 previously.	2018-02-01 19:29:30 -08:00
Nick Terrell	48acaddff9	Test for incorrect pledgeSrcSize earlier	2018-02-01 12:04:05 -08:00
Yann Collet	727bb7f090	Merge pull request #1008 from terrelln/hlog3 Fix hashLog3 size when copying cdict tables	2018-01-31 12:49:07 -08:00
Nick Terrell	9c8c69e41a	[fuzzer] Check ZSTD_initStaticCDict() for every level	2018-01-31 11:12:54 -08:00
Nick Terrell	ab3346af07	Fix hashLog3 size when copying cdict tables	2018-01-31 11:12:17 -08:00
Yann Collet	823a28a1f4	Merge pull request #1000 from facebook/progressiveFlush Progressive flush	2018-01-30 22:49:47 -08:00
Yann Collet	9c40ae7ff1	zstdmt: there is now one mutex/cond per job	2018-01-26 17:55:08 -08:00
Yann Collet	8e128eaf05	zstdmt : refactor job members grouped by sharing properties	2018-01-26 10:20:38 -08:00
Yann Collet	4f7c896113	zstdmt : fixed complex sequencing bug zstdmt would shortcut to single-thread blocking mode in some rare cases where data is sent to be compressed but is not yet ready.	2018-01-23 18:00:51 -08:00
Yann Collet	c1cc57f270	zstdmt : fix end condition (ZSTD_e_end) When ZSTD_e_end directive is provided, the question is not only "are internal buffers completely flushed", it is also "is current frame completed". In some rare cases, it was possible for internal buffers to be completely flushed, triggering a @return == 0, but frame was not completed as it needed a last null-size block to mark the end, resulting in an unfinished frame.	2018-01-23 15:19:11 -08:00
Yann Collet	ebd955e26a	zstdmt : fixed ending frame with 0-size block	2018-01-23 13:12:40 -08:00
Yann Collet	6711396d97	zstreamtest : fixed test 32 : multi-thread compression using ZSTD_compress_generic(,,ZSTD_e_end) Since it already provides ZSTD_e_end as directive, it should not be followed by ZSTDMT_endStream().	2018-01-19 22:20:53 -08:00
Yann Collet	3ad7d4951c	zstdmt : finally vanquished an elusive and rare race condition	2018-01-19 17:35:08 -08:00
Yann Collet	940634a610	zstdmt : simplify job creation job will not be created when not enough room within job Table	2018-01-19 13:25:06 -08:00
Yann Collet	dc69623453	zstdmt: fixed corruption issue in ZSTDMT_endStream() when invoked directly.	2018-01-19 12:41:56 -08:00
Yann Collet	cb5eba8e20	add `zcat` symlink support, suggested by @wtarreau added some test also updated relevant doc + fixed a mistake in `lz4` symlink support : lz4 utility doesn't remove source files by default (like zstd, but unlike gzip). The symlink must behave the same.	2018-01-19 11:26:35 -08:00
Yann Collet	6f7280fb33	fixed frame checksum issue and race conditions	2018-01-18 16:20:26 -08:00
Yann Collet	4f43ef731d	Merge branch 'dev' into constCDict	2018-01-18 13:36:43 -08:00
Yann Collet	f3b8f90b6d	changed initStatic?Dict() return type to const ZSTD_?Dict* ZSTD_create?Dict() is required to produce a ?Dict* return type because `free()` does not accept a `const type` argument. If it wasn't for this restriction, I would have preferred to create a `const ?Dict` object to emphasize the fact that, once created, a dictionary never changes (hence can be shared concurrently until the end of its lifetime). There is no such limitation with initStatic?Dict() : as stated in the doc, there is no corresponding free() function, since `workspace` is provided, hence allocated, externally, it can only be free() externally. Which means, ZSTD_initStatic?Dict() can return a `const ZSTD_?Dict*` pointer. Tested with `make all`, to catch initStatic's users, which, incidentally, also updated zstd.h documentation.	2018-01-17 14:08:48 -08:00
Yann Collet	2e23333094	ZSTDMT can now work in non-blocking mode with 1 thread it still fallbacks to single-thread blocking invocation when input is small (<1job) or when invoking ZSTDMT_compress(), which is blocking. Also : fixed a bug in new block-granular compression routine.	2018-01-16 15:28:43 -08:00
Yann Collet	0bc7a75468	Merge pull request #987 from facebook/checkTag Check tag	2018-01-16 11:52:32 -08:00
Yann Collet	f4e58455f6	ensure MOREFLAGS are not lost in root->tests Makefile invocation	2018-01-16 11:50:16 -08:00
Yann Collet	f6e17b8c7a	added tests/checkTag compared provided tag with current libzstd version	2018-01-14 17:03:45 -08:00
Yann Collet	4792ac6689	make -C tests legacy : minor flag alteration ZSTD_LEGACY_SUPPORT is a macro constant, so it should be part of CPPFLAGS, instead of CFLAGS	2018-01-14 14:09:17 -08:00
Yann Collet	99a0a8bbd2	tests/Makefile : fixed target allnothread ensures MT is disabled	2018-01-13 22:00:05 -08:00
Yann Collet	f8a05932b9	added `make list` capability to `tests/Makefile`	2018-01-13 21:54:21 -08:00
Yann Collet	863b2f8db4	Merge pull request #983 from terrelln/dict-wlog Increase windowLog from CDict based on the srcSize when known	2018-01-12 07:47:43 -08:00
Nick Terrell	4b7c4e5f41	Add test for cdict window log adjustment	2018-01-11 16:45:16 -08:00
Yann Collet	cacf47cbee	Merge branch 'dev' into dubtlazy and fixed conflicts	2018-01-11 13:25:08 -08:00
Yann Collet	04c00f9388	Merge pull request #982 from facebook/fix304 Fix for #304 and #977 : error during dictionary creation	2018-01-11 13:20:59 -08:00
Yann Collet	1d623e60a1	Merge pull request #981 from facebook/fix976 fixed bug #976, reported by @indygreg	2018-01-11 11:40:07 -08:00
Yann Collet	e8093dde09	fixed #304 Pathological samples may result in literal section being incompressible. This case is now detected, and literal distribution is replaced by one that can be written into the dictionary.	2018-01-11 11:16:32 -08:00
Yann Collet	218e9fe0fc	added a test case for dictBuilder failure cyclic data set makes the entropy stage fails now, onto a fix for #304 ...	2018-01-11 09:42:38 -08:00
Yann Collet	2103a62b3d	fixed minor warning on prototype definition	2018-01-11 04:49:19 -08:00
Yann Collet	ff795580f2	fixed bug #976 , reported by @indygreg constants in zstd.h should not depend on MIN() macro which existence is not guaranteed. Added a test to check the specific constants. The test is a bit too specific. But I have found no way to control a more generic "are all macro already defined" condition, especially as this is a valid construction (the missing macro might be defined later, intentionnally).	2018-01-10 20:33:45 -08:00
conor42	0e88f6e97b	Fix break condition in decompression noise test The bug prevents noise being added	2018-01-11 11:42:58 +10:00
Yann Collet	76321d6bbc	Merge pull request #965 from facebook/tests_obj %.o objects files in /tests	2018-01-06 13:15:53 +01:00
Yann Collet	658d6b8588	Merge branch 'dev' into dubtlazy	2018-01-06 12:40:58 +01:00
Yann Collet	a927fae2a1	fixed ZSTD_reduceIndex() following suggestions from @terrelln. Also added some comments to present logic behind ZSTD_preserveUnsortedMark().	2018-01-06 12:31:26 +01:00
Yann Collet	3d578eef0c	minor tests/Makefile change, as suggested by @terrelln https://github.com/facebook/zstd/pull/965#discussion_r160008006	2018-01-06 07:13:09 +01:00
Nick Terrell	8adebbd0f8	Add test case for PR #972	2018-01-03 15:52:18 -08:00
Yann Collet	d228b6b0d0	btlazy2 : optimization for dictionary compression we want the dictionary table to be fully sorted, not just lazily filled. Dictionary loading is a bit more intensive, but it saves cpu cycles for match search during compression.	2017-12-29 19:14:18 +01:00
Yann Collet	02f64ef955	btlazy2: fixed interaction between unsortedMark and reduceTable	2017-12-29 19:08:51 +01:00
Yann Collet	4c7f137bd2	add test case which reliably produces btlazy2 rescale overflow bug The unsorted_mark is handled like any index, which fails after a rescale.	2017-12-29 17:40:36 +01:00
Yann Collet	64482c2c97	fixed bug in dubt the chain of unsorted candidates could grow beyond lowLimit.	2017-12-29 17:04:37 +01:00
Yann Collet	ffc335bccf	complete ignore list fuzz tests artifacts	2017-12-29 14:39:49 +01:00
Yann Collet	2126ca8a5f	%.o objects files in /tests Recipe in /tests rebuild everything from source for each target. zstd is still a "small" project, so it's not prohibitive, yet, rebuilding same files over and over represents substantial redundant work. This patch replaces .c files from /lib by their corresponding .o files. They cannot be compiled and stored directly within /lib, since /tests triggers additional debug capabilities unwelcome in release binary. So the resulting .o are stored directly within /tests. It turns out, it's difficult to find several target using exactly* the same rules. Using only the default rules (debug enabled, multi-threading disabled, no legacy) a surprisingly small amount of targets share their work. It's because, in many cases there are additional modifications requested : some targets are 32-bits, some enable multi-threading, some enable legacy support, some disable asserts, some want different kind of sanitizer, etc. I created 2 sets of object files : with and without multithreading. Several targets share their work, saving compilation time when running `make all`. Also, obviously, when modifying one source file, only this one needs rebuilding. For targets requiring some different setting, build from source *.c remain the rule. The new rules have been tested within `-j` parallel compilation, and work fine with it.	2017-12-27 17:58:27 +01:00
Yann Collet	cafedcbbe4	ZSTD_resetCCtx_internal: fixed order of arguments params1 was swapped with params2. This used to be a non-issue when testing for strict equality, but now that some tests look for "sufficient size" `<=`, order matters.	2017-12-19 21:49:04 +01:00
Yann Collet	574e75354b	fuzzer: ensure existence of CHECK_Z macro beyond OS-X systems	2017-12-19 11:24:14 +01:00
Yann Collet	d88c671663	added test case for "wrong blockSize in continue mode"	2017-12-19 10:16:09 +01:00
Yann Collet	a0e0985d38	added test on small file on top of test on small stream	2017-12-14 13:32:24 -08:00
Yann Collet	281f06e01f	saves 3-bytes on small input with streaming API zstd streaming API was adding a null-block at end of frame for small input. Reason is : on small input, a single block is enough. ZSTD_CStream would size its input buffer to expect a single block of this size, automatically triggering a flush on reaching this size. Unfortunately, that last byte was generally received before the "end" directive (at least in `fileio`). The later "end" directive would force the creation of a 3-bytes last block to indicate end of frame. The solution is to not flush automatically, which is btw the expected behavior. It happens in this case because blocksize is defined with exactly the same size as input. Just adding one-byte is enough to stop triggering the automatic flush. I initially looked at another solution, solving the problem directly in the compression context. But it felt awkward. Now, the underlying compression API `ZSTD_compressContinue()` would take the decision the close a frame on reaching its expected end (`pledgedSrcSize`). This feels awkward, a responsability over-reach, beyond the definition of this API. ZSTD_compressContinue() is clearly documented as a guaranteed flush, with ZSTD_compressEnd() generating a guaranteed end. I faced similar issue when trying to port a similar mechanism at the higher streaming layer. Having ZSTD_CStream end a frame automatically on reaching `pledgedSrcSize` can surprise the caller, since it did not explicitly requested an end of frame. The only sensible action remaining after that is to end the frame with no additional input. This adds additional logic in the ZSTD_CStream state to check this condition. Plus some potential confusion on the meaning of ZSTD_endStream() with no additional input (ending confirmation ? new 0-size frame ?) In the end, just enlarging input buffer by 1 byte feels the least intrusive change. It's also a contract remaining inside the streaming layer, so the logic is contained in this part of the code. The patch also introduces a new test checking that size of small frame is as expected, without additional 3-bytes null block.	2017-12-14 11:47:02 -08:00
Yann Collet	5b2ce2c043	Merge pull request #946 from terrelln/r-o Allow -o with multiple files	2017-12-14 10:02:05 -08:00
Yann Collet	c005df136f	Merge pull request #947 from facebook/fix944 Fix #944	2017-12-14 10:01:52 -08:00
Nick Terrell	4680e85bdf	Allow -o with multiple files	2017-12-13 17:44:34 -08:00
Yann Collet	4d0dfafa7b	Merge pull request #949 from terrelln/rrm [fileio] Refuse to remove non-regular file	2017-12-13 17:36:39 -08:00
Yann Collet	d23eb9a098	zstreamtest : added missing CHECK_Z()	2017-12-13 15:35:49 -08:00
Nick Terrell	90d38f6a53	Merge pull request #945 from terrelln/dev Fix cdict compressor repcodes	2017-12-13 14:24:21 -08:00
Nick Terrell	82bc8fe0cc	[fileio] Refuse to remove non-regular file	2017-12-13 13:38:26 -08:00
Yann Collet	311878dec3	Improved tests - building cli from /tests preserves potential flags in MOREFLAGS (such as asan/usan) - MT dictionary tests check for MT capability (MT is not enabled by default for zstd32)	2017-12-13 11:48:30 -08:00
Nick Terrell	22727a7467	Fix cdict compressor repcodes	2017-12-13 11:31:20 -08:00
Yann Collet	dba8016d2d	Merge branch 'dev' into fix944	2017-12-13 11:20:09 -08:00
Yann Collet	04a1557e28	fuzzer tests : added -v command so that unit tests get listed in CI console and we can see which one failed.	2017-12-13 10:08:52 -08:00
Yann Collet	98ccc115ed	added cli test veryfying bug #944 this test fails on v1.3.2	2017-12-12 19:47:38 -08:00
Yann Collet	e9dc0ab884	zstd binary, when compiled from /tests, enables debug mode adds ZSTD_DEBUG=1 and -g	2017-12-12 18:20:04 -08:00
Yann Collet	e28305fcca	fix #944 : ZSTDMT with large files and dictionary now works correctly windowLog is now enforced from provided compression parameters, instead of being copied blindly from `cdict` where it could be smaller. also : - fix a minor bug in zstreamtest --mt : advanced parameters must be set before init - changed advanced parameter name to ZSTDMT_jobSize	2017-12-12 18:04:58 -08:00
Yann Collet	03832b7aa5	re-added test case messing with revert ... :(	2017-12-12 14:01:54 -08:00
Yann Collet	8a104fda05	Revert "Created a test case which reliably reproduces bug #944 " This reverts commit `5098d1fbe2`.	2017-12-12 12:51:49 -08:00
Yann Collet	5098d1fbe2	Created a test case which reliably reproduces bug #944 in zstreamtest.	2017-12-12 12:48:31 -08:00
Yann Collet	21f7672d43	zstreamtest : add test which detects when ZSTD_initCStream() inflate data as in #942, when ZSTD_initCStream() believes data is empty, instead of unknown.	2017-12-07 03:06:01 -05:00
Yann Collet	7e05ef851a	Merge branch 'dev' into qemu32panic	2017-12-03 11:14:36 -08:00
Yann Collet	a1b24e6262	Merge pull request #938 from terrelln/time Use util.h for timing	2017-12-01 16:40:38 -08:00
Yann Collet	6fc63f521f	Merge branch 'dev' into qemu32panic	2017-12-01 10:51:01 -08:00
Yann Collet	453fb5e830	removed -ftrapv from tests/ debug flags -ftrapv is apparently buggy for gcc. versions >= 5 are supposed to work better, but even then, some complaints say it's still flaky when optimizations are enabled. I even saw a post saying it only works if one creates its own signal handler, which would make this flag no longer transparent. on clang, it seems to work correctly. But we would need to add a method to selectively add flags depending on compiler. That's too much troubles for the intended benefit (just catch integer overflows, which we can also do using ubsan).	2017-12-01 10:30:53 -08:00
Nick Terrell	dab8cfa3c7	Combine definitions of SEC_TO_MICRO	2017-11-30 19:40:53 -08:00
Nick Terrell	9a2f6f477b	Use util.h for timing	2017-11-30 14:57:25 -08:00
Yann Collet	d3c59edac9	removed long-range-mode tests from `zstreamtest --no-big-tests`	2017-11-29 16:42:20 -08:00
Yann Collet	b3d76e0a94	Merge branch 'dev' of github.com:facebook/zstd into dev	2017-11-28 15:03:38 -08:00
Yann Collet	f5747afe0f	add integer overflow trapping to debug builds doc: https://gcc.gnu.org/onlinedocs/gcc-4.0.2/gcc/Code-Gen-Options.html	2017-11-28 15:02:48 -08:00
W. Felix Handte	62c746dcf9	Add Test on LZ4 Format Input Buffer Overrun	2017-11-28 12:06:48 -05:00
Yann Collet	bd88f633ac	zstreamtest : in `-T#s`, s considered a suffix meaning "seconds" avoid unintentionnally triggering `seedset`, so that seed gets automatically determined when not set.	2017-11-27 12:15:23 -08:00
Yann Collet	42c1e64270	slightly improved ratio at -22 merging of repcode search into btsearch introduced a small compression ratio regressio at max level : 1.3.2 : 52728769 after repMerge patch : 52760789 (+32020) A few minor changes have produced this difference. They can be hard to spot. This patch buys back about half of the difference, by no longer inserting position at hc3 when a long match is found there. It feels strangely counter-intuitive, but works : after this patch : 52742555 (-18234)	2017-11-19 14:00:55 -08:00
Yann Collet	99435dbbab	minor : search early-out on sufficient_len for hc3 and rep very very small speed and ratio increases	2017-11-19 12:58:04 -08:00
Yann Collet	23767e950a	fix one UB pointer arithmetic in encoder Instead of calculating distance between 2 memory objects, which is UB, we extract the offset from object 1, and transfer it into object 2.	2017-11-17 13:24:51 -08:00
Yann Collet	15768cabb5	fixed some complex scenarios Fixed : multithreading to compress some small data with dictionary Fixed : ZSTD_initCStream_usingCDict() Improved streaming memory usage when pledgedSrcSize is known.	2017-11-16 15:18:18 -08:00
Yann Collet	aec56a52fb	Merge pull request #908 from facebook/ubsan Modified one pointer arithmetic expression to a more conformant way.	2017-11-07 11:45:34 -08:00
Yann Collet	3b59a01039	fixed zstreamtest declaration after statement	2017-11-01 17:36:31 -07:00
Yann Collet	150354c5fe	minor refactor added some traces and assert related to hunting a potential ubsan error in 32-bits more (it ends up being a compiler-side issue : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82802). Modified one pointer arithmetic expression for a more conformant way.	2017-11-01 16:57:48 -07:00
Nick Terrell	e19b0822bc	Test large skippable frames	2017-11-01 13:10:03 -07:00
Nick Terrell	86b8134cad	[libzstd] Fix parameter selection for empty input ZSTD_compress() and friends would treat an empty input as an unknown size when selecting parameters. Thus, they would drastically overallocate the context. Tell ZSTD_getParams() that the source size is 1 when it is empty.	2017-10-25 17:24:15 -07:00
Yann Collet	97dccbbb2b	fixed zbufftest preserve "pledgedSrcSize=0" means "unknown" in init_advanced()	2017-10-19 14:06:02 -07:00
Yann Collet	9d377c6e88	Merge pull request #895 from facebook/fileSize_unknown Distinguish 0-size from size-unavailable	2017-10-18 16:50:17 -07:00
Yann Collet	1ff8a8c109	Merge pull request #891 from facebook/contentSize Content size	2017-10-17 17:24:51 -07:00
Yann Collet	5891f1d952	added a check to ensure 0-size is present in frame header	2017-10-17 16:23:20 -07:00
Yann Collet	18b795374a	UTIL_getFileSize() returns UTIL_FILESIZE_UNKNOWN on failure UTIL_getFileSize() used to return zero on failure. This made it impossible to distinguish a failure from a genuine empty file. Both cases where coalesced. Adding UTIL_FILESIZE_UNKNOWN constant has many consequences on user code, since in many places, the `0` was assumed to mean "error". This is no longer the case, and the error code must be actively checked.	2017-10-17 16:14:25 -07:00
Nick Terrell	e600b5d0da	[test] Exercise all codes in dictionary tables	2017-10-16 18:05:36 -07:00
Yann Collet	241c57adff	playtest : minor cosmetic changes	2017-10-16 14:01:42 -07:00
Yann Collet	43914f6a20	add test for presence of decompressed size field in header previous test was "display" only, it wouldn't trigger an error in case of missing header.	2017-10-13 23:47:01 -07:00
Yann Collet	beb9b4b398	fixed ZSTDMT_initCStream() when contentSizeFlag==1 by default and a wrong test in zstreamtest --mt	2017-10-13 19:09:30 -07:00
Yann Collet	213ef3b510	fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag, and a stream fuzzer test, which was incorrect (relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)	2017-10-13 19:01:58 -07:00
Yann Collet	3c1e3f8ec9	contentSizeFlag enabled by default would also fail for streaming and MT operations fixed	2017-10-13 18:32:06 -07:00
Yann Collet	fb44516641	ensure fParams.contentSizeFlag starts at 1 such default was failing for ZSTD_compressBegin/ZSTD_compressContinue fixed too	2017-10-13 17:39:13 -07:00
Yann Collet	7f6a783862	fixed a small error in decodeCorpus a compressed block must be strictly smaller than its decompressed size.	2017-10-07 15:19:52 -07:00
Yann Collet	4252621e26	playtests: do not use cat on large files some target have limitation making cat incompatible with large files (namely debian hurd-i386)	2017-10-05 20:21:59 -07:00
Yann Collet	e580dc6a4a	Merge pull request #860 from felixhandte/zstd-lz4-support-tests Add Default LZ4 Support When Available	2017-09-29 22:32:54 -07:00
Yann Collet	e963800e27	zstdmt : fixed : buffer dst0 wasn't properly set to null after usage now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources() wether previous compression was completed correctly or not.	2017-09-28 23:01:31 -07:00
Yann Collet	e0065cf660	make test : removed zstreamtest unit tests for variants slightly reduced time to create dictionary at beginning of unit tests	2017-09-28 18:34:38 -07:00
Yann Collet	47c6a95d07	zstreamtest : run unit tests only during "normal" session not during --mt, --newapi and --opaque this avoids running them 4x during `make test`	2017-09-28 18:27:22 -07:00
W. Felix Handte	d0519d4b0c	Add CLI Program Name Detection for LZ4	2017-09-28 19:18:15 -04:00
W. Felix Handte	5705d9f25a	Add basic tests for the lz4 integration	2017-09-28 19:16:43 -04:00
Yann Collet	e4ec427720	Merge branch 'dev' into shorterTests fixed conflicts	2017-09-28 12:19:28 -07:00
Yann Collet	bbef058ae6	zstreamtest --newapi : reduced maximum allocated memory	2017-09-28 11:48:45 -07:00
Yann Collet	d6770f80af	minor : rewrite unit tests using CHECK_Z macro	2017-09-28 02:14:48 -07:00
Yann Collet	9fe50ed623	fixed maximum windowLog for zstreamtest --newapi for compatibility with low memory VM	2017-09-28 01:42:06 -07:00
Yann Collet	9b5b47ac93	ensure adjustCParams adjust hLog and cLog even without srcSize It would previously exit when srcSize is unknown. But in the case of custom parameters, hLog and cLog can still be too large in comparison with windowLog. Reduces maximum memory allocated during zstreamtest --newapi	2017-09-28 01:25:40 -07:00
Yann Collet	824f75ea7c	Merge pull request #863 from facebook/newFormats magicless frames (#591)	2017-09-28 00:32:16 -07:00
Nick Terrell	d9c1e9125f	[fuzz] Small changes for oss-fuzz integration	2017-09-27 18:23:06 -07:00
Yann Collet	aa800c4793	reduced memory usage of zstreamtest --newapi to run on memory-constrained VM	2017-09-27 18:00:15 -07:00
Yann Collet	bc32b40b98	reduced zstreamtest --mt memory load adjust compression level, hence memory usage, depending on nb threads in order to run correctly on memory-starved VM.	2017-09-27 17:27:38 -07:00
Yann Collet	54a827fff0	Merge branch 'dev' into newFormats Fixed conflicts in zstdmt_compress.c	2017-09-27 16:39:40 -07:00
Yann Collet	02502191e5	separated --long tests between short and long tests A fast subset of these tests is now played in short test mode	2017-09-27 15:48:06 -07:00
Yann Collet	f9de54acfb	reduced memory requirements for --long tests in new --long test section	2017-09-27 15:38:27 -07:00
Yann Collet	bdd0f6f046	improved make clean in tests/fuzz	2017-09-27 15:20:08 -07:00
Yann Collet	60059df051	shorter make test to avoid time out on travis CI Timed tests (fuzzer) are reduced Long tests are shortened (less data generated)	2017-09-27 15:16:27 -07:00
Yann Collet	ea1f50bf73	removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal() It does not feel "right" from a dependency perspective. ZSTD_initDCtx_internal() is triggered once, on DCtx creation, while ZSTD_decompressBegin() is invoked at the beginning of each new frame, and is also a user-facing prototype. Downside : a DCtx must be init before first usage ! This was always the intention by the way, and is documented as such. This stage is automatically done within ZSTD_decompress() and variants, and also within ZSTD_decompressStream(). Only ZSTD_decompressContinue() is impacted, it must be preceded by a ZSTD_decompressBegin(), as detailed in doc. A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.	2017-09-27 13:51:05 -07:00
Yann Collet	bfabd1d4dc	fixed zstreamtest decoding error same error (wrong output buffer size) was present on --mt and --new_api tests.	2017-09-27 01:01:11 -07:00
Yann Collet	ca306c1c84	fixed a bug in zstreamtest decoder output buffer would receive a wrong size. In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size. In this version, this condition is actively checked, and the function returns an error code if this condition is not respected. This check could also be done with an assert(), but since this is a user-facing interface, it seems better to keep this check at runtime.	2017-09-27 00:39:41 -07:00
Yann Collet	4791561c4a	silence minor gcc warning -Wempty-body also silence fuzz test artefacts	2017-09-26 17:57:38 -07:00
Yann Collet	91ffe71ec8	Merge pull request #864 from terrelln/long Increase maximum window size	2017-09-26 16:26:29 -07:00
Yann Collet	df4e9bba25	fixed constant errors for gcc in c99 mode C standard does not consider a `static const int` as a constant. This is a problem for initializer, and ZSTD_STATIC_ASSERT(). Replaced by macro values	2017-09-26 14:31:06 -07:00
Yann Collet	9f0b8dfbe9	Merge branch 'dev' into newFormats	2017-09-26 14:22:39 -07:00
Nick Terrell	471aa385b3	[fuzz] Speed up round trip tests * Enforce smaller maximum values for parameters * Adjust parameters to the source size The memory usage is reduced by about 5x, which makes the fuzzers run at least twice as fast, even more so with ASAN/MSAN enabled.	2017-09-26 14:03:43 -07:00
Nick Terrell	c233bdbaee	Increase maximum window size * Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail on my Mac. * Maximum window size in 64-bit mode is 2GB, since that is the largest power of 2 that works with the overflow prevention. * Allow `--long=windowLog` to set the window log, along with `--zstd=wlog=#`. These options also set the window size during decompression, but don't override `--memory=#` if it is set. * Present a helpful error message when the window size is too large during decompression. * The long range matcher defaults to a hash log 7 less than the window log, which keeps it at 20 for window log 27. * Keep the default long range matcher window size and the default maximum window size at 27 for the API and CLI. * Add tests that use the maximum window size and hash size for compression and decompression.	2017-09-26 14:00:01 -07:00
Yann Collet	8a4d0abc1c	Merge pull request #861 from terrelln/zip [fuzz][CI] Set up regression tests	2017-09-25 17:11:35 -07:00
Yann Collet	52a1d1c6dc	added ZSTD_DCtx_reset()	2017-09-25 16:56:48 -07:00
Yann Collet	5d8fdd1641	Merge pull request #855 from terrelln/maxoff [libzstd] Increase MaxOff	2017-09-25 16:34:29 -07:00
Nick Terrell	917a213254	[fuzz] Determine flags based on compiler version	2017-09-25 15:32:36 -07:00
Nick Terrell	11e21f23cb	[fuzz] Mention the corpora in the README	2017-09-25 15:31:38 -07:00
Nick Terrell	6bb781e0f1	[fuzz] Add regressiontest targets	2017-09-25 15:31:33 -07:00
Yann Collet	62568c9a42	added capability to generate magic-less frames decoder not implemented yet	2017-09-25 14:26:26 -07:00
Nick Terrell	bbe77212ef	[libzstd] Increase MaxOff	2017-09-25 13:36:18 -07:00
Nick Terrell	bfad5568b5	[fuzz] Make simple_round_trip compile cleanly	2017-09-25 13:28:45 -07:00
Nick Terrell	23199b6daf	[fuzz] Fix fuzz.py env flags parsing	2017-09-25 13:28:18 -07:00
Nick Terrell	1c23b64049	[fuzz] fuzz.py can minimize and zip corpora * "minimize" minimizes the corpora into an output directory. * "zip" zips up the minimized corpora, which are ready to deploy.	2017-09-25 12:04:12 -07:00
Yann Collet	96f0cde31a	minor function rename ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams _usingX is clear. _advanced feels redundant	2017-09-24 16:47:02 -07:00
Yann Collet	cd3115b284	added control from frame content size at end of decompression adding check at end of single-pass ZSTD_decompressFrame(). Check within ZSTD_decompressContinue() was already added in a previous patch : `b3f33ccfb3`	2017-09-21 16:21:10 -07:00
Nick Terrell	1fe762e236	[zstdcli] Fix LDM advanced options parsing	2017-09-18 14:49:35 -07:00
Yann Collet	31829cb057	Merge pull request #847 from terrelln/fuzzer [fuzzer] Fuzz long range matching & new API	2017-09-15 12:09:00 -07:00
Nick Terrell	39357c41cb	[fuzzer] Fuzz long range matching & new API	2017-09-14 14:48:08 -07:00
Yann Collet	218c09e5b3	Merge pull request #844 from terrelln/fuzzer Fuzzer	2017-09-14 11:40:25 -07:00
Nick Terrell	9712d5ebe6	[fuzzer] Fix bugs in fuzz.py	2017-09-13 19:08:35 -07:00
Nick Terrell	a6f08b4783	[fuzzer] Fix FUZZ_seed()	2017-09-13 18:41:32 -07:00
Nick Terrell	6c6412cef9	[fuzzer] Update README.md	2017-09-13 18:23:52 -07:00
Nick Terrell	6b8236cf7e	[fuzz] Add fuzzing helper script	2017-09-13 17:45:21 -07:00
Nick Terrell	b7e1522330	Add block fuzzers	2017-09-13 17:44:41 -07:00
Nick Terrell	def3214d74	[fuzzer] Handle single empty directory	2017-09-13 17:44:30 -07:00
Yann Collet	739b620814	Merge pull request #842 from stellamplau/decodeCorpus-maxSize Add flag to limit max decompressed size in decodeCorpus	2017-09-13 17:26:55 -07:00
Nick Terrell	8b6c80ada8	Update fuzzer Makefile	2017-09-13 16:16:57 -07:00
Nick Terrell	677c2cbf89	Update fuzzer sources	2017-09-13 16:16:57 -07:00
Stella Lau	963558a072	Fix implicit conversion error	2017-09-13 16:01:16 -07:00
Stella Lau	40bf0ced7d	Add flag to limit max decompressed size in decodeCorpus	2017-09-13 15:16:56 -07:00
Yann Collet	f1571dad8f	Merge pull request #838 from stellamplau/ldm-mergeDev Add long distance matcher	2017-09-13 13:24:08 -07:00
Yann Collet	be1f2dac5b	Merge pull request #841 from facebook/utilTimeAPI modified util::time API (T19505791)	2017-09-13 11:41:01 -07:00
Yann Collet	a1bc08834f	Merge pull request #840 from stellamplau/decodeCorpus-blocks Make decodecorpus generate raw compressed blocks	2017-09-13 09:34:04 -07:00
Yann Collet	c95c0c9725	modified util::time API for easier invocation. - no longer expose frequency timer : it's either useless, or stored internally in a static variable (init is only necessary once). - UTIL_getTime() provides result by function return.	2017-09-12 18:12:46 -07:00
Stella Lau	e89065506e	Make decodecorpus generate raw compressed blocks	2017-09-12 17:18:45 -07:00
Stella Lau	3d8e313f64	Reduce ldm hash table size in test	2017-09-11 17:21:28 -07:00
Stella Lau	eb3327c10a	Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev	2017-09-11 15:00:01 -07:00
Yann Collet	b3f33ccfb3	use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream() Use same definition as public one minor : reduce allocated buffer size in some cases (when frameContentSize is known and == windowSize)	2017-09-09 14:37:28 -07:00
Yann Collet	058ed2ad33	ZSTD_decodingBufferSize_min() supporting function for bufferless streaming API (ZSTD_decompressContinue()) makes it possible to correctly size a round buffer for decoding using this API. also : added field blockSizeMax within ZSTD_frameHeader, as it's a necessary information to know when to restart at beginning of decoding buffer.	2017-09-09 01:03:29 -07:00
Yann Collet	3128e03be6	updated license header to clarify dual-license meaning as "or"	2017-09-08 00:09:23 -07:00
Stella Lau	eeff55dfa8	Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev	2017-09-06 15:56:32 -07:00
Stella Lau	9e4060200b	Add tests and fix pointer alignment	2017-09-06 09:14:05 -07:00
Stella Lau	af4068a697	Fix function name in tests/fuzz/regression_driver	2017-09-05 22:14:41 -07:00
Stella Lau	67d4a6161c	Add ldmBucketSizeLog param	2017-09-02 21:55:29 -07:00
Stella Lau	a1f04d518d	Move hashEveryLog to cctxParams and update cli	2017-09-01 15:05:47 -07:00
Stella Lau	767a0b3be1	Move ldm hashLog, bucketLog, and mml to cctxParams	2017-09-01 12:24:59 -07:00
Stella Lau	17d8e0bdcc	Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate	2017-09-01 10:19:38 -07:00
Stella Lau	8081becadc	Add long distance matching as a CCtxParam	2017-09-01 09:18:58 -07:00
Eiichi Tsukata	7492e7f1c7	tests/fuzz: change ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX ZSTD_BLOCKSIZE_ABSOLUTEMAX is changed at the commit: `fa3671eac7`	2017-09-01 16:37:39 +09:00
Eiichi Tsukata	6639395979	tests/fuzz: fix make all target names	2017-09-01 16:32:40 +09:00
Yann Collet	d7ad99b2ab	Merge branch 'longRangeMatcher' into dev	2017-08-31 18:08:37 -07:00
Yann Collet	e0cecd8736	fixed poolTests needs more dependencies from zstd for custom allocators and error codes	2017-08-31 15:13:31 -07:00
Stella Lau	6a546efb8c	Add long distance matcher Move last literals section to ZSTD_block_internal	2017-08-31 12:53:19 -07:00
Yann Collet	b0cb081dc8	last batch of header files changed to reflect new license (#825 ) only remains to update contrib/linux-kernel (@terrelln)	2017-08-31 12:20:50 -07:00
Yann Collet	e21384fffb	fixed more file headers after license change (#825 )	2017-08-31 12:11:57 -07:00
Yann Collet	e9dc204f42	fixed a bunch of headers after license change (#825 )	2017-08-31 11:24:54 -07:00
Stella Lau	90a31bfa16	Pass dictMode to ZSTDMT_initCStream; fix nits - Return error code in estimate{CCtx,CStream}Size functions	2017-08-30 16:19:07 -07:00
Stella Lau	ee65701720	Minor fixes; remove formatting only changes	2017-08-29 20:27:35 -07:00
Stella Lau	a6e20e1bd7	Add test for raw content starting with dict header	2017-08-29 18:36:18 -07:00
Stella Lau	82d636b76a	Rename applyCCtxParams()	2017-08-29 18:03:06 -07:00
Stella Lau	c88fb9267f	Replace 'byReference' with enum	2017-08-29 11:55:02 -07:00
Stella Lau	b5b9275e67	Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced()	2017-08-29 10:49:29 -07:00
Stella Lau	18224608ff	Remove ZSTD_setCCtxParameter()	2017-08-25 13:58:41 -07:00
Stella Lau	9911153723	Move jobSize and overlapLog in zstdmt to cctxParams	2017-08-25 13:14:51 -07:00
Stella Lau	eb7bbab36a	Remove ZSTD_p_refDictContent and dictContentByRef	2017-08-25 11:11:45 -07:00
Stella Lau	15fdeb9e41	Enforce nbThreads<=1 for estimateCCtxSize	2017-08-24 16:28:49 -07:00
Stella Lau	1c81f725ff	Remove duplicated testing code	2017-08-23 15:47:15 -07:00
Stella Lau	6f1a21c7e9	Remove formatting-only changes	2017-08-23 10:24:19 -07:00
Stella Lau	8fd1636776	Remove unused functions	2017-08-22 13:33:58 -07:00
Stella Lau	73c73bf16a	Reduce code duplication in zstreamtest	2017-08-21 12:41:19 -07:00
Nick Terrell	3587556873	[cover] Test small maxdict	2017-08-21 11:16:47 -07:00
Stella Lau	91b30dbe84	Remove test parameter	2017-08-21 10:09:06 -07:00
Stella Lau	f181f33bdf	Disable tests and refactor	2017-08-21 01:59:08 -07:00
Stella Lau	023b24e6d4	Add cctx param tests	2017-08-20 22:55:07 -07:00
Yann Collet	d6394cc4c3	fixed test-zstd-nolegacy	2017-08-20 10:15:44 -07:00
Yann Collet	32fb407c9d	updated a bunch of headers for the new license	2017-08-18 16:52:05 -07:00
Yann Collet	8049556928	Merge pull request #778 from terrelln/bad-huff [libzstd] Fix bug in Huffman decompresser	2017-08-07 14:05:58 -07:00
Nick Terrell	abe12b3399	[libzstd] Fix bug in Huffman decompresser The zstd format specification doesn't enforce that Huffman compressed literals (including the table) have to be smaller than the uncompressed literals. The compressor will never Huffman compress literals if the compressed size is larger than the uncompressed size. The decompresser doesn't accept Huffman compressed literals with 4 streams whose compressed size is at least as large as the uncompressed size. * Make the decompresser accept Huffman compressed literals whose size increases. * Add a test case that exposes the bug. The compressed file has to be statically generated, since the compressor won't normally produce files that expose the bug.	2017-08-07 12:37:48 -07:00
Stella Lau	e1abc2a367	Switch the sleep function to UTIL_sleepMilli	2017-08-07 11:49:13 -07:00
Stella Lau	1e366f9dea	Add test for deadlock	2017-08-02 11:27:50 -07:00
Stella Lau	5adceeed01	Allow queueSize=0 in pool.c and update poolTests	2017-07-31 10:10:16 -07:00
Yann Collet	38ba7002f2	fixed minor warning on unused variable in shell function	2017-07-20 18:39:04 -07:00
Yann Collet	5e6c5203f3	fixed fuzzer test for non OS-X platforms	2017-07-20 15:11:56 -07:00
Yann Collet	1ca1288689	added --memtest=# command to fuzzer to jump directly to relevant test section	2017-07-19 16:01:16 -07:00
Yann Collet	44b0838253	Merge pull request #770 from terrelln/test-mode [zstdcli] Fix -t in streaming mode	2017-07-18 15:40:59 -07:00
Nick Terrell	d0b27483ae	[zstdcli] Fix -t in streaming mode	2017-07-18 14:45:49 -07:00
Nick Terrell	cc1522351f	[libzstd] Fix bug in Huffman encoding Summary: Huffman encoding with a bad dictionary can encode worse than the HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible input, and even if we did, the dictionaries Huffman table could be ill suited to compressing actual data. The fast optimization doesn't seem to improve compression speed, even when I hard coded fast = 1, the speed didn't improve over hard coding it to 0. Benchmarks: $ ./zstd.dev -b1e5 Benchmarking levels from 1 to 5 1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s 2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s 3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s 4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s 5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s $ ./zstd -b1e5 Benchmarking levels from 1 to 5 1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s 2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s 3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s 4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s 5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s $ ./zstd.dev -b1e5 ../silesia.tar Benchmarking levels from 1 to 5 1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s 2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s 3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s 4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s 5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s $ ./zstd -b1e5 ../silesia.tar Benchmarking levels from 1 to 5 1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s 2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s 3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s 4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s 5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s Test Plan: I added a test case to the fuzzer which crashed with ASAN before the patch and succeeded after.	2017-07-18 13:20:40 -07:00
Yann Collet	fa3aa04ccd	Merge pull request #761 from paulcruz74/file-rename renamed pool.c to poolTests.c	2017-07-14 09:09:45 -07:00
Yann Collet	3a60efd3a9	policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760 ) Previously, ZSTDMT would refuse to create the compressor. Also : increased ZSTDMT_NBTHREADS_MAX to 256, updated doc, and added relevant test	2017-07-13 10:17:23 -07:00
Yann Collet	052a95f77c	fix : ZSTDMT_compress_advanced() correctly generates checksum when params.fParams.checksumFlag==1. This use case used to be impossible when only ZSTD_compress() was available	2017-07-11 17:18:26 -07:00
Yann Collet	ef0ff7fe7f	zstdmt: removed margin for improved memory usage	2017-07-11 08:54:29 -07:00
Yann Collet	4616fad18b	improved ZSTDMT_compress() memory usage does not need the input buffer for streaming operations also : reduced a few tests time length	2017-07-10 17:16:41 -07:00
Yann Collet	670b1fc547	optimized memory usage for ZSTDMT_compress() Previously, each job would reserve a CCtx right before being posted. The CCtx would be "part of the job description", and only released when the job is completed (aka flushed). For ZSTDMT_compress(), which creates all jobs first and only join at the end, that meant one CCtx per job. The nb of jobs used to be == nb of threads, but since latest modification, which reduces the size of jobs in order to spread the load of difficult areas, it also increases the nb of jobs for large sources / small compression level. This resulted in many more CCtx being created. In this new version, CCtx are reserved within the worker thread. It guaranteea there cannot be more CCtx reserved than workers (<= nb threads). To do that, it required to make the CCtx Pool multi-threading-safe : it can now be called from multiple threads in parallel.	2017-07-10 16:30:55 -07:00
Yann Collet	3510efb02d	fix : custom allocator correctly propagated to child contexts	2017-07-10 14:21:40 -07:00
Yann Collet	ee3423d709	extended fuzzer MT memory tests	2017-07-10 14:09:16 -07:00
Yann Collet	88da8f1816	fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter() also : compile fuzzer with MT enabled	2017-07-10 14:02:33 -07:00
Yann Collet	f9524cf366	added --memtest to fuzzer	2017-07-10 13:48:41 -07:00
Yann Collet	e32fb0c1fe	added ZSTD_sizeof_CCtx() test	2017-07-10 12:29:57 -07:00
Paul Cruz	89190ef07d	renamed pool.c to poolTests.c	2017-07-10 11:32:30 -07:00
Yann Collet	ed0243a63c	removed zbufftest from list of `all` tests	2017-07-07 16:16:14 -07:00
Yann Collet	990449b89d	new field : ZSTD_frameHeader.frameType Makes frame type (zstd,skippable) detection more straighforward. ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"	2017-07-07 15:21:35 -07:00
Yann Collet	7758ed8458	fixed fullbench, part 2	2017-07-06 02:48:00 -07:00
Yann Collet	9b2c1acfc0	fixed fullbench	2017-07-06 02:22:57 -07:00
Yann Collet	27e883371d	fixed wrong assert() condition A single job created by ZSTDMT_compress() can be < 256KB if data to compress is < 256 KB (in which case it is delegated to single thread mode)	2017-07-04 19:33:16 -07:00
Yann Collet	2cb9774f5e	more precise estimation of amount to flush at end of stream (single thread mode) also : can use DEBUGLEVEL variable in /tests	2017-07-04 12:39:26 -07:00
Yann Collet	5051dd39ca	Merge pull request #743 from facebook/fullbench compress_generic() automatic optimization opportunities	2017-07-03 21:26:38 -07:00
Yann Collet	95c4a6e2c8	Merge pull request #745 from terrelln/libfuzzer [fuzz] Add libFuzzer targets	2017-07-03 15:15:20 -07:00
Nick Terrell	bea0f0cfa0	[fuzz] Move from fuzz/ to tests/fuzz/	2017-07-03 12:40:12 -07:00
cyan4973	4b26306cb8	blindfix : fullbench's one-time leak, detected by valgrind	2017-07-01 08:03:59 -07:00
cyan4973	c07e43c2b5	added --show-leak-kind=all to valgrind tests	2017-07-01 07:05:11 -07:00
cyan4973	b5bb7c6d95	fixed Visual compilation of fullbench-dll	2017-06-29 19:59:37 -07:00
Yann Collet	e7e5a8cef7	made fullbench compatible with multi-threading fullbench 61/62 measure speed of ZSTD_compress_generic with 2 threads	2017-06-29 18:56:24 -07:00
Yann Collet	afb0aca739	zstreamtest : big tests are only enabled in 64-bits mode to avoid requesting too much memory in 32-bits mode during MT tests	2017-06-29 18:19:09 -07:00
Yann Collet	2e84bec9ac	updated fullbench to also measure ZSTD_compress_generic() will make it possible to visualize optimization opportunity for ZSTD_e_end	2017-06-29 13:03:10 -07:00
Yann Collet	7303ed5d74	minor : specify compression level in tags	2017-06-29 11:30:31 -07:00
Yann Collet	590937df20	Merge pull request #739 from facebook/refPrefix ZSTD_refPrefix	2017-06-29 04:36:03 -07:00
Yann Collet	811deaea6f	Merge pull request #736 from terrelln/cover-default-api [zdict] Make COVER the default algorithm	2017-06-28 20:25:36 -07:00
Yann Collet	ad5f29918e	Merge pull request #737 from facebook/advancedAPI2 simplified API for initStatic()	2017-06-28 20:25:22 -07:00
Yann Collet	62f7efc04a	added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job	2017-06-28 16:25:13 -07:00
Paul Cruz	7ac4724bd2	removed fnum from DISPLAY statements	2017-06-28 13:00:49 -07:00
Paul Cruz	e667d33b0b	fixed generation of buggy test, corrected DISPLAY statements for errors	2017-06-28 12:19:37 -07:00
Yann Collet	33a6639039	fixed ZSTD_refPrefix with Multithread-enabled CCtx	2017-06-28 11:09:43 -07:00
Yann Collet	1fd2df9c2c	fixed zstreamtest modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary() but it is still allowed after ZSTD_refPrefix()	2017-06-28 09:57:59 -07:00
Yann Collet	2e4274262d	controlled dictMode	2017-06-27 17:09:12 -07:00
Yann Collet	b7372933b8	implemented ZSTD_refPrefix()	2017-06-27 15:49:12 -07:00
Yann Collet	fecc721fd9	added parameter ZSTD_p_refDictContent	2017-06-27 11:46:39 -07:00
Yann Collet	09a5bbe22e	Merge pull request #735 from terrelln/xz-default [zstdcli] Support xz by default when liblzma is available	2017-06-27 09:42:02 -07:00
Nick Terrell	5b7fd7c422	[zdict] Make COVER the default algorithm	2017-06-26 21:09:22 -07:00
Yann Collet	20eeb243d1	Merge pull request #729 from paulcruz74/corpus Corpus	2017-06-26 17:47:28 -07:00
Yann Collet	dde10b23fe	refactored ZSTD_estimateDStreamSize() now uses windowSize as argument. Also : created ZSTD_estimateDStreamSize_fromFrame()	2017-06-26 17:44:26 -07:00
Yann Collet	09ae03a570	ZSTD_estimateCDictSize_advanced() ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict() ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()	2017-06-26 16:47:32 -07:00
Yann Collet	0c9a915a28	ZSTD_estimateCStreamSize_advanced()	2017-06-26 16:02:25 -07:00
Yann Collet	31af8290d1	ZSTD_estimateCCtx_advanced() ZSTD_estimateCCtx() is now a "simple" function, taking int compressionLevel as single argument. ZSTD_estimateCCtx_advanced() takes a CParams argument, which is both more complete and more complex to generate.	2017-06-26 15:52:39 -07:00
Nick Terrell	6aeb50ea0d	[zstdcli] Add unlzma and unxz symlinks	2017-06-26 11:24:36 -07:00
Paul Cruz	298fe9f23b	added cli tests to test-all	2017-06-23 17:10:41 -07:00
Paul Cruz	3a295a91f8	added additional condition so large offsets into the dictionary are not generated past windowSize	2017-06-23 15:54:51 -07:00
Paul Cruz	2085375816	fixed bug detected by the API test	2017-06-23 13:44:24 -07:00
Paul Cruz	8cd134559d	type warnings	2017-06-23 12:00:48 -07:00
Paul Cruz	4219acc60a	fixed bus error bug	2017-06-23 11:22:29 -07:00
Yann Collet	ef269c1b68	Merge pull request #725 from facebook/advancedAPI2 New Advanced API	2017-06-23 09:50:47 -07:00
Yann Collet	cca1ed14db	Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2	2017-06-23 00:13:03 -07:00
Yann Collet	32f546c763	zstreamtest : disabled multi-threading tests with --newapi --no-big-tests --no-big-tests is typically used in combination with qemu-user-static qemu-user-static allocated 4 GB of RAM upfront. On 2 GB VM, this can degenerate into a crash. It's not a problem as long as memory is not used. But with multi-threading enabled, memory fragmentation kicks in, so the amoung of RAM effectively touched increases, and can pass beyond the 2 GB limit of the VM. In single-threaded mode, there is no such issue : memory requirement is smaller, and remains well-located, so very little fragmentation is expected. This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests` work fine on Travis CI.	2017-06-23 00:09:02 -07:00
Paul Cruz	2e8cc6f12a	added sizeof for clarity	2017-06-22 15:52:33 -07:00
Paul Cruz	6d7f442120	changed makefile test	2017-06-22 15:44:01 -07:00
Paul Cruz	b325a2e4db	changed assignment	2017-06-22 15:36:28 -07:00
Paul Cruz	2935239544	added description for dictionary cli tests	2017-06-22 15:23:59 -07:00
Paul Cruz	97a8f89bec	made changes to cli test so it runs properly	2017-06-22 15:13:41 -07:00
Paul Cruz	829eb29033	added cli test for decodecorpus inside tests/Makefile. Also changed calculation of offset	2017-06-22 14:43:44 -07:00
Paul Cruz	028fb9d01b	added cli test case in Makefile	2017-06-22 13:53:39 -07:00
Paul Cruz	98751f69e7	should be updating seed whenever multiple files are generated	2017-06-22 10:23:36 -07:00
Paul Cruz	84cfa07d2d	changed format of command to --use-dict=#	2017-06-22 10:04:14 -07:00
Yann Collet	f99c2c1a22	reduced --no-big-tests even more to pass tests on qemu-aarch64	2017-06-21 23:35:58 -07:00
Paul Cruz	04094f37e9	fixed offset in this case os that it always goes past src start	2017-06-21 18:47:40 -07:00
Yann Collet	49f8459d11	fixed minor cast warning	2017-06-21 18:43:39 -07:00
Paul Cruz	0950b3159a	more meaningful names for count variables	2017-06-21 18:30:27 -07:00
Paul Cruz	0b6eedeace	malloc samples instead of static allocation	2017-06-21 18:24:19 -07:00
Yann Collet	4d3bdcf130	reduced CLevelMax for --no-big-tests	2017-06-21 18:20:02 -07:00
Paul Cruz	52a13abf3f	removed comment regarding windowSize requirement	2017-06-21 18:13:52 -07:00
Paul Cruz	32e36c2d9a	removed requirement that offset be less than windowSize	2017-06-21 18:00:47 -07:00
Yann Collet	bfc2f00080	--no-big-tests for zstreamtest Apply --no-big-tests for tsan tests	2017-06-21 17:57:14 -07:00
Paul Cruz	eeff2d0e3c	added const x2	2017-06-21 17:48:37 -07:00
Paul Cruz	04253e21bb	added const	2017-06-21 17:48:01 -07:00
Paul Cruz	2ceaec1aad	got rid of unnecessary casting to void*	2017-06-21 17:46:37 -07:00
Paul Cruz	ccae9ec807	added comment addressing use of bits in frame header descriptor	2017-06-21 17:43:21 -07:00
Yann Collet	dce789281b	fixed : decompression of skippable frames in streaming mode	2017-06-21 15:53:42 -07:00
Yann Collet	c16748b137	added test cases for new command --list	2017-06-21 12:09:53 -07:00
Yann Collet	7bd1a2900e	added ZSTD_dictMode_e to control dictionary loading mode	2017-06-21 11:50:33 -07:00
Paul Cruz	db3606e249	added test for files with no frame content size	2017-06-20 17:43:36 -07:00
Paul Cruz	acf0df678c	fixed strange spacing	2017-06-20 15:41:43 -07:00
Paul Cruz	69bc4fab25	free the context object once done with decompression	2017-06-20 15:37:14 -07:00
Paul Cruz	ceb4f65620	allocate statically so we don't have to deal with freeing	2017-06-20 15:31:54 -07:00
Paul Cruz	a73c2a444a	added tests with null files, changed condition to check that the file is more than 0 bytes	2017-06-20 14:33:08 -07:00
Paul Cruz	58c19b4202	spacing matters for test scripts	2017-06-20 14:14:53 -07:00
Yann Collet	ff8f83bd47	fixed fuzzer test	2017-06-20 12:17:32 -07:00
Paul Cruz	b07d0af2ba	added additional tests for error detection	2017-06-20 11:54:44 -07:00
Paul Cruz	43ffcdde40	const dictOffset	2017-06-20 11:27:39 -07:00
Paul Cruz	1e0d125e76	const value	2017-06-20 11:15:26 -07:00
Paul Cruz	0034129270	made changes to accommodate single file use case	2017-06-20 11:07:38 -07:00
Paul Cruz	6a99de2dd8	change test to list info on .zst files	2017-06-20 10:45:06 -07:00
Paul Cruz	36e14b1963	added description to advanced usage, changed to use-dict	2017-06-20 10:34:34 -07:00
Paul Cruz	15102069a1	removed testing code from generation code	2017-06-20 10:25:16 -07:00
Paul Cruz	ff5589c091	added comments for clarity	2017-06-20 10:17:38 -07:00
Paul Cruz	551fb7ca05	changed error condition	2017-06-20 10:14:03 -07:00
Paul Cruz	b8237c6c66	refactoring for readability	2017-06-20 10:12:13 -07:00
Paul Cruz	6f17de2132	changed computation for readability	2017-06-20 09:50:28 -07:00
Paul Cruz	4a4f9ba262	const value that doesn't change	2017-06-19 17:46:21 -07:00
Paul Cruz	3a6856edca	changed scope to avoid errors	2017-06-19 17:43:16 -07:00
Paul Cruz	7d2b5613b4	adhere to min dict size	2017-06-19 17:37:14 -07:00
Paul Cruz	ea2af1ffc4	changed dictionary to stay reasonable size during test	2017-06-19 17:34:50 -07:00
Paul Cruz	fdbb07283a	added test to runTestMode	2017-06-19 17:29:15 -07:00
Paul Cruz	bdec7b8ce7	fixed for loop declaration, fixed test case	2017-06-19 17:18:43 -07:00
Paul Cruz	23098b53e8	wrote a test for dictionary corpus	2017-06-19 17:13:00 -07:00
Paul Cruz	7a26390989	changing for loop declaration	2017-06-19 16:39:57 -07:00
Paul Cruz	147ef05f12	for loop declaration not allowed	2017-06-19 16:38:51 -07:00
Paul Cruz	791352ad0e	scope change	2017-06-19 16:32:38 -07:00
Paul Cruz	365e265a8e	changed scope so that warning disappears	2017-06-19 16:28:40 -07:00
Paul Cruz	f4f928595b	changed scope to get rid of mixed code/variables warning	2017-06-19 16:27:05 -07:00
Paul Cruz	7ea025cc7d	added in struct to keep functions more orderly	2017-06-19 16:23:00 -07:00
Paul Cruz	b91b810c64	made some small changes to dictInfo struct and initDictInfo	2017-06-19 16:05:05 -07:00
Paul Cruz	1ee4f6b36c	cleaning up code	2017-06-19 15:55:01 -07:00
Paul Cruz	2a52d72712	added comment to anotate decompressed error code	2017-06-19 14:21:35 -07:00
Paul Cruz	73ebb646d8	small stylistic changes	2017-06-19 14:20:35 -07:00
Paul Cruz	34f4e5cab2	changed if statement styling	2017-06-19 14:07:56 -07:00
Paul Cruz	1fc1a35dc4	added calls to free to match calls to malloc, made some stylistic changes, added init function for new struct	2017-06-19 14:02:27 -07:00
Paul Cruz	8f4fa43fe8	got files to decompress	2017-06-19 13:21:50 -07:00
Yann Collet	688952062b	minor declaration statement warning fix	2017-06-19 11:53:42 -07:00
Yann Collet	d7a3bffba9	new api : setting compression parameters is refused if a dictionary is already loaded	2017-06-19 11:53:01 -07:00
Paul Cruz	da4ddc1f76	fixed decompression so that it uses compressed version	2017-06-19 11:40:45 -07:00
Paul Cruz	33467add60	made changes so that HUF_writeCTable no longer fails	2017-06-19 11:27:50 -07:00
Yann Collet	ed1d039127	newapi fuzzer tests : random flush orders in main loop	2017-06-19 11:07:33 -07:00
Paul Cruz	09ac51b56b	small style changes	2017-06-19 09:24:38 -07:00
Yann Collet	f5deae8a67	new api fuzzer tests : frame parameters are randonly set	2017-06-18 23:41:38 -07:00
Yann Collet	ea3630a889	new api fuzzer set : dictionary is randomly set	2017-06-18 23:31:55 -07:00
Yann Collet	8dee0ec99e	new api fuzzer tests : compression parameters are randomly set	2017-06-18 23:25:15 -07:00
Yann Collet	bd05b19713	removed macro CLAMP from decodecorpus never used, and would duplicate with CLAMP from zstd_compress.c	2017-06-16 18:16:27 -07:00
Yann Collet	01743a36e7	fuzzer tests for new API	2017-06-16 17:56:41 -07:00
Paul Cruz	2f30433cd7	fixing general style	2017-06-15 18:06:32 -07:00
Paul Cruz	ddd1ab710c	added tests in playTest.sh	2017-06-15 16:53:03 -07:00
Paul Cruz	623b8fce85	add print statements for debugging and change malloc with updated size	2017-06-15 11:37:24 -07:00
Paul Cruz	52934fbfc8	added print statements for debugging, fixed first segfault	2017-06-15 09:47:29 -07:00
Paul Cruz	03a208e27c	changed samples to geometric distribution so that it is huffman compressible	2017-06-14 18:06:16 -07:00
Paul Cruz	d93207a79f	changed dictionary from using fixed amount of bytes for the header / entropy tables	2017-06-14 17:23:56 -07:00
Paul Cruz	664ed05ff6	changed randomly generated samples to randomly selected from the dictionary content	2017-06-14 16:42:51 -07:00
Paul Cruz	3b0e24c06b	added error message to catch case where size not given	2017-06-14 12:04:22 -07:00
Paul Cruz	aa95bc0a5f	code refactoring and adding some print statements for debugging	2017-06-14 10:54:47 -07:00
Paul Cruz	07cfc975bb	debugging error with finalizeDictionary() not writing dictionary properly	2017-06-13 17:27:02 -07:00
Paul Cruz	8dd621f788	changed makefile to include zdict functions, wrote out dictionary to path	2017-06-13 14:19:35 -07:00
Paul Cruz	f35f252e36	added code to generate dictionary using finalizeDictionary	2017-06-13 11:54:43 -07:00
Paul Cruz	37e1b1488b	removed previous double free crash, used new crash function	2017-06-09 13:57:18 -07:00
Paul Cruz	be9b0ae627	added error function for different compilation modes	2017-06-09 13:41:34 -07:00
Paul Cruz	11c3987baf	added code to extend the offset when a dictionary is detected	2017-06-09 09:48:56 -07:00
Paul Cruz	125ed59968	made sure dictionary ID was being written in the frame header of each file	2017-06-08 17:27:15 -07:00
Paul Cruz	233ee5334e	set the lower bits of frame header descriptor if dictionary is needed	2017-06-08 17:11:33 -07:00
Paul Cruz	c2d909e396	added code for generating dictionary/test files randomly. Still need to make sure dictionary ID matches	2017-06-08 17:06:30 -07:00
Yann Collet	8c910d2097	updated ZSTDMT streaming API ZSTDMT streaming API is now similar and has same capabilites as single-thread streaming API. It makes it easier to blend them together.	2017-06-03 01:15:02 -07:00
Yann Collet	cdf7e82222	Added ZSTD_initStaticCDict()	2017-05-25 18:05:49 -07:00
Yann Collet	57827f906f	added ZSTD_initStaticDDict()	2017-05-25 15:44:06 -07:00
Yann Collet	25989e361c	updated ZSTD_estimate?DictSize() to pass parameter byReference resulting ?Dict object is smaller when created byReference. Seems better than a documentation note.	2017-05-25 15:07:37 -07:00
Yann Collet	0fdc71c3dc	added ZSTD_initStaticDCtx()	2017-05-24 17:41:41 -07:00
Yann Collet	c7fe262dc9	added ZSTD_initStaticCCtx() makes it possible to statically or externally allocate CCtx. static CCtx will only use provided memory area, it will never resize nor malloc.	2017-05-23 13:20:41 -07:00
Yann Collet	60a557e7fd	fixing symbol.c test I believe it would be better to rely on fuzzer tests compiled with dll	2017-05-19 11:27:43 -07:00
Yann Collet	fa3671eac7	changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX Also : change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize() created ZSTD_BLOCKSIZELOG_MAX	2017-05-19 10:51:30 -07:00
Yann Collet	009d604e00	ZSTD_compress_generic() supports multiple successive frames also : clarified streaming API implementation	2017-05-19 10:17:59 -07:00
Yann Collet	0bdb575c31	Merge branch 'dev' into advancedAPI2	2017-05-16 16:32:29 -07:00
Yann Collet	2d4d31c18a	removed gcc compilation flag -Wbad-function-cast It makes it more difficult to directly cast the result of a function, requiring to store the result in an intermediate variable. It does not necessarily help readability, and this restriction can be difficult to overcome in some constructions, like some macros. also : fixed minor Visual conversion warnings in datagencli.c	2017-05-16 11:34:38 -07:00
Yann Collet	83d0c764dc	added several compilation flags	2017-05-15 17:15:46 -07:00
Yann Collet	a5ffe3d370	pushed enum values for strategy by one (ZSTD_fast==1) this makes it possible to use `0` to mean: "do not change strategy"	2017-05-12 16:29:19 -07:00
Nick Terrell	5f2c7213c7	Merge remote-tracking branch 'upstream/dev' into btopt * upstream/dev: (305 commits) added test for ZSTD_estimateCStreamSize() changed variable name, for clarity fixed ZSTD_estimateCStreamSize() shortened ZSTD_createCStream_Advanced() fixed symbols test added ZSTD_estimateDStreamSize() changed name frameParams into frameHeader regroup memory usage function declarations separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize() bumped version number added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() Updated ZSTD_freeCCtx() updated ZSTD_estimateCCtxSize() Updated ZSTD_sizeof_CCtx() merged CCtx and CStream as a single same object cli : -d and -t do not stop after a failed decompression added dev branch CircleCI badge added dev branch Appveyor badge keep dev branch status only creates a binary archive without the `programs` directory ...	2017-05-10 16:49:58 -07:00
Yann Collet	30ab64e21d	added test for ZSTD_estimateCStreamSize()	2017-05-10 11:30:19 -07:00
Yann Collet	461a9cc7c6	fixed symbols test	2017-05-09 16:53:09 -07:00
Yann Collet	f16f4497ca	added ZSTD_estimateDStreamSize()	2017-05-09 16:18:17 -07:00
Yann Collet	542c9dfcf8	changed name frameParams into frameHeader ZSTD_frameParams => ZSTD_frameHeader ZSTD_getFrameParams() -> ZSTD_getFrameHeader() The new naming is more distinctive from ZSTD_frameParameters, which is used during compression. ZSTD_frameHeader is clearer in its intention to described frame header content. It also implies we are decoding a ZSTD frame, hence we are at decoding stage.	2017-05-09 15:46:07 -07:00
Yann Collet	fa8dadb294	separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize() for clarity	2017-05-08 18:24:16 -07:00
Yann Collet	a1d6704d7f	added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() it complements ZSTD_estimateCCtxSize() for the special case of ZSTD_initCStream_usingDict()	2017-05-08 17:51:49 -07:00
Yann Collet	fc5145955a	updated ZSTD_estimateCCtxSize() added a parameter streaming, to estimate memory allocation size when the CCtx is used for streaming (CStream). Note : this function is not able to estimate memory cost of a potential internal CDict which can only happen when starting with ZSTD_initCStream_usingDict()	2017-05-08 17:07:59 -07:00
Yann Collet	0be6fd3429	merged CCtx and CStream as a single same object To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()	2017-05-08 16:08:01 -07:00
Yann Collet	01a1abfdb5	cli : -d and -t do not stop after a failed decompression The problematic srcfile will be named on console/log, but decompression/test will continue onto next file in the list.	2017-05-05 19:15:24 -07:00
Nick Terrell	f376d47c11	[CLI] Switch dictionary builder on CLI to cover	2017-05-02 11:18:27 -07:00
Yann Collet	a51cab6e68	Merge pull request #678 from facebook/apiChange Breaking API Change around CDict	2017-04-28 10:02:45 -07:00
Yann Collet	36e79a1a1f	tests : -g is part of DEBUGFLAGS	2017-04-27 17:29:40 -07:00
Yann Collet	c6915429f2	shortened Appveyor release tests fuzzer supports time suffix `s` for "seconds"	2017-04-27 16:24:53 -07:00
Yann Collet	a92cbb7004	Added a secondary test, checking dictID presence after setting noDictIdFLag=1	2017-04-27 15:08:56 -07:00
Yann Collet	7d283cdfa2	added test for ZSTD_initCStream_usingCDict_advanced()	2017-04-27 14:48:34 -07:00
Yann Collet	2f73427d35	added test for ZSTD_compress_usingCDict_advanced()	2017-04-27 14:39:39 -07:00
Yann Collet	7321345fd2	fixed another VS2008 init error	2017-04-27 14:19:34 -07:00
Yann Collet	f4bd857d81	created ZSTD_compress_usingCDict_advanced()	2017-04-27 11:31:55 -07:00
Yann Collet	31533bacce	Changed ZSTD_createCDict_advanced() It now only uses compressionParameters as argument. It produces many changes throughout user code, though hopefully they tend to be simple : just provide the cParams part from existing ZSTD_parameters. Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters. This change will force them to revisit this strategy and fix it, since frame parameters are effectively silently ignored in current version.	2017-04-27 00:29:04 -07:00
Sean Purcell	4de8632957	Add LZ4 compress/decompress support to CLI	2017-04-24 16:48:25 -07:00
Yann Collet	230d7acc7d	cli : add support for --threads=# command updated documentation add relevant test case	2017-04-21 11:38:13 -07:00
Yann Collet	c17e020c9a	disable assert when compiling paramgrill paramgrill is a benchmark calibration function. Speed accuracy is critical, it cannot be altered by assert.	2017-04-20 12:50:02 -07:00
Yann Collet	2c5514c759	fixed ZSTDMT_initCStream_advanced() Must use the new ZSTD_compressBegin_usingCDict_advanced() to enforce correct frame parameters	2017-04-18 22:52:41 -07:00
Yann Collet	0bb381dad8	added test for ZSTD_initCStream_advanced() when params.fParams.contentSizeFlags = 1 and pledgedSrcSize = 0 then srcSize should be considered 0 (empty), and not "unknown"	2017-04-18 15:08:52 -07:00
Yann Collet	4f818182b8	clarified frame parameters for ZSTD_compress*_usingCDict() created ZSTD_compressBegin_usingCDict_internal(), which gives direct control to frame Parameters. ZSTD_resetCStream_internal() now points into it.	2017-04-17 18:29:06 -07:00
Yann Collet	9c4c970399	Merge branch 'dev' into compressionFlow	2017-04-17 16:33:09 -07:00
Sean Purcell	d845dab69c	Fix input size too small to trigger zstdmt	2017-04-17 12:10:58 -07:00
Sean Purcell	c424ec2eae	Add multithreading tests to playTests.sh	2017-04-17 11:41:46 -07:00
Yann Collet	2afb075382	Merge branch 'dev' into compressionFlow	2017-04-14 15:08:31 -07:00
Yann Collet	f913cbed33	fixed : memory leak in fuzzer test	2017-04-13 22:46:51 -07:00
Yann Collet	20d5e03893	content size is controlled at bufferless level so it's active for all entry points Also : added relevant test (wrong content size) in fuzzer	2017-04-11 18:34:02 -07:00
Sean Purcell	d37e1df2ab	Fix message	2017-04-11 17:33:26 -07:00
Sean Purcell	eb70d219fd	Add test of file > 4GB to playTests	2017-04-11 17:15:13 -07:00
Yann Collet	4ee6b15dac	force contentSizeFlag=0 when using ZSTD_initCStream_usingCDict() because by definition srcSize is not known when using this prototype. added relevant test Note : this use was already working, because at a later stage (both ZSTD_compressBegin_usingCDict() and ZSTD_copyCCtx()) pledgedSrcSize=0 is translated into "unknown", no matter the frame parameter. This is not correct, but of little importance, as the medium term plan is to no longer set fParams within CDict	2017-04-11 11:59:44 -07:00
Yann Collet	4b987ad8ce	Introduce ZSTD_initCStream_internal() This is now the regroup point for ZSTD_initCStream*() functions ZSTD_initCStream_advanced() now properly checks for parameters validity. Also : added <assert.h> usage inside zstd_compress.c Needs ZSTD_DEBUG=1 macro to be triggered. Will be triggered by default from `tests` directory	2017-04-10 17:50:44 -07:00
Yann Collet	1fa3b75369	Merge pull request #648 from facebook/zstdmt improved zstreamtest --mt to trap bug #644	2017-04-07 06:09:52 -07:00
Sean Purcell	1652172b2d	Add refuse to overwrite test and fix format	2017-04-06 17:06:30 -07:00
Sean Purcell	9da11c6aae	Add test for overwriting read-only	2017-04-06 12:58:49 -07:00
Yann Collet	ce80098f14	improved zstreamtest --mt to trap bug #644	2017-04-05 16:34:09 -07:00
Yann Collet	9631e603ae	paramgrill : results from optimizer use cli format Best found configuration is displayed using documented cli advanced parameter syntax.	2017-04-04 16:54:33 -07:00
Yann Collet	89b32f3db0	fix paramgrill -O# ensure proposed config does not require more memory than necessary	2017-04-04 16:41:11 -07:00
Yann Collet	2724f25567	fixed paramgrill -O# find optimal settings for a minimum speed	2017-04-04 16:31:17 -07:00
Yann Collet	c2007388a5	fixed bench.c : optional advanced parameters applied before creating cdict	2017-04-04 15:35:06 -07:00
Nick Terrell	39a6cc5172	Make ZSTD_compress_usingCDict() respect contentSizeFlag	2017-04-03 21:09:55 -07:00
Nick Terrell	62ecad3819	Fix ZSTD_initCStream_usingCDict() to use dictionary	2017-04-03 21:05:59 -07:00
Yann Collet	30c7698970	optimize ZSTDMT_compress() memory usage does no longer allocate temporary buffers when there is enough room in dstBuffer to decompress directly there. (previous method would skip that for 1st chunk only). Also : fix ZSTD_compressBound() for small srcSize	2017-03-31 18:27:03 -07:00
Yann Collet	6476c51b86	Merge pull request #637 from facebook/zstdmt Zstdmt	2017-03-30 14:18:37 -07:00
Nick Terrell	5152fb2cb2	Convert all tabs to spaces	2017-03-29 18:51:58 -07:00
Yann Collet	1e1e26f2ac	fixed #634 : ZSTDMT_compressCCtx() doesn't provide frame content size in header	2017-03-29 17:09:59 -07:00
Yann Collet	0ef6803707	added ZSTD_getFrameContentSize() to ZSTDMT unit test in fuzzer Now tests fail, because ZSTDMT_compress() doesn't fill frame content size correctly.	2017-03-29 16:58:57 -07:00
Yann Collet	41fefd573a	Improved speed tests Better compatibility with Mac OS-X Force attribution to a selected core # (Linux)	2017-03-26 23:52:19 -07:00
Yann Collet	4c41d37fcc	changed test for new syntax --dictID= and --maxdict=	2017-03-24 18:36:56 -07:00
Yann Collet	f332ece468	dictBuilder fails to create dictionary on certain input Properly expressed with an error code (see zstd_errors.h) and a cli return code != 0	2017-03-23 16:24:02 -07:00
Sean Purcell	680e4e0953	Ignore symbolic links unless --force specified	2017-03-23 11:52:09 -07:00
Sean Purcell	042ba122ae	Change g_displayLevel to int and fix DISPLAYUPDATE flush	2017-03-23 11:21:59 -07:00
Sean Purcell	1b418bd3da	Merge remote-tracking branch 'origin/dev' into appveyor	2017-03-21 10:59:54 -07:00
Sean Purcell	5a2b857029	Update appveyor.yml to have short and long tests	2017-03-20 17:04:44 -07:00
Sean Purcell	c771977489	Fix decodecorpus clang compile errors	2017-03-20 16:02:24 -07:00
Sean Purcell	2f6c7e6a53	Fix windows compiler warnings for decodecorpus	2017-03-20 13:45:12 -07:00
Yann Collet	170861da70	Merge pull request #610 from iburinoc/slowfuzzer Strongly limit compression level when source is large for tests running on QEMU	2017-03-20 13:36:01 -07:00
Sean Purcell	0b0b10ec83	Add --no-big-tests flag for qemu fuzz-tests	2017-03-20 11:32:24 -07:00
Sean Purcell	7ebf2de02d	Add ability to strongly limit fuzzer test size with flag	2017-03-20 11:25:00 -07:00
Yann Collet	35c147ab6a	Merge pull request #605 from iburinoc/term Fix zstd not erroring on compressing to terminal without forceStdout	2017-03-17 14:02:48 -07:00
Sean Purcell	9a38dfa794	Only run IS_CONSOLE tests with a TTY	2017-03-17 12:32:18 -07:00
Sean Purcell	d973071e90	Add tests for compressed data on console	2017-03-16 16:25:19 -07:00
Sean Purcell	f5e50519e0	Prevent fuzz testers from combining large dicts with high clevel	2017-03-15 15:04:54 -07:00
Yann Collet	3773bd318f	Merge pull request #601 from terrelln/nxz Add xz and lzma support.	2017-03-13 22:00:03 -07:00
Nick Terrell	aa8bcf360f	Add xz and lzma support. Finish feature started by @inikep. * Add xz and lzma compression and decompression support to target `xzstd`. * Fix bug in gzip decompression that silently accepted truncated files. * Add gzip frame composition tests. * Add xz/lzma compatibility tests. * Add xz/lzma frame composition tests.	2017-03-13 18:45:41 -07:00
Sean Purcell	120df494e9	Update builds to not support legacy v01-v03	2017-03-13 14:44:08 -07:00
Sean Purcell	7c8f5d5bc7	Make test times overwritable	2017-03-09 16:05:23 -08:00
Sean Purcell	daec40db24	Update .travis.yml and Makefile for medium tests	2017-03-09 16:05:22 -08:00
Nick Terrell	eeb31eed39	s/ZSTD_btopt2/ZSTD_btultra/g	2017-03-09 11:44:25 -08:00
Yann Collet	8b1d004031	added -Wformat-security flag, as recommended by @pixelb	2017-03-05 21:17:32 -08:00
Yann Collet	27526c7201	make : added target shortest shortest only run fast part of playTests.sh . cc @iburinoc	2017-03-01 17:02:49 -08:00
Yann Collet	c1c040eae1	added gzip tests also : made sure zstd --format=gzip -V would fail if gzip compatibility is not supported	2017-03-01 16:49:20 -08:00
Yann Collet	51598510c0	Merge pull request #580 from facebook/speedStream Improve streaming decompression speed	2017-03-01 10:59:51 -08:00
Yann Collet	a33ae64204	fixed decoding skippable frames	2017-02-28 01:15:28 -08:00
Yann Collet	c0b1731bce	added test for decompression with NULL dict and NULL DDict previous version of ZSTD_decompressMultiFrame() would fail that test	2017-02-28 01:02:46 -08:00
Przemyslaw Skibinski	8b3560e196	update gzip tests	2017-02-28 09:41:23 +01:00
Yann Collet	1d7f30f9d4	Merge branch 'decompressStream' into dev	2017-02-27 20:55:22 -08:00
Yann Collet	a81c2e7e44	Merge pull request #573 from facebook/ddict Improved DDict memory usage	2017-02-27 20:54:42 -08:00
Yann Collet	952d06fa9c	fullbench : -i0 displays list of functions to bench	2017-02-27 17:58:02 -08:00
Yann Collet	67d86a74a5	added test case : --rm on stdin must remain silent (instead of failing)	2017-02-27 16:09:20 -08:00
Yann Collet	ea7589ce07	Merge pull request #571 from inikep/dev11 gzip tests	2017-02-27 13:54:33 -08:00
Przemyslaw Skibinski	5d848527e6	use "./gzip" for gzip tests	2017-02-27 22:02:03 +01:00
Przemyslaw Skibinski	b43d75154d	update gzip tests	2017-02-27 09:07:35 +01:00
Przemyslaw Skibinski	1479c98661	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2017-02-27 08:56:43 +01:00
Sean Purcell	2302bfa4bf	Merge branch 'dev' into testcorpus	2017-02-26 22:27:31 -08:00
Yann Collet	bd7fa21deb	added ZSTD_refDDict() Now DDict does no longer depends on DCtx duplication	2017-02-26 14:43:07 -08:00
Yann Collet	8dff956dbf	Added DDict unit test in fuzzer also : slightly modified loadEntropy : know src must points at start of dictionary	2017-02-25 10:11:15 -08:00
Yann Collet	682ae8a10e	Merge pull request #567 from inikep/dev11 faster Travis tests	2017-02-25 06:57:31 -08:00
Sean Purcell	22468b0cc3	Fix some more ARM compile errors https://travis-ci.org/facebook/zstd/jobs/204807461 Can't get them to reproduce the compile errors locally, tested fix by forcing that test to run on Travis.	2017-02-24 10:55:42 -08:00
Przemyslaw Skibinski	a66b764d79	added tests for gzip	2017-02-24 16:09:17 +01:00
Przemyslaw Skibinski	8740d6bcf5	fix uninitialized value warning	2017-02-24 09:24:55 +01:00
Yann Collet	df9f9296e3	attempt to fix pthreat linking error replacing -lpthread by -pthread	2017-02-24 00:16:05 -08:00
Yann Collet	831b4890ce	minor tests/Makefile refactoring and update of zstd_manual,html	2017-02-23 23:09:10 -08:00
Sean Purcell	d590291d86	Fix -Wsign-compare issues in decodecorpus.c https://travis-ci.org/facebook/zstd/jobs/204423280	2017-02-23 16:11:08 -08:00
Yann Collet	7e224ea03e	Merge pull request #562 from iburinoc/testcorpus Create a tool that generates random, valid, Zstd frames for decoder testing	2017-02-23 15:19:36 -08:00
Sean Purcell	3cd8d50c34	Update CLI and link from educational decoder	2017-02-23 13:06:50 -08:00
Sean Purcell	485ca8c352	Update tests/README.md	2017-02-23 10:27:00 -08:00
Sean Purcell	f119b62055	Create a tool that generates random, valid, Zstd frames for decoder testing Note: Does not handle dictionaries currently	2017-02-22 16:08:34 -08:00
Sean Purcell	9050e1925e	Change name to to findFrameCompressedSize and add skippable support	2017-02-22 12:12:34 -08:00
Przemyslaw Skibinski	337ec875b6	minor tweaks	2017-02-22 10:31:30 +01:00
Przemyslaw Skibinski	3d836bfd18	travis.yml: fix versionsTest target	2017-02-22 09:36:42 +01:00
Przemyslaw Skibinski	346ce32ade	legacy.c: fix memory leaks	2017-02-21 20:10:21 +01:00
Przemyslaw Skibinski	684858e7b7	fix memory leaks	2017-02-21 18:17:24 +01:00
Yann Collet	f2da1b1f22	Merge pull request #550 from iburinoc/matchoverlenwrite execSequence copies up to 2*WILDCOPY_OVERLENGTH extra	2017-02-16 18:14:47 -08:00
Yann Collet	ff176d3208	Merge pull request #548 from inikep/gz_compress Circle CI tests	2017-02-16 14:57:59 -08:00
Sean Purcell	0ed3901b05	Update overlength match test case	2017-02-16 13:36:57 -08:00
Sean Purcell	887eaa9e21	Fix wildcopy overwriting data still in window	2017-02-15 16:43:45 -08:00
Przemyslaw Skibinski	9b5a1e9d97	added circle.yml	2017-02-14 20:06:41 +01:00
Nick Terrell	74b81ada25	Don't run test-pool with QEMU > make test -n ... ./pool > make test -n QEMU_SYS=valgrind ... ./legacy # ./pool not run	2017-02-14 10:08:18 -08:00
Sean Purcell	5069b6c2c3	Merge branch 'dev' into multiframe	2017-02-10 10:08:55 -08:00
Yann Collet	a28c34cb7a	Merge pull request #538 from iburinoc/errorstring Fix ZSTD_getErrorString and add tests	2017-02-10 03:59:56 -08:00
Sean Purcell	84b37cc1f1	Fix failing unit test	2017-02-09 12:27:39 -08:00
Sean Purcell	478d7174e5	Merge branch 'dev' into multiframe	2017-02-09 11:54:22 -08:00
Sean Purcell	2db7249265	Make pledgedSrcSize meaning clear for other functions - Added tests - Moved new size functions to static link only	2017-02-09 11:49:58 -08:00
Przemyslaw Skibinski	80c1e04c33	Merge branch 'dev' into dev11	2017-02-09 09:42:16 +01:00
Sean Purcell	e0b3265e87	Fix ZSTD_getErrorString and add tests	2017-02-08 17:28:49 -08:00
Sean Purcell	f07ddf88e8	Test multiframe legacy decoding with simple and streaming APIs	2017-02-08 15:13:20 -08:00
Sean Purcell	ba2ad9f25c	ZSTD_decompress now handles multiple frames	2017-02-08 14:50:10 -08:00
Sean Purcell	4e709712e1	Decompressed size functions now handle multiframes and distinguish cases - Add ZSTD_findDecompressedSize - Traverses multiple frames to find total output size - Add ZSTD_getFrameContentSize - Gets the decompressed size of a single frame by reading header - Deprecate ZSTD_getDecompressedSize	2017-02-08 14:50:10 -08:00
Nick Terrell	b5e46b1255	Remove test-longmatch from test target and only run it once	2017-02-08 12:00:21 -08:00
Przemyslaw Skibinski	4b4f8c2d71	turn off test-pool for qemu-ppc64-static	2017-02-08 13:58:04 +01:00
Yann Collet	48bed91606	Merge pull request #527 from facebook/zstdmt zstdmt refinements	2017-01-31 16:36:46 -08:00
Yann Collet	92c98a5b21	zstreamtest uses random section sizes for fuzzing	2017-01-30 12:50:31 -08:00
Yann Collet	cd23dd24af	zstreamtest uses random overlapLog for fuzzing	2017-01-30 12:46:35 -08:00
Yann Collet	64bf8ffce6	report @terrelln patch to ST fuzzer tests	2017-01-27 17:25:07 -08:00
Nick Terrell	d98bf49224	Fix segfault in zstreamtest MT It was reading beyond the end of the input buffer because no errors were detected. Once that was fixed, it wasn't making forward progress because no errors were detected and it was waiting for input.	2017-01-27 15:42:36 -08:00
Yann Collet	717c65d690	Merge pull request #519 from inikep/dev11 Dev11	2017-01-26 14:23:44 -08:00
Yann Collet	8dafb1acf5	CLI : automatically set overlap size to max (windowSize) for max compression level	2017-01-25 17:01:13 -08:00
Yann Collet	06e7697f96	added test of new parameter ZSTD_p_forceWindow	2017-01-25 16:39:03 -08:00
Yann Collet	bb0027405a	fixed zstdmt corruption issue when enabling overlapped sections see Asana board for detailed explanation on why and how to fix it	2017-01-25 16:25:38 -08:00
Przemyslaw Skibinski	3bb010a667	.travis.yml: optimized order of short tests	2017-01-25 11:19:35 +01:00
Yann Collet	bd6bc22612	playtest.sh : changed sdiff into $DIFF	2017-01-22 15:54:14 -08:00
Yann Collet	317604e0ad	fixed : compilation of zstreamtest in dll mode	2017-01-20 17:18:41 -08:00
Yann Collet	d7e3cb58c5	Resolved merge conflict dev+zstdmt	2017-01-20 16:44:50 -08:00
Yann Collet	19d670ba9d	Added ZSTDMT_initCStream_advanced() variant Correctly compress with custom params and dictionary Added relevant fuzzer test in zstreamtest Also : new macro ZSTDMT_SECTION_LOGSIZE_MIN, which sets a minimum size for a full job (note : a flush() command can still generate a partial job anytime)	2017-01-19 15:32:07 -08:00
Yann Collet	0f984d94c4	changed MT enabling macro to ZSTD_MULTITHREAD	2017-01-19 14:05:07 -08:00
Yann Collet	f22adae984	fixed minor warning (unused variable) in fuzzer	2017-01-19 13:46:30 -08:00
Yann Collet	736788f8e8	added streaming fuzzer tests for MT API Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize which is possible when many flushes are issued while there is not enough dst buffer to flush completed ones.	2017-01-19 12:15:29 -08:00
Przemyslaw Skibinski	8a0bc30a2d	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2017-01-17 13:02:29 +01:00
Przemyslaw Skibinski	d72f4b6b7a	added "Makefile is validated"	2017-01-17 12:40:06 +01:00
Yann Collet	33fce03045	added test checking dictID when using ZSTD_initCStream_usingCDict() It shows that dictID is not properly added into frame header	2017-01-16 19:46:22 -08:00
Sean Purcell	c44c4d5223	Fix missing 'OK' logging on fuzzer testcase	2017-01-12 09:38:33 -08:00
Sean Purcell	834ab50fa3	Fixed decompress_usingDict not propagating corrupted dictionary error	2017-01-11 17:31:34 -08:00
Nick Terrell	a8b4fe0481	Add COVER dictionary builder to fuzzer unit tests	2017-01-02 18:45:19 -08:00
Nick Terrell	3a1fefcf00	Simplify COVER parameters	2017-01-02 17:51:38 -08:00
Nick Terrell	cbb3ce376b	Add cover cli to playtests	2017-01-02 14:43:11 -08:00
Nick Terrell	5ca0fd2045	Shorten thread pool tests	2016-12-31 22:39:32 -05:00
Nick Terrell	d132433534	Switch thread pool test to threading.h	2016-12-31 19:10:13 -05:00
Nick Terrell	9c499648e3	Add thread pool tests	2016-12-29 23:41:03 -08:00
Yann Collet	c07d2e3a31	Merge pull request #499 from inikep/dev11 improved *BSD and Solaris compatibility	2016-12-23 21:32:03 +01:00
Nick Terrell	78a0072d5a	Fix failing test due to deprecation warning	2016-12-22 17:36:16 -08:00
Przemyslaw Skibinski	5f5a902453	"make test" is now compatible with Solaris	2016-12-22 18:05:07 +01:00
Yann Collet	795d95309f	Merge pull request #493 from inikep/fullbench Fullbench extDict variant	2016-12-21 15:12:06 +01:00
Nick Terrell	9d08597364	Add test for invalid offset rep codes	2016-12-20 11:13:45 -08:00
Przemyslaw Skibinski	4beb51f17c	tests of ZSTD_compressContinue_extDict	2016-12-20 10:17:21 +01:00
Yann Collet	d564faa3c6	fix : ZSTD_initCStream_srcSize() correctly set srcSize in frame header	2016-12-18 21:39:15 +01:00
Yann Collet	31ff2a23be	fix Travis long test list; added zbufftest-dll	2016-12-17 19:10:10 +01:00
Yann Collet	d46ecb58a5	added dll compilation tests	2016-12-17 16:28:12 +01:00
Nick Terrell	61e62c014f	Test that all API symbols are exported	2016-12-16 13:29:23 -08:00
Yann Collet	7d5e344ce5	Merge pull request #480 from inikep/dev11 advanced mode where each parameter can be selected on the command line	2016-12-15 22:27:50 +01:00
Przemyslaw Skibinski	f9a56668a6	improved playTests.sh (3	2016-12-14 18:43:06 +01:00
Przemyslaw Skibinski	24a4236111	improved playTests.sh (2)	2016-12-14 18:07:31 +01:00
Przemyslaw Skibinski	ab5ed6fa7f	improved playTests.sh	2016-12-14 17:10:38 +01:00
Przemyslaw Skibinski	9b4fa0ddf7	playTests.sh: added Advanced compression parameters	2016-12-14 16:50:00 +01:00
Yann Collet	e795c8a5f6	Added ZSTD_initCStream_srcSize(). Added relevant test cases in zstreamtest	2016-12-13 17:00:14 +01:00
Dimitry Andric	12df6da83b	Fix running test suite on FreeBSD * Remove last bashism from tests/playTests.sh * Use gmd5sum from the sysutils/coreutils port	2016-12-12 19:22:47 +01:00
Yann Collet	c261f71f6a	minor variation of rescale fix	2016-12-12 00:25:07 +01:00
Nick Terrell	b547d212a1	Fix longmatch test build errors.	2016-12-10 23:17:36 -08:00
Nick Terrell	5cc85cf183	Switch uint64_t to U64	2016-12-10 19:31:55 -08:00
Nick Terrell	0acae734f1	Add exposing test case	2016-12-10 19:12:13 -08:00
Yann Collet	94d1a93d28	changed environment variable comparison to sh compatible	2016-12-06 12:02:56 -08:00
Yann Collet	f586bdfe23	fixed fuzzer test	2016-12-06 06:11:46 +01:00
Yann Collet	825dffbc43	moved zbuff source files into lib/deprecated	2016-12-05 19:28:19 -08:00
Yann Collet	e7a41a5955	added : dictID retrieval functions. added : unit tests for dictID retrieval functions	2016-12-05 16:21:06 -08:00
Przemyslaw Skibinski	7c6bbc3298	updated FIO_passThrough	2016-12-05 18:31:14 +01:00
Przemyslaw Skibinski	6b508b1770	updated test-gzstd	2016-12-05 18:02:40 +01:00
Yann Collet	9ffbeea875	API : changed : streaming decompression : implicit reset on starting new frames	2016-12-02 18:37:38 -08:00
Yann Collet	743b33f57e	fix zstdcat	2016-12-02 15:18:57 -08:00
Przemyslaw Skibinski	0efaf7e7b1	added test-gzstd	2016-11-30 15:20:24 +01:00
Yann Collet	4f5350f610	long matches support overflow	2016-11-29 13:12:24 -08:00
Przemyslaw Skibinski	cc3887085f	updated build\README.md	2016-11-21 13:58:58 +01:00
Przemyslaw Skibinski	8bb86e330b	create DLL with Windows	2016-11-21 12:51:01 +01:00
Yann Collet	0d761dbe95	Merge pull request #453 from inikep/dev11 fullbench-dll	2016-11-16 15:45:30 -08:00
Yann Collet	52afb3993e	zbuff API now generates deprecation warnings	2016-11-16 08:50:54 -08:00
Przemyslaw Skibinski	811b34d962	fix Visual Studio warnings	2016-11-15 19:02:39 +01:00
Przemyslaw Skibinski	179555c1d1	working fullbench-dll	2016-11-15 18:05:46 +01:00
Przemyslaw Skibinski	eb977a42ef	"file" moved to tests/Makefile	2016-11-14 17:26:58 +01:00
Przemyslaw Skibinski	e579ab5faa	introduced QEMU_SYS	2016-11-14 12:57:05 +01:00
Przemyslaw Skibinski	9adf7bfd8a	fixed MinGW compilation (2)	2016-11-03 15:38:13 +01:00
Yann Collet	861cd06ded	fix test-zstd-nolegacy	2016-11-03 01:11:56 -07:00
Yann Collet	4c0b44f8ce	minor display improvement in fuzzer when nb of tests is very large	2016-11-01 11:13:22 -07:00
Yann Collet	589f011761	changed ZBUFF_* by ZSTD_Stream() within fullbench, in anticipation of future deprecation of ZBUFF_.	2016-10-28 15:17:38 -07:00
Yann Collet	b9550d6d43	fixed benchmark mode typo . added benchmark mode tests	2016-10-28 14:43:24 -07:00
Przemyslaw Skibinski	81c334b024	test-zstd-speed.py: default timeout changed to 1800 seconds	2016-10-28 20:40:21 +02:00
Przemyslaw Skibinski	53e7f5c2e4	test-zstd-speed.py: converted to python3 and added "--timeout"	2016-10-28 19:24:16 +02:00
Yann Collet	335ad5d4d4	added ZSTD_initDStream_usingDDict() . slightly optimized ZSTD_initDStream() when no dictionary . fixed ZSTD_sizeof_CStream() .	2016-10-25 17:47:02 -07:00
Yann Collet	9516234e67	first sketch for ZSTD_initCStream_usingCDict()	2016-10-25 16:19:52 -07:00
Przemyslaw Skibinski	5c5f01f3da	added ZSTD_btopt2 strategy	2016-10-25 12:25:07 +02:00
Yann Collet	d7b120ab5c	added long commands --memory= and --memlimit-decompress=	2016-10-14 14:22:32 -07:00
Yann Collet	1122349ac2	added long comment --memlimit=	2016-10-14 14:07:11 -07:00
Yann Collet	d4cda27b63	new command -M#, to limit memory usage during decompression (#403 )	2016-10-14 13:32:20 -07:00
Yann Collet	a17fd7312a	changed error_public.h into zstd_errors.h	2016-10-11 16:41:09 -07:00
Nick Terrell	3e82f2b61a	Remove references to GPLv2 license License headers added to Makefiles were taken from `zstd/Makefile`.	2016-10-11 15:28:40 -07:00
Yann Collet	cf409a7e2a	fixed : init*_advanced() followed by reset() with different pledgedSrcSiz	2016-09-26 16:41:05 +02:00
Yann Collet	2f2639438a	zstreamtest can fuzztest pledgedSrcSize	2016-09-26 14:06:08 +02:00
Yann Collet	58d5dfea54	zstreamtest uses ZSTD_reset?Stream	2016-09-25 01:34:03 +02:00
Yann Collet	97b378a6f8	Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize.	2016-09-21 17:20:19 +02:00
Yann Collet	714464f05d	fixed : cli : forgotten mandatory argument	2016-09-21 16:05:03 +02:00
Yann Collet	27b5ac666e	Implemented "command must be followed by argument" protection suggested by @terrelln (#375)	2016-09-21 14:31:22 +02:00
inikep	3e70cf8036	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev	2016-09-15 20:41:40 +02:00
inikep	dd8905b351	test-zstd-speed.py: better description of options	2016-09-15 20:41:37 +02:00
inikep	ed0ea8d271	test-zstd-speed.py: added "-D dictName"	2016-09-15 20:31:29 +02:00
inikep	d28afac4f8	test-zstd-speed.py: added support for directories	2016-09-15 19:56:04 +02:00
Yann Collet	55f276949c	removed option unsupported by travis	2016-09-15 17:23:15 +02:00
Yann Collet	3e47dbcc8c	fixed memory leak	2016-09-15 17:00:02 +02:00

... 15 16 17 18 19 ...

1736 Commits