townforge/zstd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9,629 Commits 1 Branch 0 Tags 29 MiB
cc
Commit Graph

188 Commits

Author SHA1 Message Date
George Lu
88dda92285 Reduce Duplication 
Change Defaults
Asserts actually disabled in paramgrill + fullbench
 2018-08-14 14:24:41 -07:00
cyan4973
3f535007e4 fix %zu support under minGW 
and relevant test on Appveyor
 2018-07-30 16:56:18 +02:00
cyan4973
3d4b09a5aa support %zu under mingw 2018-07-30 16:29:20 +02:00
Yann Collet
689bfecd48
Merge pull request #1188 from GeorgeLu97/BenchModule 
Bench module
 2018-07-02 13:33:27 -07:00
Yann Collet
328ec7e34c fix test-zstd can be run with parallel compilation 
fix #1221
 2018-06-29 16:52:21 -07:00
Yann Collet
c9e8ee93a7 removed specific --opaqueapi test 
from zstreamtest.

This test is now integrated within --newapi,
which dynamically switches between the 2 modes randomly.

The main outcome is reduced testing time.
 2018-06-18 19:20:37 -07:00
George Lu
8522346322 Make Fullbench use new function 
Rearrange Args
Add nothing function
Use new function, change locals to match
New Display
Comment cleanup
Change builds
 2018-06-15 11:37:49 -04:00
Yann Collet
f70f829ff5
Merge pull request #1187 from facebook/fix1186 
fix dctx initialization within ZSTD_decompress in stack mode
 2018-06-14 16:22:22 -04:00
Yann Collet
7fee966f02 fix dctx initialization within ZSTD_decompress in stack mode 
when ZSTD_HEAPMODE=0 (which is not default).

Also : added an associated test (test-fuzzer-stackmode)
run on travis CI

fix #1186
 2018-06-14 10:22:24 -04:00
Yann Collet
fa41bcc2c2 grouped debug functions into debug.h 
There were 2 competing set of debug functions
within zstd_internal.h and bitstream.h.
They were mostly duplicate, and required care to avoid messing with each other.

There is now a single implementation, shared by both.

Significant change :
The macro variable ZSTD_DEBUG does no longer exist,
it has been replaced by DEBUGLEVEL,
which required modifying several source files.
 2018-06-13 15:43:09 -04:00
Yann Collet
c986dbf241
Merge pull request #1168 from GeorgeLu97/paramgrillfeatures 
Have paramgrill share bench.c benchmarking function
 2018-06-13 11:38:29 -04:00
Ryan Schmidt
b567ce9d68 Fix name of macOS 2018-06-09 14:31:17 -05:00
George Lu
0e808d608b Make paramgrill use bench.c benchmarking 2018-06-08 12:01:05 -07:00
Yann Collet
776128d16f fix corner case when requiring cost of an FSE symbol 
ensure that, when frequency[symbol]==0,
result is (tableLog + 1) bits
with both upper-bit and fractional-bit estimates.

Also : enable BIT_DEBUG in /tests
 2018-05-24 13:59:11 -07:00
Yann Collet
453d487807 fixed minor declaration warning 2018-03-20 18:03:56 -07:00
Yann Collet
0bc7a75468
Merge pull request #987 from facebook/checkTag 
Check tag
 2018-01-16 11:52:32 -08:00
Yann Collet
f4e58455f6 ensure MOREFLAGS are not lost in root->tests Makefile invocation 2018-01-16 11:50:16 -08:00
Yann Collet
f6e17b8c7a added tests/checkTag 
compared provided tag with current libzstd version
 2018-01-14 17:03:45 -08:00
Yann Collet
4792ac6689 make -C tests legacy : minor flag alteration 
ZSTD_LEGACY_SUPPORT is a macro constant,
so it should be part of CPPFLAGS, instead of CFLAGS
 2018-01-14 14:09:17 -08:00
Yann Collet
99a0a8bbd2 tests/Makefile : fixed target allnothread 
ensures MT is disabled
 2018-01-13 22:00:05 -08:00
Yann Collet
f8a05932b9 added make list capability to tests/Makefile 2018-01-13 21:54:21 -08:00
Yann Collet
3d578eef0c minor tests/Makefile change, as suggested by @terrelln 
https://github.com/facebook/zstd/pull/965#discussion_r160008006
 2018-01-06 07:13:09 +01:00
Yann Collet
2126ca8a5f %.o objects files in /tests 
Recipe in /tests rebuild everything from source for each target.
zstd is still a "small" project, so it's not prohibitive,
yet, rebuilding same files over and over represents substantial redundant work.

This patch replaces *.c files from /lib by their corresponding *.o files.
They cannot be compiled and stored directly within /lib,
since /tests triggers additional debug capabilities unwelcome in release binary.
So the resulting *.o are stored directly within /tests.

It turns out, it's difficult to find several target using *exactly* the same rules.
Using only the default rules (debug enabled, multi-threading disabled, no legacy)
a surprisingly small amount of targets share their work.

It's because, in many cases there are additional modifications requested :
some targets are 32-bits, some enable multi-threading, some enable legacy support,
some disable asserts, some want different kind of sanitizer, etc.

I created 2 sets of object files : with and without multithreading.
Several targets share their work, saving compilation time when running `make all`.
Also, obviously, when modifying one source file, only this one needs rebuilding.

For targets requiring some different setting, build from source *.c remain the rule.

The new rules have been tested within `-j` parallel compilation, and work fine with it.
 2017-12-27 17:58:27 +01:00
Yann Collet
311878dec3 Improved tests 
- building cli from /tests preserves potential flags in MOREFLAGS (such as asan/usan)
- MT dictionary tests check for MT capability (MT is not enabled by default for zstd32)
 2017-12-13 11:48:30 -08:00
Yann Collet
dba8016d2d Merge branch 'dev' into fix944 2017-12-13 11:20:09 -08:00
Yann Collet
04a1557e28 fuzzer tests : added -v command 
so that unit tests get listed in CI console
and we can see which one failed.
 2017-12-13 10:08:52 -08:00
Yann Collet
e9dc0ab884 zstd binary, when compiled from /tests, enables debug mode 
adds ZSTD_DEBUG=1 and -g
 2017-12-12 18:20:04 -08:00
Yann Collet
453fb5e830 removed -ftrapv from tests/ debug flags 
-ftrapv is apparently buggy for gcc.
versions >= 5 are supposed to work better,
but even then, some complaints say it's still flaky when optimizations are enabled.
I even saw a post saying it only works if one creates its own signal handler,
which would make this flag no longer transparent.

on clang, it seems to work correctly.
But we would need to add a method to selectively add flags depending on compiler.
That's too much troubles for the intended benefit
(just catch integer overflows, which we can also do using ubsan).
 2017-12-01 10:30:53 -08:00
Yann Collet
b3d76e0a94 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-11-28 15:03:38 -08:00
Yann Collet
f5747afe0f add integer overflow trapping to debug builds 
doc: https://gcc.gnu.org/onlinedocs/gcc-4.0.2/gcc/Code-Gen-Options.html
 2017-11-28 15:02:48 -08:00
W. Felix Handte
62c746dcf9 Add Test on LZ4 Format Input Buffer Overrun 2017-11-28 12:06:48 -05:00
Yann Collet
23767e950a fix one UB pointer arithmetic in encoder 
Instead of calculating distance between 2 memory objects, which is UB,
we extract the offset from object 1, and transfer it into object 2.
 2017-11-17 13:24:51 -08:00
Nick Terrell
e600b5d0da [test] Exercise all codes in dictionary tables 2017-10-16 18:05:36 -07:00
Yann Collet
7f6a783862 fixed a small error in decodeCorpus 
a compressed block must be strictly smaller than its decompressed size.
 2017-10-07 15:19:52 -07:00
Yann Collet
e580dc6a4a Merge pull request #860 from felixhandte/zstd-lz4-support-tests 
Add Default LZ4 Support When Available
 2017-09-29 22:32:54 -07:00
Yann Collet
e0065cf660 make test : removed zstreamtest unit tests for variants 
slightly reduced time to create dictionary
at beginning of unit tests
 2017-09-28 18:34:38 -07:00
W. Felix Handte
d0519d4b0c Add CLI Program Name Detection for LZ4 2017-09-28 19:18:15 -04:00
W. Felix Handte
5705d9f25a Add basic tests for the lz4 integration 2017-09-28 19:16:43 -04:00
Yann Collet
60059df051 shorter make test to avoid time out on travis CI 
Timed tests (fuzzer) are reduced
Long tests are shortened (less data generated)
 2017-09-27 15:16:27 -07:00
Yann Collet
d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
Yann Collet
e0cecd8736 fixed poolTests 
needs more dependencies from zstd for custom allocators and error codes
 2017-08-31 15:13:31 -07:00
Yann Collet
b0cb081dc8 last batch of header files changed to reflect new license (#825) 
only remains to update contrib/linux-kernel (@terrelln)
 2017-08-31 12:20:50 -07:00
Stella Lau
9911153723 Move jobSize and overlapLog in zstdmt to cctxParams 2017-08-25 13:14:51 -07:00
Stella Lau
15fdeb9e41 Enforce nbThreads<=1 for estimateCCtxSize 2017-08-24 16:28:49 -07:00
Stella Lau
1c81f725ff Remove duplicated testing code 2017-08-23 15:47:15 -07:00
Stella Lau
91b30dbe84 Remove test parameter 2017-08-21 10:09:06 -07:00
Stella Lau
f181f33bdf Disable tests and refactor 2017-08-21 01:59:08 -07:00
Stella Lau
023b24e6d4 Add cctx param tests 2017-08-20 22:55:07 -07:00
Yann Collet
d6394cc4c3 fixed test-zstd-nolegacy 2017-08-20 10:15:44 -07:00
Yann Collet
fa3aa04ccd Merge pull request #761 from paulcruz74/file-rename 
renamed pool.c to poolTests.c
 2017-07-14 09:09:45 -07:00
Yann Collet
88da8f1816 fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter() 
also : compile fuzzer with MT enabled
 2017-07-10 14:02:33 -07:00
Paul Cruz
89190ef07d renamed pool.c to poolTests.c 2017-07-10 11:32:30 -07:00
Yann Collet
ed0243a63c removed zbufftest from list of all tests 2017-07-07 16:16:14 -07:00
Yann Collet
27e883371d fixed wrong assert() condition 
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
 2017-07-04 19:33:16 -07:00
Yann Collet
2cb9774f5e more precise estimation of amount to flush at end of stream (single thread mode) 
also : can use DEBUGLEVEL variable in /tests
 2017-07-04 12:39:26 -07:00
cyan4973
c07e43c2b5 added --show-leak-kind=all to valgrind tests 2017-07-01 07:05:11 -07:00
Yann Collet
e7e5a8cef7 made fullbench compatible with multi-threading 
fullbench 61/62 measure speed of ZSTD_compress_generic with 2 threads
 2017-06-29 18:56:24 -07:00
Yann Collet
20eeb243d1 Merge pull request #729 from paulcruz74/corpus 
Corpus
 2017-06-26 17:47:28 -07:00
Paul Cruz
298fe9f23b added cli tests to test-all 2017-06-23 17:10:41 -07:00
Paul Cruz
6d7f442120 changed makefile test 2017-06-22 15:44:01 -07:00
Paul Cruz
2935239544 added description for dictionary cli tests 2017-06-22 15:23:59 -07:00
Paul Cruz
97a8f89bec made changes to cli test so it runs properly 2017-06-22 15:13:41 -07:00
Paul Cruz
829eb29033 added cli test for decodecorpus inside tests/Makefile. Also changed calculation of offset 2017-06-22 14:43:44 -07:00
Paul Cruz
028fb9d01b added cli test case in Makefile 2017-06-22 13:53:39 -07:00
Yann Collet
01743a36e7 fuzzer tests for new API 2017-06-16 17:56:41 -07:00
Paul Cruz
8dd621f788 changed makefile to include zdict functions, wrote out dictionary to path 2017-06-13 14:19:35 -07:00
Yann Collet
009d604e00 ZSTD_compress_generic() supports multiple successive frames 
also : clarified streaming API implementation
 2017-05-19 10:17:59 -07:00
Yann Collet
2d4d31c18a removed gcc compilation flag -Wbad-function-cast 
It makes it more difficult to directly cast the result of a function,
requiring to store the result in an intermediate variable.
It does not necessarily help readability,
and this restriction can be difficult to overcome in some constructions,
like some macros.

also : fixed minor Visual conversion warnings in datagencli.c
 2017-05-16 11:34:38 -07:00
Yann Collet
83d0c764dc added several compilation flags 2017-05-15 17:15:46 -07:00
Yann Collet
36e79a1a1f tests : -g is part of DEBUGFLAGS 2017-04-27 17:29:40 -07:00
Yann Collet
c17e020c9a disable assert when compiling paramgrill 
paramgrill is a benchmark calibration function.
Speed accuracy is critical, it cannot be altered by assert.
 2017-04-20 12:50:02 -07:00
Yann Collet
4b987ad8ce Introduce ZSTD_initCStream_internal() 
This is now the regroup point for ZSTD_initCStream*() functions

ZSTD_initCStream_advanced() now properly checks for parameters validity.

Also : added <assert.h> usage inside zstd_compress.c
Needs ZSTD_DEBUG=1 macro to be triggered.
Will be triggered by default from `tests` directory
 2017-04-10 17:50:44 -07:00
Sean Purcell
5a2b857029 Update appveyor.yml to have short and long tests 2017-03-20 17:04:44 -07:00
Sean Purcell
0b0b10ec83 Add --no-big-tests flag for qemu fuzz-tests 2017-03-20 11:32:24 -07:00
Sean Purcell
120df494e9 Update builds to not support legacy v01-v03 2017-03-13 14:44:08 -07:00
Sean Purcell
7c8f5d5bc7 Make test times overwritable 2017-03-09 16:05:23 -08:00
Sean Purcell
daec40db24 Update .travis.yml and Makefile for medium tests 2017-03-09 16:05:22 -08:00
Yann Collet
8b1d004031 added -Wformat-security flag, as recommended by @pixelb 2017-03-05 21:17:32 -08:00
Yann Collet
27526c7201 make : added target shortest 
shortest only run fast part of playTests.sh .
cc @iburinoc
 2017-03-01 17:02:49 -08:00
Yann Collet
df9f9296e3 attempt to fix pthreat linking error 
replacing -lpthread by -pthread
 2017-02-24 00:16:05 -08:00
Yann Collet
831b4890ce minor tests/Makefile refactoring 
and update of zstd_manual,html
 2017-02-23 23:09:10 -08:00
Sean Purcell
f119b62055 Create a tool that generates random, valid, Zstd frames for decoder testing 
Note: Does not handle dictionaries currently
 2017-02-22 16:08:34 -08:00
Przemyslaw Skibinski
337ec875b6 minor tweaks 2017-02-22 10:31:30 +01:00
Przemyslaw Skibinski
3d836bfd18 travis.yml: fix versionsTest target 2017-02-22 09:36:42 +01:00
Yann Collet
ff176d3208 Merge pull request #548 from inikep/gz_compress 
Circle CI tests
 2017-02-16 14:57:59 -08:00
Przemyslaw Skibinski
9b5a1e9d97 added circle.yml 2017-02-14 20:06:41 +01:00
Nick Terrell
74b81ada25 Don't run test-pool with QEMU 
> make test -n
    ...
    ./pool
    > make test -n QEMU_SYS=valgrind
    ...
    ./legacy
    # ./pool not run
 2017-02-14 10:08:18 -08:00
Sean Purcell
5069b6c2c3 Merge branch 'dev' into multiframe 2017-02-10 10:08:55 -08:00
Sean Purcell
478d7174e5 Merge branch 'dev' into multiframe 2017-02-09 11:54:22 -08:00
Sean Purcell
2db7249265 Make pledgedSrcSize meaning clear for other functions 
- Added tests
- Moved new size functions to static link only
 2017-02-09 11:49:58 -08:00
Przemyslaw Skibinski
80c1e04c33 Merge branch 'dev' into dev11 2017-02-09 09:42:16 +01:00
Sean Purcell
f07ddf88e8 Test multiframe legacy decoding with simple and streaming APIs 2017-02-08 15:13:20 -08:00
Nick Terrell
b5e46b1255 Remove test-longmatch from test target and only run it once 2017-02-08 12:00:21 -08:00
Przemyslaw Skibinski
4b4f8c2d71 turn off test-pool for qemu-ppc64-static 2017-02-08 13:58:04 +01:00
Yann Collet
717c65d690 Merge pull request #519 from inikep/dev11 
Dev11
 2017-01-26 14:23:44 -08:00
Przemyslaw Skibinski
3bb010a667 .travis.yml: optimized order of short tests 2017-01-25 11:19:35 +01:00
Yann Collet
d7e3cb58c5 Resolved merge conflict dev+zstdmt 2017-01-20 16:44:50 -08:00
Yann Collet
19d670ba9d Added ZSTDMT_initCStream_advanced() variant 
Correctly compress with custom params and dictionary
Added relevant fuzzer test in zstreamtest

Also :
new macro ZSTDMT_SECTION_LOGSIZE_MIN, which sets a minimum size for a full job
(note : a flush() command can still generate a partial job anytime)
 2017-01-19 15:32:07 -08:00
Yann Collet
0f984d94c4 changed MT enabling macro to ZSTD_MULTITHREAD 2017-01-19 14:05:07 -08:00
Yann Collet
736788f8e8 added streaming fuzzer tests for MT API 
Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize
which is possible when many flushes are issued
while there is not enough dst buffer to flush completed ones.
 2017-01-19 12:15:29 -08:00