townforge/zstd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9,629 Commits 1 Branch 0 Tags 29 MiB
cc
Commit Graph

4226 Commits

Author SHA1 Message Date
Nick Terrell
374f868354 Update whitespace 2017-05-10 17:48:42 -07:00
Nick Terrell
5f2c7213c7 Merge remote-tracking branch 'upstream/dev' into btopt 
* upstream/dev: (305 commits)
  added test for ZSTD_estimateCStreamSize()
  changed variable name, for clarity
  fixed ZSTD_estimateCStreamSize()
  shortened ZSTD_createCStream_Advanced()
  fixed symbols test
  added ZSTD_estimateDStreamSize()
  changed name frameParams into frameHeader
  regroup memory usage function declarations
  separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
  bumped version number
  added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
  Updated ZSTD_freeCCtx()
  updated ZSTD_estimateCCtxSize()
  Updated ZSTD_sizeof_CCtx()
  merged CCtx and CStream as a single same object
  cli : -d and -t do not stop after a failed decompression
  added dev branch CircleCI badge
  added dev branch Appveyor badge
  keep dev branch status only
  creates a binary archive without the `programs` directory
  ...
 2017-05-10 16:49:58 -07:00
Yann Collet
ba41b26405 Merge pull request #689 from facebook/cctxMerge 
Cctx merge
 2017-05-10 14:53:54 -07:00
Yann Collet
cef02d9317 changed variable name, for clarity 
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
 2017-05-10 11:14:08 -07:00
Yann Collet
669346fe8b fixed ZSTD_estimateCStreamSize() 
https://github.com/facebook/zstd/pull/689#discussion_r115637721
 2017-05-10 11:08:00 -07:00
Yann Collet
6fb2f24132 shortened ZSTD_createCStream_Advanced() 
https://github.com/facebook/zstd/pull/689#discussion_r115637613
 2017-05-10 11:06:06 -07:00
Yann Collet
f16f4497ca added ZSTD_estimateDStreamSize() 2017-05-09 16:18:17 -07:00
Yann Collet
542c9dfcf8 changed name frameParams into frameHeader 
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()

The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.

ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
 2017-05-09 15:46:07 -07:00
Yann Collet
5a36c069e7 regroup memory usage function declarations 
in a single paragraph in zstd.h, for clarity
 2017-05-09 15:11:30 -07:00
Yann Collet
fa8dadb294 separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize() 
for clarity
 2017-05-08 18:24:16 -07:00
Yann Collet
51652522a2 bumped version number 2017-05-08 17:52:46 -07:00
Yann Collet
a1d6704d7f added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() 
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
 2017-05-08 17:51:49 -07:00
Yann Collet
7855366598 Updated ZSTD_freeCCtx() 
which can also contain streaming buffers now.
Redirected ZSTD_freeCStream() towards it.
 2017-05-08 17:15:00 -07:00
Yann Collet
fc5145955a updated ZSTD_estimateCCtxSize() 
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).

Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
 2017-05-08 17:07:59 -07:00
Yann Collet
791d744279 Updated ZSTD_sizeof_CCtx() 
can now contain buffers if object used as CStream.
ZSTD_sizeof_CStream() is now just a thin wrapper of ZSTD_sizeof_CCtx().
 2017-05-08 16:17:30 -07:00
Yann Collet
0be6fd3429 merged CCtx and CStream as a single same object 
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
 2017-05-08 16:08:01 -07:00
Yann Collet
d47709b6ea Merge pull request #654 from iburinoc/splittable 
[RFC] Splittable Format and API
 2017-05-08 13:41:56 -07:00
Yann Collet
a00e9599f1 removed -g from DEBUGFLAGS 
It inflates binary sizes, which is negative for the Windows build.
It also makes it impossible to check if 2 different source codes
get nonetheless compiled to the same binary,
since checksum will be different, due to integrated source code.
 2017-05-04 17:24:29 -07:00
Yann Collet
606c04c228 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-05-02 12:13:52 -07:00
Yann Collet
072484a3bf Merge pull request #683 from terrelln/odev 
[CLI] Make cover the default dictionary builder
 2017-05-02 12:13:23 -07:00
Nick Terrell
f376d47c11 [CLI] Switch dictionary builder on CLI to cover 2017-05-02 11:18:27 -07:00
Nick Terrell
020b960e13 [cover] Make optimization faster 2017-05-02 11:02:48 -07:00
Nick Terrell
f2d9ef1dc0 [cover] Optimize case where d <= 8 2017-05-02 11:02:43 -07:00
Nick Terrell
865918dd04 Fix typo in zdict.h 2017-05-02 11:02:37 -07:00
Yann Collet
b184589c4c minor code refactoring for clarity 2017-05-01 11:35:47 -07:00
Yann Collet
33c38b0925 fixed const in prototype, that Visual doesn't accept 2017-05-01 11:12:30 -07:00
Yann Collet
f39a6731ec sync bitstream.h from fse library 2017-05-01 09:56:03 -07:00
Yann Collet
202082f285 sync bitstream from FSE project 
add assert into unsafe *_fast() variants
 2017-04-28 17:00:31 -07:00
Yann Collet
89f50deec7 minor code refactoring 
clearer tables
 2017-04-28 16:52:36 -07:00
Yann Collet
68a7d3d49a added HUF_PUBLIC_API macro to huf.h 
to make it possible to control symbol visibility.
Also : better separation and comments between "public" and "static" sections
 2017-04-28 12:46:48 -07:00
Yann Collet
a51cab6e68 Merge pull request #678 from facebook/apiChange 
Breaking API Change around CDict
 2017-04-28 10:02:45 -07:00
Yann Collet
29297c6751 Changed default level 18 (large input) 
Previous -18 : 4.7 MB/s, R:3.833
New -18 : 5.1 MB/s. R:3.825

It's a better fit within -17 (6.8 MB/s) and -19 (4.0 MB/s)
The new level 18 also uses significantly less memory.
And, it makes a good transition between level 17 (mml5)
and level 19 (mml3).
Up to now, there was no level with mml4.

(note : minmatch setting can have a large impact on some (specific) datasets)
 2017-04-27 17:44:01 -07:00
Yann Collet
a92cbb7004 Added a secondary test, checking dictID presence after setting noDictIdFLag=1 2017-04-27 15:08:56 -07:00
Yann Collet
d3694e6c70 removed C4204 2017-04-27 14:29:35 -07:00
Yann Collet
1c3ab0c77f fixed init error on Visual 2008 2017-04-27 12:57:11 -07:00
Yann Collet
8b669535f8 bumped version number to v1.2.0 2017-04-27 12:50:20 -07:00
Yann Collet
77bf59ef50 added ZSTD_initCStream_usingCDict_advanced() 2017-04-27 11:43:04 -07:00
Yann Collet
f4bd857d81 created ZSTD_compress_usingCDict_advanced() 2017-04-27 11:31:55 -07:00
Yann Collet
69a54d138a fixed compilation warning : declaration-after-statement 2017-04-27 01:11:26 -07:00
Yann Collet
31533bacce Changed ZSTD_createCDict_advanced() 
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
 2017-04-27 00:29:04 -07:00
Yann Collet
768df129d2 changed ZSTD_compressBegin_usingCDict() 
No longer takes `pledgedSrcSize` as argument
this is in line with similar functions ZSTD_compress_usingCDict()
and ZSTD_initCStream_usingCDict().
 2017-04-26 15:42:10 -07:00
Yann Collet
e42afbc6fa Comply with suggested comments by @terrelln 
created FSE_CTABLE_SIZE() and FSE_DTABLE_SIZE()
 2017-04-26 11:39:35 -07:00
Sean Purcell
7d37ca1d5b Merge remote-tracking branch 'origin/dev' into splittable 2017-04-21 14:18:39 -07:00
Yann Collet
7271203bdb transferred entropy scratch space from CCtx into workSpace 
Saved 6 KB
 2017-04-20 23:21:19 -07:00
Yann Collet
a408645f50 made some room for entropy scratch space 2017-04-20 23:09:39 -07:00
Yann Collet
71aaa32c3c transferred FSE tables from CCtx into workspace 
Saved 5 KB from CCtx
 2017-04-20 23:03:38 -07:00
Yann Collet
71ddeb67b1 made room in workspace for FSE tables 
still need to be transferred from CCtx into workspace
 2017-04-20 22:54:54 -07:00
Yann Collet
a34a39c183 changed size evaluation of entropy tables 
so that memcpy() does no longer depends on fse pointer being a static table
 2017-04-20 18:26:25 -07:00
Yann Collet
7bb60b17d8 init entropy table pointers only once 
per workSpace resize
 2017-04-20 17:38:56 -07:00
Yann Collet
e6fa70a0a1 reorganized ZSTD_resetCCtx_internal() 
clearer separation between variables and buffers
clearer buffers category
kept static buffers at the beginning, favoring cache locality
(it will be easier to add FSE tables there later)

This break a few assumptions that hashTable was always at the beginning.
This is fixed.
And remaining assumptions (namely that tables stand next to each other in memory)
are now tested with assert.
 2017-04-20 17:28:31 -07:00
Yann Collet
c17e020c9a disable assert when compiling paramgrill 
paramgrill is a benchmark calibration function.
Speed accuracy is critical, it cannot be altered by assert.
 2017-04-20 12:50:02 -07:00
Yann Collet
16f9c572fc Merge branch 'dev' into compressionFlow 2017-04-20 11:16:40 -07:00
Yann Collet
e348dad305 minor long line reformatting 2017-04-20 11:14:13 -07:00
Yann Collet
e847730452 slightly refined README comments on lib-mt 2017-04-18 23:15:28 -07:00
Yann Collet
2c5514c759 fixed ZSTDMT_initCStream_advanced() 
Must use the new ZSTD_compressBegin_usingCDict_advanced()
to enforce correct frame parameters
 2017-04-18 22:52:41 -07:00
Sean Purcell
98cf7fcb2a Update README 2017-04-18 17:03:37 -07:00
Sean Purcell
0f7bd772e6 Update seekable API to simplify IO 2017-04-18 16:48:30 -07:00
Yann Collet
a4cab80183 added ZSTD_copyCCtx_internal() 
which respects provided fParams.
 2017-04-18 14:54:54 -07:00
Sean Purcell
ca6fae7808 Add MT enabled targets for libzstd 2017-04-18 14:13:01 -07:00
Yann Collet
30fb499208 Changed ZSTD_resetCCtx_advanced() into ZSTD_resetCCtx_internal() 
for naming consistency :
_advanced() can be invoked
while _internal() are strictly static
 2017-04-18 14:08:50 -07:00
Yann Collet
715b9aa113 created ZSTD_compressBegin_usingCDict_advanced() 2017-04-18 13:55:53 -07:00
Yann Collet
af4f45b682 Improved code comments for block functions 2017-04-18 03:17:44 -07:00
Yann Collet
4f818182b8 clarified frame parameters for ZSTD_compress*_usingCDict() 
created ZSTD_compressBegin_usingCDict_internal(),
which gives direct control to frame Parameters.
ZSTD_resetCStream_internal() now points into it.
 2017-04-17 18:29:06 -07:00
Yann Collet
c47c68f6ca proper evaluation of Huffman CTable size 2017-04-17 16:14:21 -07:00
Sean Purcell
5ee1135f30 s/chunk/frame/ 2017-04-12 11:15:50 -07:00
Yann Collet
88009a8ba2 removed srcSize control from CStream 
since it's already done from lower bufferless API level
 2017-04-12 00:51:24 -07:00
Yann Collet
20d5e03893 content size is controlled at bufferless level 
so it's active for all entry points

Also : added relevant test (wrong content size) in fuzzer
 2017-04-11 18:34:02 -07:00
Sean Purcell
d048fefef7 Move seekable format content to /contrib 2017-04-11 14:38:56 -07:00
Sean Purcell
45f3bc4801 Add format specification 2017-04-11 13:53:09 -07:00
Sean Purcell
a3b7c22604 Make seekable streams work w/ small buffers, misc fixes 2017-04-11 13:53:09 -07:00
Sean Purcell
c3ba15e48f Seekable compression demo 2017-04-11 13:53:09 -07:00
Yann Collet
4ee6b15dac force contentSizeFlag=0 when using ZSTD_initCStream_usingCDict() 
because by definition srcSize is not known when using this prototype.
added relevant test

Note : this use was already working, because at a later stage
(both ZSTD_compressBegin_usingCDict() and ZSTD_copyCCtx())
pledgedSrcSize=0 is translated into "unknown", no matter the frame parameter.
This is not correct, but of little importance,
as the medium term plan is to no longer set fParams within CDict
 2017-04-11 11:59:44 -07:00
Yann Collet
ab9162ebb4 simplified call graph 
by calling ZSTD_compressBegin_internal() instead of ZSTD_compressBegin_advanced()
 2017-04-11 10:46:20 -07:00
Yann Collet
e88034fe26 simplified ZSTD_initCStream*() flow 
all variants converge towards ZSTD_initCStream_stage2()
 2017-04-10 22:24:02 -07:00
Yann Collet
4b987ad8ce Introduce ZSTD_initCStream_internal() 
This is now the regroup point for ZSTD_initCStream*() functions

ZSTD_initCStream_advanced() now properly checks for parameters validity.

Also : added <assert.h> usage inside zstd_compress.c
Needs ZSTD_DEBUG=1 macro to be triggered.
Will be triggered by default from `tests` directory
 2017-04-10 17:50:44 -07:00
Yann Collet
0181fef545 ensure cctx internal buffer is correctly sized in case of memory error 2017-04-06 01:25:26 -07:00
Yann Collet
36c2a03757 updated comments for ZSTD_resetCStream() 2017-04-05 22:06:21 -07:00
Yann Collet
003a244324 DStream : ensure correct size of internal buffers in case of error 2017-04-05 15:28:56 -07:00
Yann Collet
02d37aa1c1 ensure correct size of internal buffers in case of error 2017-04-05 14:53:51 -07:00
Nick Terrell
405d2a1027 Explicitly convert scratchBuffer to unsigned* 2017-04-04 16:35:31 -07:00
Nick Terrell
16a739cab0 Switch call of FSE_count() to FSE_count_wksp() 2017-04-04 16:17:21 -07:00
Yann Collet
7cf78f1be7 Protects ZSTD_compressBegin_usingCDict() vs NULL cdict dereference 
Will issue an error (GENERIC) is cdict==NULL
 2017-04-04 12:38:14 -07:00
Nick Terrell
26b046a7c4 Remove unnecessary dictID store 2017-04-03 21:46:28 -07:00
Nick Terrell
39a6cc5172 Make ZSTD_compress_usingCDict() respect contentSizeFlag 2017-04-03 21:09:55 -07:00
Nick Terrell
62ecad3819 Fix ZSTD_initCStream_usingCDict() to use dictionary 2017-04-03 21:05:59 -07:00
Yann Collet
30c7698970 optimize ZSTDMT_compress() memory usage 
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).

Also : fix ZSTD_compressBound() for small srcSize
 2017-03-31 18:27:03 -07:00
Yann Collet
3f75d52527 Changed ZSTD_compressBound() 
required so that if Total = A+B
compressBound(Total) <= compressBound(A) + compressBound(B)
under condition of a minimum size for A and B

Will help for ZSTDMT_compress() memory allocation
 2017-03-31 17:11:38 -07:00
Yann Collet
7b70a1969e Merge branch 'dev' into zstdmt 2017-03-31 16:22:33 -07:00
Yann Collet
53203e7c38 Merge pull request #640 from facebook/memAccess 
Changed memory strategy to __packed for gcc
 2017-03-31 15:49:12 -07:00
Yann Collet
eea7858e2b fixed minor warnings in debug code 2017-03-30 16:47:19 -07:00
Yann Collet
34cc487d05 overlap at full windowSize for max compression level 
as it provides max compression ratio
 2017-03-30 16:23:22 -07:00
Yann Collet
458e955c23 improved ZSTDMT_compress() 
Use a bit more threads by default.
Uses overlap segments to boost compression ratio (like the streaming variant)
 2017-03-30 15:51:58 -07:00
Yann Collet
6476c51b86 Merge pull request #637 from facebook/zstdmt 
Zstdmt
 2017-03-30 14:18:37 -07:00
Yann Collet
274f59919d Changed memory strategy to __packed for gcc 
Method 1 __packed is always as good or better than memcpy().
But it's not portable, as it depends on compiler extension.

For gcc, __pakced directive works fine.
Furthermore, gcc has serious performance issues with memcpy() on ARM 32 bits.
See #620
 2017-03-30 12:52:14 -07:00
Nick Terrell
5152fb2cb2 Convert all tabs to spaces 2017-03-29 18:51:58 -07:00
Yann Collet
ca5a8bbe36 re-added patch ... 2017-03-29 17:15:27 -07:00
Yann Collet
2e2e78de47 removed unnecessary restriction on minmatchLength 
it's now transparently translated to nearest value when unsupported
(7->6) (3->4)
 2017-03-29 16:02:47 -07:00
Yann Collet
26769d88bc Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-29 15:21:30 -07:00
Yann Collet
933ce4a1dd fix : minmatch 7 conversion 
minmatch 7 now converted to minmatch 6 for strategies which do not support 7
Used to folded into "default", which applied minmatch 4
 2017-03-29 14:35:38 -07:00
Sean Purcell
4708394bdd Remove extra 'F' from skippable magic mask 2017-03-29 11:46:57 -07:00
Yann Collet
4cf0093571 restored bonus rule 2017-03-26 14:51:00 -07:00
Yann Collet
69017bf253 Merge branch 'dev' into LegacyDictBuilder 2017-03-26 14:39:13 -07:00
Yann Collet
582760818f minor refactor 
add const
changed if for easier to add new conditions
 2017-03-26 03:04:56 -07:00
Yann Collet
858f72eeb8 fixed dictBuilder issue 
dictionary loading would fail during entropy analysis
 2017-03-26 02:50:00 -07:00
Yann Collet
ecee9f2ef8 fixed conversion warnings 2017-03-26 00:59:14 -07:00
Yann Collet
0246d5c531 Merge pull request #630 from facebook/advancedCliCommands 
changed advanced commands --maxdict= and --dictID=
 2017-03-26 00:13:35 -07:00
Yann Collet
4c41d37fcc changed test for new syntax 
--dictID= and --maxdict=
 2017-03-24 18:36:56 -07:00
Yann Collet
d41f707e88 minor improvement : remove duplicates with 1 char prefix difference 2017-03-24 17:56:45 -07:00
Yann Collet
b364caf455 Merge pull request #628 from facebook/dictBuilder_limits 
Ensure all limits derived from same constants
 2017-03-24 17:54:42 -07:00
Yann Collet
2238870eb6 Merge pull request #625 from facebook/loadCDict 
limited CDict acceptation criteria to be the same as DDict
 2017-03-24 16:06:20 -07:00
Yann Collet
96aa3019b2 changed advanced commands --maxdict= and --dictID= 
now works with the `=` variant, which is the recommended one.
Old variant `--dictID #` still works, for compatibility with existing scripts.
Long term objective is to remove the old variant..
 2017-03-24 16:04:29 -07:00
Yann Collet
9da3b215ec Ensure all limits derived from same constants 
Now uses ZDICT_DICTSIZE_MIN and ZDICT_CONTENTSIZE_MIN
from zdict.h.

Also : reduced values to 256 and 128 respectively
 2017-03-24 15:02:09 -07:00
Yann Collet
ebe9963cf6 Merge pull request #626 from facebook/stricterDictBuilder 
dictBuilder fails to create dictionary on certain input
 2017-03-24 14:27:28 -07:00
Yann Collet
16a0b10781 fixed ZSTD_loadZstdDictionary() 
forgot to add the dictionary content
(tests were not failing, just compressing less).

Also : added size protections when adding dict content
since hc/bt table filling would fail if size < 8
 2017-03-24 12:46:46 -07:00
Yann Collet
23776ce290 fixed ERROR_GENERIC on dstSize_tooSmall 
required by users which depends on this error code to size dest buffer
 2017-03-23 17:59:50 -07:00
Yann Collet
f332ece468 dictBuilder fails to create dictionary on certain input 
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
 2017-03-23 16:24:02 -07:00
Yann Collet
bea78e8fc2 limited CDict acceptation criteria to be the same as DDict 2017-03-23 15:46:06 -07:00
Sean Purcell
042ba122ae Change g_displayLevel to int and fix DISPLAYUPDATE flush 2017-03-23 11:21:59 -07:00
Nick Terrell
eaf69b07f0 Zero pointers after freeing 2017-03-21 13:20:59 -07:00
Yann Collet
f3dfcdccd1 bump version number 2017-03-21 12:18:28 -07:00
Przemyslaw Skibinski
8086d623ca updated build of Windows packages 2017-03-18 11:19:09 +01:00
Yann Collet
7e35b352c6 Merge pull request #602 from iburinoc/doc 
Add functions missing from manual, and fix parameter alignment
 2017-03-14 14:08:41 -07:00
Sean Purcell
dec2b96536 Add functions missing from manual, and fix parameter alignment 2017-03-14 11:24:09 -07:00
Sean Purcell
9830aeeea6 Fix legacy support=0 case and accidental double include of version headers 2017-03-13 17:19:37 -07:00
Sean Purcell
120df494e9 Update builds to not support legacy v01-v03 2017-03-13 14:44:08 -07:00
Sean Purcell
334cb34edb ZSTD_LEGACY_SUPPORT defines lowest supported version 2017-03-13 14:32:30 -07:00
Sean Purcell
784082f49c Change gotoDict type to uPtrDiff 2017-03-10 10:34:45 -08:00
Sean Purcell
8fe5c6862c Fix undefined behaviour in decompressor 2017-03-10 10:17:42 -08:00
Nick Terrell
f35ef5c8e9 Whitespace only: tabs to spaces 2017-03-09 12:51:33 -08:00
Nick Terrell
eeb31eed39 s/ZSTD_btopt2/ZSTD_btultra/g 2017-03-09 11:44:25 -08:00
Nick Terrell
e65aab8e0f Remove 'mem.h' dependency from ZSTD_WINDOWLOG_MAX 2017-03-08 15:40:13 -08:00
Yann Collet
a41a4ed39a Merge pull request #594 from terrelln/bugs 
Small fixes
 2017-03-08 14:56:07 -08:00
Nick Terrell
81512e9ebe Avoid '#define inline /* ... */' 
Take definition of `FORCE_INLINE` from `zstd_internal.h`.
 2017-03-08 14:00:21 -08:00
Nick Terrell
e06c303475 Fix ZSTD_sizeof_CStream() 2017-03-08 13:45:10 -08:00
Sean Purcell
881abe44f1 Reduce point at which we reduce offsets to protect against UB 2017-03-07 16:58:08 -08:00
Sean Purcell
3437bf2feb Add build targets to the Makefile, and update CircleCI tests 2017-03-06 15:05:02 -08:00
Yann Collet
8b1d004031 added -Wformat-security flag, as recommended by @pixelb 2017-03-05 21:17:32 -08:00
Yann Collet
1f2c95c5f3 minor code refactor in HUF module 2017-03-05 21:07:20 -08:00
Yann Collet
5d801278dc Merge pull request #586 from terrelln/repeat-heuristic 
Always check Huffman tables for ZSTD_lazy+
 2017-03-03 19:38:56 -08:00
Nick Terrell
54c4babd8f Always check Huffman tables for ZSTD_lazy+ 
The compressor always reuses the existing Huffman table if the literals
size is at most 1 KiB. If the compression strategy is `ZSTD_lazy` or
stronger always check to see if reusing the previous table or creating
a new table is better.

This doesn't yet weigh in decompression speed. I don't want to add any
heuristics there until I have real data to work with to ensure that the
heuristic works for at least one use case, preferably more.
 2017-03-03 16:49:38 -08:00
Yann Collet
1af570bd05 Merge pull request #585 from terrelln/cover-leak 
Fix COVER_optimizeTrainFromBuffer() resource leaks
 2017-03-02 20:46:35 -08:00
Yann Collet
f44b55c18d Merge pull request #584 from terrelln/huff-repeat 
Allow compressor to repeat Huffman tables
 2017-03-02 17:20:11 -08:00
Yann Collet
fe5d27062e disable prefetch-decode for 32-bits target 
This decoder variant is detrimental to x86 architecture
likely due to register pressure.

Note that the variant is disabled for all 32-bits targets.
It's unclear if it would help for different architectures,
such as ARM, MIPS or PowerPC.
 2017-03-02 17:09:21 -08:00
Nick Terrell
d051cd5b43 Use workspace for count and CTable 2017-03-02 16:38:07 -08:00
Nick Terrell
976e325b2e Fix COVER_optimizeTrainFromBuffer() resource leaks 
Thanks to @nemequ for reporting the resource leaks.
 2017-03-02 15:54:39 -08:00
Sean Purcell
553f67e0c1 Remove 'generic' inline strategy 
Seems to avoid performance loss for compression.
Same strategy tested on decompression side, did not appear to improve
speed.
 2017-03-02 15:18:13 -08:00
Sean Purcell
3d95925a59 Merge remote-tracking branch 'origin/dev' into m32 2017-03-02 15:17:56 -08:00
Nick Terrell
a419777eb1 Allow compressor to repeat Huffman tables 
* Compressor saves most recently used Huffman table and reuses it
  if it produces better results.
* I attempted to preserve CPU usage profile.
  I intentionally left all of the existing heuristics in place.
  There is only a speed difference on the second block and later.
  When compressing large enough blocks (say >= 4 KiB) there is
  no significant difference in compression speed.
  Dictionary compression of one block is the same speed for blocks
  with literals <= 1 KiB, and after that the difference is not
  very significant.
* In the synthetic data, with blocks 10 KB or smaller, most blocks
  can't use repeated tables because the previous block did not
  contain a symbol that the current block contains.
  Once blocks are about 12 KB or more, most previous blocks have
  valid Huffman tables for the current block, and the compression
  ratio and decompression speed jumped.
* In silesia blocks as small as 4KB can frequently reuse the
  previous Huffman table (85%), but it isn't as profitable, and
  the previous Huffman table only gets used about 3% of the time.
* Microbenchmarks show that `HUF_validateCTable()` takes ~55 ns
  and `HUF_estimateCompressedSize()` takes ~35 ns.
  They are decently well optimized, the first versions took 90 ns
  and 120 ns respectively. `HUF_validateCTable()` could be twice as
  fast, if we cast the `HUF_CElt*` to a `U32*` and compare to 0.
  However, `U32` has an alignment of 4 instead of 2, so I think that
  might be undefined behavior.
* I've ran `zstreamtest` compiled normally, with UASAN and with MSAN
  for 4 hours each.

The worst case for the speed difference is a bunch of small blocks
in the same frame. I modified `bench.c` to compress the input in a
single frame but with blocks of the given block size, set by `-B`.
Benchmarks on level 1:

|  Program  | Block size |   Corpus  | Ratio | Compression MB/s | Decompression MB/s |
|-----------|------------|-----------|-------|------------------|--------------------|
| zstd.base |        256 | synthetic | 2.364 |            110.0 |              297.0 |
|      zstd |        256 | synthetic | 2.367 |            108.9 |              297.0 |
| zstd.base |        256 | silesia   | 2.204 |             93.8 |              415.7 |
|      zstd |        256 | silesia   | 2.204 |             93.4 |              415.7 |
| zstd.base |        512 | synthetic | 2.594 |            144.2 |              420.0 |
|      zstd |        512 | synthetic | 2.599 |            141.5 |              425.7 |
| zstd.base |        512 | silesia   | 2.358 |            118.4 |              432.6 |
|      zstd |        512 | silesia   | 2.358 |            119.8 |              432.6 |
| zstd.base |       1024 | synthetic | 2.790 |            192.3 |              594.1 |
|      zstd |       1024 | synthetic | 2.794 |            192.3 |              600.0 |
| zstd.base |       1024 | silesia   | 2.524 |            148.2 |              464.2 |
|      zstd |       1024 | silesia   | 2.525 |            148.2 |              467.6 |
| zstd.base |       4096 | synthetic | 3.023 |            300.0 |             1000.0 |
|      zstd |       4096 | synthetic | 3.024 |            300.0 |             1010.1 |
| zstd.base |       4096 | silesia   | 2.779 |            223.1 |              623.5 |
|      zstd |       4096 | silesia   | 2.779 |            223.1 |              636.0 |
| zstd.base |      16384 | synthetic | 3.131 |            350.0 |             1150.1 |
|      zstd |      16384 | synthetic | 3.152 |            350.0 |             1630.3 |
| zstd.base |      16384 | silesia   | 2.871 |            296.5 |              883.3 |
|      zstd |      16384 | silesia   | 2.872 |            294.4 |              898.3 |
 2017-03-02 13:27:52 -08:00
Yann Collet
fdb0fd34b3 Merge pull request #583 from terrelln/set-dictid 
Set dictID to 0 for content only dictionaries
 2017-03-02 13:15:31 -08:00
Nick Terrell
3475b9b431 Set dictID to 0 for content only dictionaries 2017-03-02 12:33:02 -08:00
Sean Purcell
d44703d145 Offsets >= 32MB in 32-bits mode 2017-03-01 16:27:56 -08:00
Yann Collet
76f0494089 xxhash can be included twice in any order 
Previously,

followed by :

would fail to include the static definitions,
because the second include was simply skipped by guard macro.

Now it works as intended :
the missing static part is included during the second include.
 2017-03-01 13:29:29 -08:00
Yann Collet
4bcc69b761 solves warnings when compiling with global XXH_STATIC_LINKING_ONLY 
XXH_STATIC_LINKING_ONLY protection macro is intended to be triggered just before the include.
The main idea is to keep this setting local :
user module shall explicitly understand and accept the static linking restriction
which becomes transparent when triggering the macro at project level.
Global definition also triggers redefinition warnings for user modules which do locally define the macro.

This new version compiles lib and cli without warning when the macro is set globally.
That's not a scenario to be recommended, since it trades a local effect for a global one,
but it was easy enough to provide from zstd side.
 2017-03-01 11:33:25 -08:00
Yann Collet
31432cc57d Merge pull request #579 from iburinoc/multiframe 
Check to ensure ddict isn't null before dereference
 2017-03-01 11:02:04 -08:00
Sean Purcell
a81d4fee58 Check to ensure ddict isn't null before dereference 2017-02-28 15:28:29 -08:00
Yann Collet
22d79762ef fixed multi frames 2017-02-28 02:12:42 -08:00
Yann Collet
a33ae64204 fixed decoding skippable frames 2017-02-28 01:15:28 -08:00
Yann Collet
d1760113ec Improved speed of ZSTD_decompressStream() 
When ZSTD_decompressStream() detects
that there is enough space in dst
to complete decompression in a single pass,
delegates to ZSTD_decompress(),
for an extra ~5% speed boost
 2017-02-28 00:14:28 -08:00
Yann Collet
a81c2e7e44 Merge pull request #573 from facebook/ddict 
Improved DDict memory usage
 2017-02-27 20:54:42 -08:00
Yann Collet
dccd6b6f65 cli : fix : --rm is silent when input is stdin 
previously, app would produce an error message, and stop.
 2017-02-27 15:57:50 -08:00
Yann Collet
0b9b894b2d reduced ZSTD_DDict memory usage 
saved 128 KB
 2017-02-27 00:27:30 -08:00
Yann Collet
bd7fa21deb added ZSTD_refDDict() 
Now DDict does no longer depends on DCtx duplication
 2017-02-26 14:43:07 -08:00
Yann Collet
d73eebc00f loadEntropy works on new ZSTD_entropy_t type 2017-02-26 10:16:42 -08:00
Yann Collet
8629f0e41f created entropy structure type 2017-02-25 18:33:31 -08:00
Yann Collet
8dff956dbf Added DDict unit test in fuzzer 
also : slightly modified loadEntropy :
know src must points at start of dictionary
 2017-02-25 10:11:15 -08:00
Yann Collet
14312d833e zstdmt : fix : loading prefix from previous segments 
There used to be a (very small) chance that
loading prefix from previous segment
would be confused with a real zstd dictionary.
For that to happen, the prefix needs to start
with the same value as dictionary magic.
That's 1 chance in 4 billions if all values have equal probability.
But in fact, since some values are more common (0x00000000 for example)
others are less common, and dictionary magic was selected to be one of them,
so probabilities are likely even lower.

Anyway, this risk is no down to zero
by adding a new CCtx parameter : ZSTD_p_forceRawDict

Current parameter policy : the parameter "stick" to its CCtx,
so any dictionary loading after ZSTD_p_forceRawDict is set
will be loaded in "raw" ("content only") mode,
even if CCtx is re-used multiple times with multiple different dictionary.
It's up to the user to reset this value differently if it needs so.
 2017-02-23 23:42:12 -08:00
Yann Collet
831b4890ce minor tests/Makefile refactoring 
and update of zstd_manual,html
 2017-02-23 23:09:10 -08:00
Yann Collet
cce8d8ba2b Merge pull request #560 from iburinoc/findcompressedsize 
Change name to to findFrameCompressedSize and add skippable support
 2017-02-23 13:39:23 -08:00
Sean Purcell
83038d236a Fix bug in FSE distribution normalization 2017-02-22 13:52:48 -08:00
Sean Purcell
64417cd2ff Describe ambiguity around skippable frames 2017-02-22 13:29:01 -08:00
Sean Purcell
9757cc811b Update comment 2017-02-22 12:28:21 -08:00
Sean Purcell
9050e1925e Change name to to findFrameCompressedSize and add skippable support 2017-02-22 12:12:34 -08:00
Przemyslaw Skibinski
d8114e5802 zstd_compress.c: fix memory leaks 2017-02-21 18:59:56 +01:00
Anders Oleson
517577bf53 spelling fixes in comments 
i.e. occurred labeled Huffman
 2017-02-20 12:08:59 -08:00
Sean Purcell
6b010dec80 execSequence copies up to 2*WILDCOPY_OVERLENGTH extra 2017-02-16 12:05:40 -08:00
Sean Purcell
887eaa9e21 Fix wildcopy overwriting data still in window 2017-02-15 16:43:45 -08:00
Yann Collet
2252d29a5a Merge branch 'dev' of github.com:facebook/zstd into dev 2017-02-15 12:00:50 -08:00
Yann Collet
4596037042 updated fse version 
feature minor refactoring (removing FSE_abs())
also : fix a few minor issues recently introduced in examples
 2017-02-15 12:00:03 -08:00
Yann Collet
44f82d781f Merge pull request #545 from terrelln/force-window 
[zstdmt] Fix MSAN failure with ZSTD_p_forceWindow
 2017-02-15 10:20:15 -08:00
Yann Collet
f0b9a8dddb Merge pull request #547 from inikep/dev11 
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
 2017-02-14 12:29:00 -08:00
Yann Collet
9696bfc2ad Merge pull request #544 from ds77/avoid-empty 
Portable way to avoid empty unit warning in threading.c
 2017-02-14 00:54:55 -08:00
Przemyslaw Skibinski
b876b96ce1 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-14 09:26:03 +01:00
Nick Terrell
ecf90ca24b [zstdmt] Fix MSAN failure with ZSTD_p_forceWindow 
Reproduction steps:

```
make zstreamtest CC=clang CFLAGS="-O3 -g -fsanitize=memory -fsanitize-memory-track-origins"
./zstreamtest -vv -t4178 -i4178 -s4531
```

How to get to the error in gdb (may be a more efficient way):

* 2 breaks at zstd_compress.c:2418  -- in ZSTD_compressContinue_internal()
* 2 breaks at zstd_compress.c:2276  -- in ZSTD_compressBlock_internal()
* 1 break at zstd_compress.c:1547

Why the error occurred:

When `zc->forceWindow == 1`, after calling `ZSTD_loadDictionaryContent()` we
have `zc->loadedDictEnd == zc->nextToUpdate == 0`. But, we've really loaded up
to `iend` into the dictionary. Then in `ZSTD_compressBlock_internal()` we see
that `current > zc->nextToUpdate + 384`, so we load the last 192 bytes a second
time. In this case the bytes we are loading are a block of all 0s, starting in
the previous block. So when we are loading the last 192 bytes, we find a `match`
in the future, 183 bytes beyond `ip`. Since the block is all 0s, the match
extends to the end of the block. But in `ZSTD_count()` we only check that
`pIn < pInLoopLimit`, but since `pMatch > pIn`, `pMatch` eventually points past
the end of the buffer, causing the MSAN failure.

The fix:

The line changed sets sets `zc->nextToUpdate` to the end of the dictionary.
This is the behavior that existed before `ZSTD_p_forceWindow` was introduced.
This fixes the exposing test case. Since the code doesn't fail without
`zc->forceWindow`, it makes sense that this works. I've run the command
`./zstreamtest -T2mn` 64 times without failures. CI should also verify nothing
obvious broke.
 2017-02-13 19:11:22 -08:00
Yann Collet
58af614ef2 push version and NEWS to v1.1.4 2017-02-13 18:32:44 -08:00
ds77
08e6a88a97 avoid empty translation unit warning without #pragma 2017-02-14 00:46:47 +01:00
Przemyslaw Skibinski
09c8e5390d __builtin_bswap requires gcc 4.3+ 2017-02-13 12:45:53 +01:00
Sean Purcell
d7bfcac18a Expose frameSrcSize to experimental API 2017-02-10 11:55:44 -08:00
Sean Purcell
5069b6c2c3 Merge branch 'dev' into multiframe 2017-02-10 10:08:55 -08:00
Yann Collet
bbba42acd1 Merge pull request #537 from terrelln/small-bugs 
Fix small bugs
 2017-02-10 04:35:43 -08:00
Yann Collet
a28c34cb7a Merge pull request #538 from iburinoc/errorstring 
Fix ZSTD_getErrorString and add tests
 2017-02-10 03:59:56 -08:00
Sean Purcell
269b2cd3d8 Documentation updates 2017-02-09 13:25:30 -08:00
Sean Purcell
2db7249265 Make pledgedSrcSize meaning clear for other functions 
- Added tests
- Moved new size functions to static link only
 2017-02-09 11:49:58 -08:00
Nick Terrell
545987996a Fix deprecation warnings for clang with C++14 2017-02-08 17:38:17 -08:00
Sean Purcell
e0b3265e87 Fix ZSTD_getErrorString and add tests 2017-02-08 17:28:49 -08:00
Sean Purcell
0f5c95af44 Disambiguate pledgedSrcSize == 0 
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
 2017-02-08 15:12:46 -08:00
Sean Purcell
ba2ad9f25c ZSTD_decompress now handles multiple frames 2017-02-08 14:50:10 -08:00
Sean Purcell
4e709712e1 Decompressed size functions now handle multiframes and distinguish cases 
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
 2017-02-08 14:50:10 -08:00
Przemyslaw Skibinski
cdf5a7bd9f Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-08 13:49:35 +01:00
Nick Terrell
71c5263c00 Attribute cover dictionary code 2017-02-07 11:35:07 -08:00
Przemyslaw Skibinski
7060aee8c2 platform.h added to build_package.bat 2017-02-06 19:43:13 +01:00
Yann Collet
b54e235bf3 fixed Mac OS-X specific directory in $(RM) list 
these directories are now removed with -r command
 2017-02-05 10:22:58 -08:00
Yann Collet
c2a4632789 release builds use less debug symbols and warnings 
release build are triggered through either `make`,
or their specific target `make zstd-release` and `make lib-release`.
 2017-02-02 20:54:41 -08:00
Yann Collet
48bed91606 Merge pull request #527 from facebook/zstdmt 
zstdmt refinements
 2017-01-31 16:36:46 -08:00
Yann Collet
b2e1b3d670 fixed overlapLog==0 => no overlap 2017-01-30 14:54:46 -08:00
Yann Collet
3672d06d06 zstdmt : section size is set to be a minimum of overlapSize 
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
 2017-01-30 13:35:45 -08:00
Yann Collet
88df1aed61 changed advanced parameter overlapLog 
Follows a positive logic (increasing value => increasing overlap)
which is easier to use
 2017-01-30 11:00:00 -08:00
Yann Collet
b5fd15ccb2 fixed : legacy decoders v04 and v05 2017-01-30 10:45:58 -08:00
Yann Collet
cc3d1bc262 Merge pull request #525 from terrelln/covermt 
Multithreaded COVER dictionary training
 2017-01-30 10:15:33 -08:00
Nick Terrell
43474313f8 Fix documentation about memory usage 2017-01-27 18:43:05 -08:00
Nick Terrell
b42dd27ef5 Add include guards and extern C 2017-01-27 16:00:19 -08:00
Yann Collet
f6d4a786fc reduced zstdmt latency when using small custom section sizes with high compression levels 
Previous version was requiring a fairly large initial amount of input data
before starting to create compression jobs.
This new version starts the process much sooner.
 2017-01-27 15:55:30 -08:00
Nick Terrell
c43c27127f Merge branch 'dev' into buck 
* dev:
  updated NEWS
  fixed MSAN warnings in legacy decoders
  Fix cmake build
  updated NEWS
  Edits as per comments, and change wildcard 'X' to '?'
  Fix Visual Studios project
  Fix pool.c threading.h import
  Fix zstdmt_compress.h include
  Fixed commented issues
  Updated format specification to be easier to understand
  improved #232 fix
  Fixed https://github.com/facebook/zstd/issues/232
  .travis.yml: different tests for "master" branch
  .travis.yml: optimized order of short tests
  .travis.yml: test jobs 12-15
  JOB_NUMBER -eq 9
  improved ZSTD_compressBlock_opt_extDict_generic
 2017-01-27 12:05:48 -08:00
Nick Terrell
2fe9126591 Add multithread support to COVER 2017-01-27 11:56:02 -08:00
Yann Collet
609c123a01 Merge pull request #522 from terrelln/benchmt 
Fix some includes
 2017-01-27 11:40:25 -08:00
Yann Collet
cafdd31a38 fixed MSAN warnings in legacy decoders 
In some extraordinary circumstances,
*Length field can be generated from reading a partially uninitialized memory segment.
Data is correctly identified as corrupted later on,
but the read taints some later pointer arithmetic operation.
 2017-01-27 10:44:03 -08:00
Nick Terrell
9c018cc140 Add BUCK files for Nuclide support 2017-01-27 10:43:12 -08:00
Przemyslaw Skibinski
29157320fb improved ZSTD_compressBlock_opt_extDict_generic 2017-01-27 10:43:02 -08:00
Nick Terrell
e628eaf87a Fix pool.c threading.h import 2017-01-26 15:29:10 -08:00
Yann Collet
717c65d690 Merge pull request #519 from inikep/dev11 
Dev11
 2017-01-26 14:23:44 -08:00
Yann Collet
ef33d00532 fixed : ZSTD_setCCtxParameter() properly exposed in DLL 2017-01-26 12:24:21 -08:00
Yann Collet
4a62f79ec9 fixed clang documentation warning 2017-01-26 09:16:56 -08:00
Yann Collet
8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet
06e7697f96 added test of new parameter ZSTD_p_forceWindow 2017-01-25 16:39:03 -08:00
Yann Collet
bb0027405a fixed zstdmt corruption issue when enabling overlapped sections 
see Asana board for detailed explanation on why and how to fix it
 2017-01-25 16:25:38 -08:00
Yann Collet
943cff9c37 fixed zstdmt cli freeze issue with large nb of threads 
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
 2017-01-25 12:35:19 -08:00
Yann Collet
dc8dae596a overlapped section, for improved compression 
Sections 2+ read a bit of data from previous section
in order to improve compression ratio.
This also costs some CPU, to reference read data.

Read data is currently fixed to window>>3 size
 2017-01-24 22:32:12 -08:00
Yann Collet
f14a669054 refactor job creation 
code shared accross ZSTDMT_{compress,flush,end}Stream(),
for easier maintenance
 2017-01-24 17:41:49 -08:00
Yann Collet
512cbe8c10 zstdmt cli and API allow selection of section sizes 
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.

It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()

The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
 2017-01-24 17:08:53 -08:00
Yann Collet
3488a4a473 ZSTDMT now supports frame checksum 2017-01-24 11:48:40 -08:00
Przemyslaw Skibinski
96f152f708 improved ZSTD_compressBlock_opt_extDict_generic 2017-01-24 13:18:50 +01:00
Yann Collet
94364bf87a refactor ZSTDMT streaming flush code 
now shared by both ZSTDMT_compressStream() and ZSTDMT_flushStream()
 2017-01-23 11:50:44 -08:00
Yann Collet
1cbf251e43 ZSTDMT streaming : fall back to (regular) single thread mode 
when nbThreads==1
 2017-01-23 01:43:58 -08:00
Yann Collet
84581ff8d7 ZSTDMT_compressCCtx : fallback to single-thread mode when nbChunks==1 2017-01-23 01:20:27 -08:00
Yann Collet
1a2547f654 ZSTDMT_compressStream() becomes blocking when required to ensure forward progresses 
In some (rare) cases, job list could be blocked by a first job still being processed,
while all following ones are completed, waiting to be flushed.
In such case, the current job-table implementation is unable to accept new job.
As a consequence, a call to ZSTDMT_compressStream() can be useless (nothing read, nothing flushed),
with the risk to trigger a busy-wait on the caller side
(needlessly loop over ZSTDMT_compressStream() ).

In such a case, ZSTDMT_compressStream() will block until the first job is completed and ready to flush.
It ensures some forward progress by guaranteeing it will flush at least a part of the completed job.
Energy-wasting busy-wait is avoided.
 2017-01-22 23:49:52 -08:00
Yann Collet
c593348722 ZSTDMT_initCStream_usingDict() can outlive dict 
Like ZSTD_initCStream_usingDict(),
ZSTDMT_initCStream_usingDict() now keep a copy of dict internally.
This way, dict can be released :
it does not longer have to outlive all future compression sessions.
 2017-01-22 16:44:15 -08:00
Yann Collet
9d6f7637ec protected (mutex) read to jobCompleted, as suggested by @terrelln 2017-01-21 22:14:08 -08:00
Yann Collet
0cf74fa957 optimized pool allocation by 1 slot 2017-01-21 22:06:49 -08:00
Yann Collet
6ed29a8f44 minor : tab to spaces 2017-01-21 21:56:36 -08:00
Yann Collet
317604e0ad fixed : compilation of zstreamtest in dll mode 2017-01-20 17:18:41 -08:00
Yann Collet
d7e3cb58c5 Resolved merge conflict dev+zstdmt 2017-01-20 16:44:50 -08:00
cyan4973
2e3b659ae1 fixed minor warnings (Visual, conversion, doxygen) 2017-01-20 14:43:09 -08:00
cyan4973
5fba09fa41 updated util's time for Windows compatibility 
Correctly measures time on Posix systems when running with
Multi-threading

Todo : check Windows measurement under multi-threading
 2017-01-20 12:57:31 -08:00
Yann Collet
b459aad5b4 renamed savedRep into repToConfirm 2017-01-19 17:33:37 -08:00
Yann Collet
500014af49 zstd cli can now compress using multi-threading 
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads()  (fileio)
 2017-01-19 17:04:28 -08:00
Yann Collet
19d670ba9d Added ZSTDMT_initCStream_advanced() variant 
Correctly compress with custom params and dictionary
Added relevant fuzzer test in zstreamtest

Also :
new macro ZSTDMT_SECTION_LOGSIZE_MIN, which sets a minimum size for a full job
(note : a flush() command can still generate a partial job anytime)
 2017-01-19 15:32:07 -08:00
Yann Collet
0f984d94c4 changed MT enabling macro to ZSTD_MULTITHREAD 2017-01-19 14:05:07 -08:00
Yann Collet
736788f8e8 added streaming fuzzer tests for MT API 
Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize
which is possible when many flushes are issued
while there is not enough dst buffer to flush completed ones.
 2017-01-19 12:15:29 -08:00
Yann Collet
32dfae6f98 fixed Multi-threaded compression 
MT compression generates a single frame.
Multi-threading operates by breaking the frames into independent sections.
But from a decoder perspective, there is no difference :
it's just a suite of blocks.

Problem is, decoder preserves repCodes from previous block to start decoding next block.
This is also valid between sections, since they are no different than changing block.

Previous version would incorrectly initialize repcodes to their default value at the beginning of each section.
When using them, there was a mismatch between encoder (default values) and decoder (values from previous block).

This change ensures that repcodes won't be used at the beginning of a new section.
It works by setting them to 0.
This only works with regular (single segment) variants : extDict variants will fail !
Fortunately, sections beyond the 1st one belong to this category.

To be checked : btopt strategy.
This change was only validated from fast to btlazy2 strategies.
 2017-01-19 10:32:55 -08:00
Yann Collet
37226c1e9f Simplified compressChunk job 
minor refactoring : compression done in a single call on first chunk
Avoid a mutable hSize variable and eventual recombination to cSize at the end
 2017-01-19 10:18:17 -08:00
Yann Collet
dab5ea93f2 Merge pull request #515 from iburinoc/emptydict 
Don't create dict in streaming apis if dictSize == 0
 2017-01-19 09:02:42 -08:00
Yann Collet
6073b3e6b8 ZSTDMT_endStream : nullify input buffer after flush 
There will be no more input after ZSTDMT_endStream invocation :
only flush/end is allowed (to fully collect compressed result).
 2017-01-18 15:32:38 -08:00
Yann Collet
3a01c46b26 ZSTDMT_initCStream() supports restart from invalid state 
ZSTDMT_initCStream() will correcly scrub for resources
when it detects that previous compression was not properly finished.
 2017-01-18 15:18:17 -08:00
Yann Collet
4885f591b3 trap compression errors, collect back resources from workers 2017-01-18 14:11:37 -08:00
Sean Purcell
0b5370ae38 Prefix notes with /**< 2017-01-18 13:45:02 -08:00
Yann Collet
563ef8acf4 CCtxPool starts empty, as suggested by @terrelln 
Also : make zstdmt now a target from root
 2017-01-18 12:12:10 -08:00
Yann Collet
a6db7a7b9b fixed cmaketest 
(buffer_t){NULL,0} is not considered a constant.
{NULL,0} is.
 2017-01-18 11:57:34 -08:00
Yann Collet
0d6b8f65a9 ZSTDMT_free() scrubs potentially unfinished jobs to release their resources 
In some complex scenarios (free() without finishing compression),
it is possible that some resources are still into jobs
and not collected back into pools.
In which case, previous version of free() would miss them.
This would be equivalent to a leak.

New version ensures that it even foes after such resource.
It requires job consumers to properly mark resources as released,
by replacing entries by NULL after releasing back to the pool.

Obviously, it's not recommended to free() zstdmt context mid-term,
still that's now a supported scenario.

The same methodology is also used to ensure proper resource collection
after an error is detected.

Still to do :
- detect compression errors (not just allocation ones)
- properly manage resource when init() is called without finishing previous compression.
 2017-01-17 17:46:33 -08:00
Yann Collet
d0a1d45582 ZSTDMT_{flush,end}Stream() now block on next job completion when nothing to flush 
The main issue was to avoid a caller to continually loop on {flush,end}Stream()
when there was nothing ready to be flushed but still some compression work ongoing in a worker thread.
The continuous loop would have resulted in wasted energy.
The new version makes call to {flush,end}Stream blocking when there is nothing ready to be flushed.
Of course, if all worker threads have exhausted job, it will return zero (all flush completed).

Note : There are still some remaining issues to report error codes
and properly collect back resources into pools when an error is triggered.
 2017-01-17 16:15:18 -08:00
Yann Collet
a73c412932 completed ZSTDMT streaming compression 
Provides the baseline compression API :
size_t ZSTDMT_initCStream(ZSTDMT_CCtx* zcs, int compressionLevel);
size_t ZSTDMT_compressStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
size_t ZSTDMT_flushStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output);
size_t ZSTDMT_endStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output);

Not tested yet
 2017-01-17 15:31:16 -08:00
Sean Purcell
57d423c5df Don't create dict in streaming apis if dictSize == 0 2017-01-17 14:31:35 -08:00
Przemyslaw Skibinski
8a0bc30a2d Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-01-17 13:02:29 +01:00
Przemyslaw Skibinski
d72f4b6b7a added "Makefile is validated" 2017-01-17 12:40:06 +01:00
Gregory Szorc
7d6f478d15 Set dictionary ID in ZSTD_initCStream_usingCDict() 
When porting python-zstandard to use ZSTD_initCStream_usingCDict()
so compression dictionaries could be reused, an automated test
failed due to compressed content changing.

I tracked this down to ZSTD_initCStream_usingCDict() not
setting the dictID field of the ZSTD_CCtx attached to the
ZSTD_CStream instance.

I'm not 100% convinced this is the correct or full solution,
as I'm still seeing one automated test failing with this change.
 2017-01-14 17:44:54 -08:00
Yann Collet
5b726dbe4d fix gcc-arm warning "suggest braces around empty body" 2017-01-12 17:46:46 +01:00
Yann Collet
ad9f6bd123 zstdmt : fix : resources properly collected even when early fail 
In previous version, main function would return early when detecting a job error.
Late threads resources were therefore not collected back into pools.
New version just register the error, but continue the collecting process.
All buffers and context should be released back to pool before leaving main function.
 2017-01-12 03:06:35 +01:00
Sean Purcell
834ab50fa3 Fixed decompress_usingDict not propagating corrupted dictionary error 2017-01-11 17:31:34 -08:00
Yann Collet
b05c4828ea zstdmt : correctly check for cctx and buffer allocation 
Result from getBuffer and getCCtx could be NULL when allocation fails.
Now correctly checks : job creation stop and last job reports an allocation error.
releaseBuffer and releaseCCtx are now also compatible with NULL input.

Identified a new potential issue :
when early job fails, later jobs are not collected for resource retrieval.
 2017-01-12 02:01:28 +01:00
Yann Collet
107bcbbbc2 zstdmt : changed internal naming from frame to chunk 
Since the result of mt compression is a single frame,
changed naming, which implied the concatenation of multiple frames.

minor : ensures that content size is written in header
 2017-01-12 01:25:46 +01:00
Yann Collet
5eb749e734 ZSTDMT_compress() creates a single frame 
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()

As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().

This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
 2017-01-11 18:21:25 +01:00
Yann Collet
04cbc36499 minor refactor (release CCtx 1st) and comment clarification 2017-01-11 16:08:08 +01:00
Yann Collet
085179bb78 fixed ZSTDMT_createCCtx() : checked inner objects are properly created 2017-01-11 15:58:05 +01:00
Yann Collet
8ce1cc2bec improved ZSTD_createCCtxPool() cancellation 
use ZSTD_freeCCtxPool() to release the partially created pool.
avoids to duplicate logic.

Also : identified a new difficult corner case :
when freeing the Pool, all CCtx should be previously released back to the pool.
Otherwise, it means some CCtx are still in use.
There is currently no clear policy on what to do in such a case.
Note : it's supposed to never happen.
Since pool creation/usage is static, it has no external user,
which limits risks.
 2017-01-11 15:44:26 +01:00
Yann Collet
47557ba2b2 fixed ZSTDMT_createCCtxPool() when inner CCtx creation fails 2017-01-11 15:35:56 +01:00
Nick Terrell
8d984699db Document memory requirements for COVER algorithm 2017-01-09 18:20:10 -08:00
Nick Terrell
555e281637 Handle large input size in 32-bit mode correctly 2017-01-09 18:20:06 -08:00
Nick Terrell
3a1fefcf00 Simplify COVER parameters 2017-01-02 17:51:38 -08:00
Nick Terrell
96b39f65fa Add COVER dictionary builder 2017-01-02 13:22:51 -08:00
Yann Collet
6334b04d61 compile object files, for faster recompilation 2017-01-02 03:22:18 +01:00
Yann Collet
f1cb55192c fixed linux warnings 2017-01-02 01:11:55 +01:00
Yann Collet
0ec6a95ba1 minor fixes 2017-01-02 00:49:42 +01:00
Yann Collet
2ec635a162 use pthread_cond to send signals between threads 2017-01-01 17:31:33 +01:00
Nick Terrell
bb13387d7d Fix pool for threading.h 2016-12-31 19:10:47 -05:00
Nick Terrell
4204e03e77 Add threading.h condition variables 2016-12-31 19:10:29 -05:00
Yann Collet
3b9d434356 extended ZSTDMT code support for non-MT systems and WIN32 (preliminary) 2016-12-31 16:32:19 +01:00
Yann Collet
c8efc1c874 simplified Buffer Pool 2016-12-31 14:45:33 +01:00
Yann Collet
3b29dbd9e8 new zstdmt version using generic treadpool 2016-12-31 06:04:25 +01:00
Yann Collet
c6a6417458 bench correctly measures time for multi-threaded compression (posix only) 2016-12-31 03:31:26 +01:00
Yann Collet
f765a375a5 Merge pull request #504 from terrelln/thread-pool 
[zstdmt] Add thread pool
 2016-12-30 15:31:49 +01:00
Nick Terrell
e777a5be6b Add a thread pool for ZSTDMT and COVER 2016-12-29 23:39:44 -08:00
Yann Collet
e70912c72b Changed : input divided into roughly equal parts. 
Debug : can measure time waiting for mutexes to unlock.
 2016-12-29 01:24:01 +01:00
Yann Collet
6c0ed9483a compression threads use ZSTD_compressCCtx() 2016-12-28 17:08:28 +01:00
Yann Collet
8d7432914f Merge pull request #503 from inikep/dev11 
Dev11
 2016-12-28 16:50:39 +01:00
Yann Collet
ce9e1452fd protect buffer pool with a mutex 2016-12-28 15:31:19 +01:00
Przemyslaw Skibinski
75f3a3a335 changed default PREFIX and MANDIR 2016-12-28 12:32:41 +01:00
Yann Collet
3d93f2fce7 first zstdmt sketch 2016-12-27 07:19:36 +01:00
Yann Collet
39c105c605 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-23 22:25:31 +01:00
Yann Collet
aca113f4f5 fixed ZSTD_sizeof_?Dict() 2016-12-23 22:25:03 +01:00
Yann Collet
c07d2e3a31 Merge pull request #499 from inikep/dev11 
improved *BSD and Solaris compatibility
 2016-12-23 21:32:03 +01:00
Przemyslaw Skibinski
63b0014b96 BSD: improved "make install" 2016-12-23 10:05:49 +01:00
Nick Terrell
1b5d4a7d53 ZDICT_finalizeDictionary() flipped comparison 2016-12-22 18:14:57 -08:00
Nick Terrell
bcbe77e994 ZDICT_finalizeDictionary() flipped comparison 
`ZDICT_finalizeDictionary()` had a flipped comparison.
I also allowed `dictBufferCapacity == dictContentSize`.
It might be the case that the user wants to fill the dictionary
completely up, and then let zstd take exactly the space it needs
for the entropy tables.
 2016-12-22 18:01:14 -08:00
Nick Terrell
78a0072d5a Fix failing test due to deprecation warning 2016-12-22 17:36:16 -08:00
Yann Collet
d76d1a9ef0 added ZDICT_finalizeDictionary() 2016-12-22 20:18:43 +01:00
Przemyslaw Skibinski
b999170311 Solaris: working "make -C lib install" 2016-12-22 20:14:37 +01:00
Yann Collet
ba75e9d8c3 fix : zlib wrapper compile in gnu90 mode 2016-12-21 19:57:18 +01:00
Yann Collet
0819abe3c1 added ZSTD_createDDict_byReference() body 2016-12-21 19:25:15 +01:00
Yann Collet
4e5eea61a8 added ZSTD_createDDict_byReference() 2016-12-21 16:44:35 +01:00
Yann Collet
8333106b8a Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-21 16:44:24 +01:00
Yann Collet
0d7e84899f Merge pull request #489 from inikep/v112 
improved detection of POSIX
 2016-12-21 16:42:46 +01:00
Yann Collet
1f57c2ed32 added : ZSTD_createCDict_byReference() 2016-12-21 16:20:11 +01:00
Przemyslaw Skibinski
2f6ccee6af platform.h: removed Compiler Options 2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski
5736db219e fix basic types redefinition 2016-12-21 09:26:00 +01:00
Nick Terrell
8157a4c3cc Fix dictionary loading bug causing an MSAN failure 
Offset rep codes must be in the range `[1, dictSize)`.
Fix dictionary loading to reject `0` as a offset rep code.
 2016-12-20 10:47:52 -08:00
Przemyslaw Skibinski
f8046b8e72 Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112 
# Conflicts:
#	appveyor.yml
 2016-12-19 08:20:26 +01:00
Yann Collet
d564faa3c6 fix : ZSTD_initCStream_srcSize() correctly set srcSize in frame header 2016-12-18 21:39:15 +01:00
Yann Collet
1496c3dc47 Fix : size estimation when some samples are very large 2016-12-18 11:58:23 +01:00
Yann Collet
d46ecb58a5 added dll compilation tests 2016-12-17 16:28:12 +01:00
Nick Terrell
8de46ab51a Export all API functions 2016-12-16 13:27:30 -08:00
Przemyslaw Skibinski
0a1caeef6d VS: fixed 32-bit DLL compilation 2016-12-15 12:12:46 +01:00
Przemyslaw Skibinski
4e10bd339d appveyor.yml: added tests of fullbench-dll fullbench-lib 2016-12-15 12:09:23 +01:00
Przemyslaw Skibinski
60f10aab6c introduced ZSTDLIB_VISIBILITY 2016-12-15 11:32:31 +01:00
Yann Collet
2b36b238d3 changed variable name to estimatedSrcSize, to emphasize it does not need to be exact 2016-12-13 17:59:55 +01:00
Yann Collet
e795c8a5f6 Added ZSTD_initCStream_srcSize(). 
Added relevant test cases in zstreamtest
 2016-12-13 17:00:14 +01:00
Yann Collet
5397a66b19 minor BMI version check 2016-12-13 15:21:06 +01:00
Yann Collet
35168679bd Merge pull request #478 from terrelln/wildcopy-ub 
Fix execSequence wildcopy undefined behavior
 2016-12-13 11:33:00 +01:00
Nick Terrell
064a143520 Fix execSequence wildcopy undefined behavior 
execSequence relied on pointer overflow to handle cases where
`sequence.matchLength < 8`.  Instead of passing an `size_t` to
wildcopy, pass a `ptrdiff_t`.
 2016-12-12 19:01:23 -08:00
Nick Terrell
e474aa55b4 Fix decompression buffer overrun 
Allows an adversary to write up to 3 bytes beyond the end of the buffer.
Occurs if the match overlaps the `extDict` and `currentPrefix`, and the
match length in the `currentPrefix` is less than `MINMATCH`, and
`op-(16-MINMATCH) >= oMatchEnd > op-16`.
 2016-12-12 18:05:30 -08:00
Yann Collet
c3a5c4bef8 introduced cycleLog 2016-12-12 00:47:30 +01:00
Yann Collet
c261f71f6a minor variation of rescale fix 2016-12-12 00:25:07 +01:00
Nick Terrell
3826207a70 Simplify segfault fix 
Take advantage of the fact that `chainLog <= windowLog`.
 2016-12-10 18:46:55 -08:00
Nick Terrell
0012332ce0 Fix compression segfault 
When the overflow protection kicks in, it makes sure that ip - ctx->base
isn't too large.  However, it didn't ensure that saved offsets are
still valid.  This change ensures that any valid offsets (<= windowLog)
are still representable after the update.

The bug would shop up on line 1056, when `offset_1 > current + 1`, which
causes an underflow.  This in turn, would cause a segfault on line 1063.

The input must necessarily be longer than 1 GB for this issue to occur.
Even then, it only occurs if one of the last 3 matches is larger than
the chain size and block size.
 2016-12-09 17:15:33 -08:00
Yann Collet
383b8088a3 minor lib build refactoring 2016-12-08 18:42:27 -08:00
Yann Collet
6e754fe76a fixed lib soname. 
example : simple_compression : size overflow check
 2016-12-08 18:26:56 -08:00
Przemyslaw Skibinski
7687913178 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-12-08 10:42:42 +01:00
Yann Collet
426a9d4b71 changed : dll : only approved ZSTD symbols are now exposed. All other symbols remain internal. 2016-12-07 16:39:34 -08:00
Przemyslaw Skibinski
4da53219a0 zstd Manual updated to 1.1.2 2016-12-07 11:18:40 +01:00
Yann Collet
379908be3d fixed zstd.h for manual 2016-12-06 10:36:15 -08:00
Yann Collet
9dfd0af164 ZBUFF_ as a wrapper to ZSTD streaming API. 2016-12-06 17:16:41 +01:00
Yann Collet
825dffbc43 moved zbuff source files into lib/deprecated 2016-12-05 19:28:19 -08:00
Yann Collet
8f8e2b0b4a fixed initialization warning 2016-12-05 18:00:50 -08:00
Yann Collet
e7a41a5955 added : dictID retrieval functions. 
added : unit tests for dictID retrieval functions
 2016-12-05 16:21:06 -08:00
Yann Collet
9ffbeea875 API : changed : streaming decompression : implicit reset on starting new frames 2016-12-02 18:37:38 -08:00
Yann Collet
2238312c2f fix dict loading 2016-12-02 11:36:11 -08:00
Przemyslaw Skibinski
821bf1febc fixed Doxygen trailing comment 2016-12-02 16:13:41 +01:00
Yann Collet
b89af20353 reduced table sizes for HUF_readDTableX4 2016-12-01 18:24:59 -08:00
Yann Collet
a0d742b1e4 introduced HUF_buildCTable_wksp(), to reduce stack memory usage 2016-12-01 17:47:30 -08:00
Yann Collet
643d9a234b replaced usage of FSE_buildCTable by FSE_buildCTable_wksp, using less stack space in the process 2016-12-01 16:24:04 -08:00
Yann Collet
e928f7e16d introduced ext_wksp variants of count to reduce stack memory usage 2016-12-01 16:13:35 -08:00
Yann Collet
979cab412b fixed some minor visual silent cast warnings. 
introduced FSE_count_parallel_wksp().
 2016-11-30 18:10:38 -08:00
Yann Collet
5e00b848a8 FSE_compress_wksp() uses less stack space 2016-11-30 16:46:13 -08:00
Yann Collet
d79a9a00d9 Introduced FSE_compress_wksp() and FSE_buildCTable_wksp() to reduce stack memory usage 2016-11-30 15:52:20 -08:00
Yann Collet
766431909f introduced FSE_decompress_wksp(), to use less stack space 2016-11-30 12:36:45 -08:00
Yann Collet
95eb43be09 updated pkg config file 2016-11-30 11:06:58 -08:00
Yann Collet
42247705a3 Merge pull request #461 from obache/neatsrc/pkgconfig 
libzstd.pc.in: Change to use variables for libdir and includedir
 2016-11-30 20:03:42 +01:00
Yann Collet
ff504de391 minor decompression speed improvement 2016-11-29 17:42:46 -08:00
Yann Collet
25f46dcc0f minor const 2016-11-29 16:59:27 -08:00
Yann Collet
a56ac2815c restored normal decoder speed 2016-11-29 15:30:23 -08:00
Yann Collet
37870d7a66 fixed minor visual warning 2016-11-29 14:31:57 -08:00
Yann Collet
013a2b58ae Merge pull request #464 from terrelln/guards 
Fix ZSTD_STATIC_LINKING_ONLY with double include
 2016-11-29 23:06:10 +01:00
Yann Collet
167c494748 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-11-29 14:05:15 -08:00
Yann Collet
4f5350f610 long matches support overflow 2016-11-29 13:12:24 -08:00
Nick Terrell
05c00f2ff7 Fix ZSTD_STATIC_LINKING_ONLY with double include 2016-11-29 11:54:39 -08:00
OBATA Akio
c53eea7c1a libzstd.pc.in: Change to use variables for libdir and includedir 2016-11-29 16:47:53 +09:00
Yann Collet
52e136ed3d long decoder compatible with round and separate buffers 2016-11-28 19:59:11 -08:00
Yann Collet
ce3527ca0c combined normal and long decoder 2016-11-28 18:38:52 -08:00
Yann Collet
8993bee997 restored normal mode 2016-11-28 16:11:30 -08:00
Yann Collet
764e70a4f3 added decodeSequencesLong 2016-11-28 15:50:16 -08:00
Yann Collet
73f88a66f1 added prefetch 2016-11-23 15:43:30 -08:00
Yann Collet
50524bf0da delayed decompression 2016-11-23 15:11:07 -08:00
Przemyslaw Skibinski
fc4193bda5 fixed g++ warnings 2016-11-23 18:17:18 +01:00
Przemyslaw Skibinski
9ca65af810 zstd_opt.h: improved price function 2016-11-23 17:22:54 +01:00
Przemyslaw Skibinski
ad3e94512c fixed warnings from static analyzer in zstd_opt.h 2016-11-21 20:22:12 +01:00
Przemyslaw Skibinski
eec700a3b7 exclude zbuff_compress.c and zbuff_decompress.c 2016-11-21 14:34:03 +01:00
Przemyslaw Skibinski
62d19a6f39 lib\README.md: added Using MinGW+MSYS to create DLL 2016-11-21 14:22:08 +01:00
Przemyslaw Skibinski
b85f767743 files to generate ZSTD Windows binary package 2016-11-21 14:10:55 +01:00
Przemyslaw Skibinski
8bb86e330b create DLL with Windows 2016-11-21 12:51:01 +01:00
Przemyslaw Skibinski
93a09eedf1 added libzstd.def 2016-11-21 12:33:27 +01:00
Przemyslaw Skibinski
5a17223691 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-11-18 11:47:01 +01:00
Przemyslaw Skibinski
3d18088b38 updated windres 2016-11-17 18:04:41 +01:00
Yann Collet
0d761dbe95 Merge pull request #453 from inikep/dev11 
fullbench-dll
 2016-11-16 15:45:30 -08:00
Yann Collet
52afb3993e zbuff API now generates deprecation warnings 2016-11-16 08:50:54 -08:00
Przemyslaw Skibinski
179555c1d1 working fullbench-dll 2016-11-15 18:05:46 +01:00
Nick Terrell
4359d21ad7 Merge two memset() calls into one 2016-11-14 17:52:51 -08:00
Nick Terrell
24701de877 Fix uninitialized memory read 2016-11-14 13:57:05 -08:00
Yann Collet
8e4901eccd removed zbuff.h from include installation 2016-11-08 15:45:39 -08:00
Yann Collet
fd3be6bc97 bump version number to 1.1.2 2016-11-07 14:35:41 -08:00
Nick Terrell
dc904ad17b Fix bug in zstd v0.{5, 6} dictionary decompression 
Introduced by bb68062c59.
 2016-11-04 16:18:59 -07:00
Przemyslaw Skibinski
38b590ad69 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 
# Conflicts:
#	lib/Makefile
 2016-11-04 10:10:54 +01:00
Yann Collet
407a11f63e fixed Visual compatibility 2016-11-03 15:52:01 -07:00
Yann Collet
11812260d1 Merge pull request #439 from terrelln/dev 
ZSTD_compress_usingDict() specify dict == NULL
 2016-11-03 14:15:36 -07:00
Nick Terrell
c8a9ac312b Fix dynamic libzstd symlinks 2016-11-03 12:32:48 -07:00
Przemyslaw Skibinski
3a415594b1 fixed MinGW compilation 2016-11-03 12:59:20 +01:00
Yann Collet
7347869fb6 fixed make install 2016-11-02 22:28:37 -07:00
Nick Terrell
d82efd8a70 ZSTD_compress_usingDict() when dict gets loaded 
Specify that when `dict == NULL || dictSize < 8` no dictionary
gets loaded.
Also add some periods.
 2016-11-02 18:07:16 -07:00
Yann Collet
179b19776f fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h 
Added : ZSTD_isFrame() in experimental section
 2016-11-02 17:30:49 -07:00
Yann Collet
f3f13211ae Fix #419 : no warning when setting custom LDFLAGS 2016-11-02 17:02:45 -07:00
Yann Collet
0a5a5fb7fd Fix #418 : printing selected segments in zdict debug mode can segfault with certain pathological patterns 2016-11-02 13:57:55 -07:00
Yann Collet
31e660e7aa more accurate default maximum window size 2016-10-29 03:56:45 -07:00
Yann Collet
2115724c22 Merge pull request #430 from terrelln/exec-sequences 
ZSTD_execSequence() accepts match in last 7 bytes
 2016-10-28 10:45:05 -07:00
Nick Terrell
10bfd0c0d5 Fix ZSTD_execSequence() performance regression 
Commit ae1cb3b3d0 caused the regression.
It is an instruction alignment issue, because if it is `U64 i` instead
of `U32 i`, the regression returns.  This patch fixes the regression
in gcc, but only gets some of the clang performance back.

Benchmarks:
Run on `silesia.tar`.  I only show levels 1-5 because the performance
regression was uniform across all levels.  I did one run on levels
1-19 and it looked good.

| Build | Level | Before | While | After |
|-------|-------|-------:|------:|------:|
| gcc   |     1 |  931.4 | 904.4 | 932.8 |
| gcc   |     2 |  849.1 | 822.6 | 851.2 |
| gcc   |     3 |  815.6 | 790.6 | 818.9 |
| gcc   |     4 |  794.1 | 770.7 | 798.0 |
| gcc   |     5 |  785.7 | 760.7 | 788.8 |
| clang |     1 |  705.5 | 683.2 | 693.8 |
| clang |     2 |  670.0 | 649.2 | 660.7 |
| clang |     3 |  659.6 | 639.8 | 651.4 |
| clang |     4 |  652.5 | 634.7 | 645.9 |
| clang |     5 |  646.9 | 625.5 | 637.7 |
 2016-10-27 16:19:57 -07:00
Yann Collet
ee5b725823 ZSTD_initCStream() optimization : do not allocate a CDict when no dictionary used 2016-10-27 14:20:55 -07:00
Nick Terrell
eb7873a048 ZSTD_execSequence() accepts match in last 7 bytes 
The zstd reference compressor will not emit a match in the last 7
bytes of a block.  The decompressor will also not accept a match
in the last 7 bytes.  This patch makes the decompressor accept a
match in the last 7 bytes.
 2016-10-25 21:24:15 -07:00
Yann Collet
335ad5d4d4 added ZSTD_initDStream_usingDDict() . 
slightly optimized ZSTD_initDStream() when no dictionary .
fixed ZSTD_sizeof_CStream() .
 2016-10-25 17:47:02 -07:00
Yann Collet
9516234e67 first sketch for ZSTD_initCStream_usingCDict() 2016-10-25 16:19:52 -07:00
Yann Collet
62d9a7ddfd Merge pull request #429 from inikep/btopt2 
Btopt2
 2016-10-25 14:48:43 -07:00
Przemyslaw Skibinski
5c5f01f3da added ZSTD_btopt2 strategy 2016-10-25 12:25:07 +02:00
Yann Collet
7b5948cca7 Merge pull request #426 from terrelln/fixes 
Fix various {A, M}SAN bugs
 2016-10-24 23:42:26 -07:00
Yann Collet
37d130031d updated comments on context re-use 2016-10-24 17:22:12 -07:00
Nick Terrell
b2c39a22b0 Fix compiler narrowing warning 2016-10-24 14:50:13 -07:00
Nick Terrell
f698ad6deb Merge remote-tracking branch 'upstream/dev' into fixes 
* upstream/dev:
  added doc\zstd_manual.html
  added contrib\gen_html
  zstd_compression_format.md moved to doc/
  Fix small bug in ZSTD_execSequence()
  improved ZSTD_compressBlock_opt_extDict_generic
  protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
  zstd_opt.h: small improvement in compression ratio
  improved dicitonary segment merge
  use implicit rules to compile zstd_decompress.c
  detect early impossible decompression scenario in legacy decoder v0.5
  no repeat mode in legacy v0.5
  fixed invalid invocation of dictionary in legacy decoder v0.5
  fix edge case
  fix command line interpretation
  fixed minor corner case
  zstd.h: added the Introduction section
  fixed clang 3.5 warnings
  zstd.h: updated comments
 2016-10-24 13:10:13 -07:00
Yann Collet
4239a207dd Merge pull request #425 from inikep/dev11 
Doc
 2016-10-24 11:11:40 -07:00
Nick Terrell
f9c9af3c2e Reject dictionaries with incomplete entropy tables 
If a dictionary specifies that a symbol has probability zero in its
`matchLength`, `literalLength`, or `offset` FSE table, but the symbol
appears when compressing input, the compressor fails.

Ensure that dictionaries support all `matchLength`, and `literalLength`
codes.  They must also support all of the `offset` codes required to
represent every possible offset that can appear in the first block.
 2016-10-24 10:42:44 -07:00
Przemyslaw Skibinski
984b66cd72 added contrib\gen_html 2016-10-24 15:59:51 +02:00
Przemyslaw Skibinski
3ee94a7600 zstd_compression_format.md moved to doc/ 2016-10-24 15:58:07 +02:00
Yann Collet
97611611a3 Merge pull request #423 from terrelln/exec-seq-patch 
Fix small bug in ZSTD_execSequence()
 2016-10-21 17:02:06 -07:00
Nick Terrell
ae1cb3b3d0 Fix small bug in ZSTD_execSequence() 
`memmove(op, match, sequence.matchLength)` is not the desired behavior.
Overlap is allowed, and handled as if we did `*op++ = *match++`, which
is not how `memmove()` handles overlap.

Only triggered if both of the following conditions are met:
* The match spans extDict & currentPrefixSegment
* `oLitEnd <= oend_w < oLitEnd + length1 < oMatchEnd <= oend`.

These two conditions imply that the block is less than 15 bytes long.
This bug isn't triggered by the streaming API, because it allocates
enough space for the window size + the block size, so there cannot be
a match that is within 8 bytes of the end and overlaps with itself.
It cannot be triggered by the block decompression API because all of
the decompressed data is in the currentPrefixSegment.

Introduced by commit 7158584399
 2016-10-21 12:13:44 -07:00
Przemyslaw Skibinski
4732074a71 improved ZSTD_compressBlock_opt_extDict_generic 2016-10-21 11:19:00 +02:00
Yann Collet
da3bd8b6de protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob 2016-10-20 20:11:00 -07:00
Przemyslaw Skibinski
d365ae3497 zstd_opt.h: small improvement in compression ratio 2016-10-20 11:49:02 +02:00
Przemyslaw Skibinski
575ab00db7 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-10-20 11:01:52 +02:00
Nick Terrell
d760529a05 Fix stack buffer overrun when weightTotal == 0 
If `weightTotal == 0`, then `BIT_highbit32(weightTotal)` is
undefined behavior in the case that it calls `__builtin_clz()`.
If `tableLog == HUF_TABLELOG_ABSOLUTEMAX` then we will access one
byte beyond the end of the buffer.
 2016-10-19 11:39:11 -07:00
Nick Terrell
bb68062c59 Unitialized memory read in ZSTD_decodeSeqHeaders() 
Caused by two things:
1. Not checking that `ip` is in range except for the first byte.
2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.
 2016-10-18 16:41:33 -07:00
Yann Collet
52c1bf93fe improved dicitonary segment merge 2016-10-18 16:34:58 -07:00
Nick Terrell
7b06ad7a05 Backport fix from commit 125d817 
This fixes a read of unitialized memory.
Full commit hash: 125d81774f.
 2016-10-18 14:52:34 -07:00
Nick Terrell
f45b157d95 Backport fix from commit 9e8b09a 
Fixes uninitialized memory reads.
Full commit hash: 9e8b09a7bd
 2016-10-18 14:22:49 -07:00
Yann Collet
f7906d5955 detect early impossible decompression scenario in legacy decoder v0.5 2016-10-18 13:48:32 -07:00
Yann Collet
9313c8d953 no repeat mode in legacy v0.5 2016-10-18 13:36:15 -07:00
Yann Collet
83d7bdee4b fixed invalid invocation of dictionary in legacy decoder v0.5 2016-10-18 12:25:43 -07:00
Yann Collet
197a55ee7b fix edge case 2016-10-18 11:27:52 -07:00
Nick Terrell
fd98087047 Fix stack buffer overflow in HUF_readCTable() 
If `w ==0` on line 153, then `CTable[n].nbBits == tableLog + 1`.
Then `nbPerRank[CTable[n].nbBits]` and `valPerRank[CTable[n].nbBits]`
are stack buffer overflows.
 2016-10-17 18:16:59 -07:00
Yann Collet
06573e17be fixed minor corner case 2016-10-17 17:28:28 -07:00
Nick Terrell
bfd943ace5 Fix buffer overrun in ZSTD_loadDictEntropyStats() 
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadDictEntropyStats()`.  This caused `FSE_buildCTable()`
to stack/heap overflow in a few places.

The benchmarks look good, there is no obvious compression performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e10 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 271.6 MB/s , 716.8 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 204.8 MB/s , 671.1 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.8 MB/s , 658.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 136.4 MB/s , 665.3 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.9 MB/s , 647.0 MB/s
   6#silesia.tar       : 211988480 ->  62979643 (3.366),  65.2 MB/s , 670.4 MB/s
   7#silesia.tar       : 211988480 ->  61974560 (3.421),  44.9 MB/s , 688.2 MB/s
   8#silesia.tar       : 211988480 ->  61028308 (3.474),  32.4 MB/s , 711.9 MB/s
   9#silesia.tar       : 211988480 ->  60416751 (3.509),  21.1 MB/s , 718.1 MB/s
  10#silesia.tar       : 211988480 ->  60174239 (3.523),  22.2 MB/s , 721.8 MB/s

  > ./compress_zstds/zstd.opt.1 -i10 -b1 -e10 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 273.8 MB/s , 722.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 203.2 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 157.4 MB/s , 666.5 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 132.1 MB/s , 661.9 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.8 MB/s , 641.6 MB/s
   6#silesia.tar       : 211988480 ->  62979643 (3.366),  63.1 MB/s , 677.0 MB/s
   7#silesia.tar       : 211988480 ->  61974560 (3.421),  44.3 MB/s , 678.2 MB/s
   8#silesia.tar       : 211988480 ->  61028308 (3.474),  33.1 MB/s , 708.9 MB/s
   9#silesia.tar       : 211988480 ->  60416751 (3.509),  21.5 MB/s , 710.1 MB/s
  10#silesia.tar       : 211988480 ->  60174239 (3.523),  21.9 MB/s , 723.9 MB/s
 2016-10-17 16:55:52 -07:00
Nick Terrell
4db751668f Fix buffer overrun in ZSTD_loadEntropy() 
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadEntropy()`.  This caused `FSE_buildDTable(dctx->MLTable, ...)`
to overwrite the beginning of `dctx->hufTable`.

The benchmarks look good, there is no obvious performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 268.2 MB/s , 701.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.5 MB/s , 666.9 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 154.9 MB/s , 655.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 128.9 MB/s , 648.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.4 MB/s , 633.4 MB/s

  > ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 266.1 MB/s , 703.7 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.0 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.2 MB/s , 656.2 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 133.2 MB/s , 647.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.3 MB/s , 633.3 MB/s
 2016-10-17 15:51:15 -07:00
Nick Terrell
ccfcc643da Check if dict is empty before reading first byte 2016-10-17 11:46:03 -07:00
Yann Collet
2b361cf2f1 minor opt 2016-10-14 16:09:07 -07:00
Yann Collet
7933434fdf Merge branch 'dev' of github.com:facebook/zstd into dev 2016-10-14 13:32:35 -07:00
Yann Collet
d4cda27b63 new command -M#, to limit memory usage during decompression (#403) 2016-10-14 13:32:20 -07:00
Nick Terrell
3b9cdf9220 Fix ubsan failures (pass NULL to memcpy) 2016-10-12 20:54:42 -07:00
Yann Collet
5d919e7ac3 added ZSTD_error_frameParameter_windowTooLarge (#403) 2016-10-12 17:29:24 -07:00
Yann Collet
e19111c42f make creates libzstd binaries (#415) 2016-10-12 11:09:36 -07:00
Yann Collet
8b70d012f0 fix cmake 2016-10-12 10:23:53 -07:00
Yann Collet
38fb0dc4cf Merge pull request #416 from terrelln/exec-sequence 
Fix ZSTD_execSequence() edge case
 2016-10-12 10:17:53 -07:00
Nick Terrell
7158584399 Fix ZSTD_execSequence() edge case 2016-10-12 10:05:26 -07:00
Yann Collet
f52cd03e73 bumped version number 2016-10-11 17:29:27 -07:00
Yann Collet
ef2357d0d3 created error_private.c, so that a single list of error strings get included 2016-10-11 17:24:50 -07:00
Yann Collet
14efab827b added zstd_errors.h to include installation 2016-10-11 16:51:29 -07:00
Yann Collet
a17fd7312a changed error_public.h into zstd_errors.h 2016-10-11 16:41:09 -07:00
Yann Collet
18b51b99c0 sync fse 2016-10-11 08:21:09 -07:00
inikep
2d2613399a zstd.h: added the Introduction section 2016-10-06 16:28:21 +02:00
inikep
ba1db376ac fixed clang 3.5 warnings 2016-10-06 14:22:48 +02:00
inikep
82057aa7ec zstd.h: updated comments 2016-10-06 13:23:52 +02:00
Yann Collet
df6797447f update dictionary builder warning comments 2016-09-27 15:14:32 +02:00
Yann Collet
47094ea66b added comment on filePos 2016-09-26 18:03:33 +02:00
Yann Collet
cf409a7e2a fixed : init*_advanced() followed by reset() with different pledgedSrcSiz 2016-09-26 16:41:05 +02:00
Yann Collet
2f2639438a zstreamtest can fuzztest pledgedSrcSize 2016-09-26 14:06:08 +02:00
Christophe Chevalier
dc245e91cb Changed to use ZSTDLIBv06_API and ZSTDLIBv07_API for DLL exports to fix warning 
- changed name to prevent collision with ZSTDLIB_API used by non-legacy dll exports
 2016-09-23 17:09:36 +02:00
Yann Collet
21412bb3f6 Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-09-22 15:57:56 +02:00
Yann Collet
51f4d566c2 small decompression speed boost for very small data 2016-09-22 15:57:28 +02:00
Yann Collet
97b378a6f8 Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize. 2016-09-21 17:20:19 +02:00
Yann Collet
993060e0f2 cli : better adaptation to small files 2016-09-21 16:46:08 +02:00
Yann Collet
1eb2fdc74f bumped version number 2016-09-18 12:21:47 +02:00
Yann Collet
a6bdf55759 fixed memory leak 2016-09-15 17:02:06 +02:00
Yann Collet
644a8da88a fixed minor conversion warning 2016-09-15 16:16:21 +02:00
Yann Collet
4cb212938c introduced ZSTD_resetCStream() 2016-09-15 14:54:07 +02:00
Yann Collet
fa0c09760c variable renaming 2016-09-15 14:11:01 +02:00
Yann Collet
d7c6589df8 support ZSTD_sizeof_*() on NULL 
added ZSTD_sizeof_CDict()
 2016-09-15 02:57:27 +02:00
Yann Collet
e91c4b4cef introduced ZSTD_resetDStream() . 
added : ZSTD_sizeof_DDict()
 2016-09-14 16:55:44 +02:00
Yann Collet
d092d77cfc minor variable renaming 2016-09-14 16:14:57 +02:00
Yann Collet
64deef3bee Fixed srcSize=1 2016-09-14 00:16:07 +02:00
Yann Collet
26ec254066 new strategy for faster DDict decompression 2016-09-13 16:52:16 +02:00
Yann Collet
ac175d46d4 updated comments 2016-09-13 00:51:47 +02:00
Yann Collet
a3481d6de0 make uninstall 2016-09-12 05:04:26 +02:00
Yann Collet
b3060f7a9e changed streaming decoder behavior : now, when all compressed frame is consumed, it means decompression is completed, with regenerated data fully flushed. 2016-09-09 16:44:16 +02:00
Yann Collet
01c199226a updated decompression streaming example 2016-09-08 19:29:04 +02:00
Yann Collet
5c6d244973 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-09-07 14:54:54 +02:00
Yann Collet
ac8bace6b1 support large skippable frames 2016-09-07 14:54:23 +02:00
Yann Collet
0e07bf3f60 added comments on searchLength min / max (#337) 2016-09-07 06:33:02 +02:00
Yann Collet
95d07d7447 introduced CHECK_E 2016-09-06 16:38:51 +02:00
Yann Collet
3e21ec5b01 introduced CHECK_F 2016-09-06 15:36:19 +02:00
Yann Collet
5c956d593c FORCE_INLINE common definition 2016-09-06 15:05:19 +02:00
Yann Collet
edbcd9f5b2 fixed zbufftest 2016-09-06 14:30:57 +02:00
Yann Collet
b624922b14 fixed checksum 2016-09-06 11:16:57 +02:00
Yann Collet
a7737f6a60 improved compression on small files when using same parameters 2016-09-06 09:44:59 +02:00
Yann Collet
7ae67bb18a small compression speed gains with using_CDict 2016-09-06 06:28:05 +02:00
Yann Collet
1d4208c029 clarified streaming decompression inlined doc 2016-09-06 05:16:40 +02:00
Yann Collet
7c83dfd5c2 ZSTD_frameHeaderSize_prefix (#340), as result of ZSTD_initStream 2016-09-05 19:47:43 +02:00
Yann Collet
fa72f6bdce clarified inline doc for streaming 2016-09-05 17:39:56 +02:00
Yann Collet
c73a8109bb Merge pull request #344 from inikep/dev10 
unified error codes for legacy decoders
 2016-09-05 07:46:33 -07:00
inikep
45db83f98d ZSTD_decodeLiteralsBlock renamed to ZSTDv01_decodeLiteralsBlock 2016-09-05 14:46:24 +02:00
inikep
476964f6a1 ZSTD_decodeSeqHeaders renamed to ZSTDv01_decodeSeqHeaders 2016-09-05 13:34:57 +02:00
inikep
c13faa1b0f legacy decoders: restored #include <intrin.h> for VC++ 2016-09-05 13:25:07 +02:00
inikep
8161e7321a unified error codes for legacy decoders 2016-09-05 12:29:51 +02:00
Thomas Klausner
b85cdabd50 Enable install targets for NetBSD. 2016-09-04 14:37:57 +02:00
Yann Collet
33a0465a51 fixed a few links 2016-09-02 22:11:49 -07:00
Yann Collet
d56dbc02d3 removed g_displayLevel 2016-09-02 17:28:41 -07:00
Yann Collet
855766d73d clarified dictionary in format description 2016-09-02 17:04:49 -07:00
Yann Collet
d725427a3c g_time => local displayTime 2016-09-02 15:32:39 -07:00
Yann Collet
1563bfeabc fixing FORCE_INLINE for older compilers (#330) 2016-09-02 11:44:21 -07:00
Yann Collet
7304eb7c09 bumped version number 2016-09-01 15:49:26 -07:00
Yann Collet
901e85fe26 version bump 2016-08-31 07:51:25 -07:00
Yann Collet
1c59c20903 removed redundant files 2016-08-31 07:15:44 -07:00
Yann Collet
599c69d917 minor Makefile updates 2016-08-30 13:33:20 -07:00
David Lam
e10f7f3dcb merge 2016-08-30 12:03:36 -07:00
Yann Collet
4ded9e591c added boilerplate 2016-08-30 11:06:28 -07:00
Yann Collet
3b15f1f10f minor refactor 2016-08-30 09:58:50 -07:00
Yann Collet
240795bef7 Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-08-30 06:51:55 -07:00
Yann Collet
14200a20f0 Fixed issue #304, reported by @borzunov 2016-08-30 06:51:00 -07:00
David Lam
da9d3b7057 Cleanup some errors in typedef comments and remove duplicated HOWTO from zbuff_decompress.c 2016-08-29 17:31:51 -07:00
Yann Collet
09c3c8e885 Merge pull request #307 from inikep/dev08 
updated README.md
 2016-08-29 16:32:33 -07:00
inikep
6416b0d705 updated README.md 2016-08-29 13:04:26 +02:00
Yann Collet
23b6e05d8e ZSTD_malloc() and ZSTD_free(), to simplify customMem 2016-08-28 21:05:43 -07:00
Yann Collet
5f53b0335e fixed continuation context 2016-08-28 10:00:49 -07:00
Yann Collet
767d8f66fa legacy contexts can be re-used 2016-08-28 08:19:47 -07:00
Yann Collet
4bf317dd00 first version supporting legacy streams (transparent decoding) 2016-08-28 07:43:34 -07:00
Yann Collet
e19a9ef05d update compression level table 2016-08-26 20:02:49 +02:00
Yann Collet
9a021c1aae fixed some minor clang warnings 2016-08-26 09:05:06 +02:00
Yann Collet
cb5a320705 made -Wdocumentation a clang only flag 2016-08-26 08:06:36 +02:00
Yann Collet
87c18b2ebd fixed multiple minor warnings for XCode 2016-08-26 01:43:47 +02:00
Yann Collet
0d59a6f73a removed debug strings 2016-08-25 22:42:46 +02:00
Yann Collet
5a02b69215 reinforced fix for huge files 2016-08-25 22:24:59 +02:00
Yann Collet
96bdd87de4 fixed : compression bug on very large files 2016-08-25 14:34:42 +02:00
inikep
a3a47ec4d0 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into Other 2016-08-24 21:25:49 +02:00
Yann Collet
a2cdffe556 fixed wrong parameter issue 2016-08-24 19:42:15 +02:00
inikep
e416e30019 remove unnecessary comments 2016-08-24 17:32:09 +02:00
inikep
4e90f6c1e0 removed ZSTD_LOG_ENCODE and ZSTD_LOG_BLOCK 2016-08-24 17:24:11 +02:00
inikep
83388e109f removed ZSTD_LOG_PARSER 2016-08-24 17:22:20 +02:00
inikep
8a36f8527c removed stats in debug mode 2016-08-24 17:19:12 +02:00
inikep
57ef4b1a0d zstd_v07.c: removed unused macros 2016-08-24 17:16:56 +02:00
Yann Collet
24b68a5b23 update cLevel table for 256KB 2016-08-24 14:22:26 +02:00
Yann Collet
c54692faeb improved level 3 2016-08-24 01:45:46 +02:00
Yann Collet
17e482efdd added ZSTD_setDStreamParameter() 2016-08-23 16:58:10 +02:00
Yann Collet
3071c3e303 STREAM_WINDOW_MAX : protect streaming from unreasonable memory requirements 2016-08-23 01:34:34 +02:00
Yann Collet
70e3b31306 fixed playtests on os-x 2016-08-23 01:18:06 +02:00
Yann Collet
cb3276329a added sizeof CStream and DStream 2016-08-23 00:31:59 +02:00
Yann Collet
d1733f7417 fixed crc bug in rare timing conditions within bench.c 2016-08-21 01:04:46 +02:00
Yann Collet
8baf78a291 minor coding style 2016-08-20 13:04:20 +02:00
Yann Collet
1bee2d5e08 slight decompression speed improvement 2016-08-20 02:59:04 +02:00
Yann Collet
0cfe2ec2fd sync fse version 2016-08-20 00:26:26 +02:00
Yann Collet
af1960396b sync huff0 2016-08-19 19:38:19 +02:00
Yann Collet
7be46bf8f2 promoted streaming API to stable 
(except _advanced() variants)
 2016-08-19 18:39:36 +02:00
Yann Collet
fdba57d513 update version number 2016-08-19 18:32:30 +02:00
Yann Collet
da3fbcb302 Added ZDICT_getDictID() 2016-08-19 14:23:58 +02:00
Yann Collet
a5dbf9f629 Merge pull request #297 from borzunov/dev 
Export functions related to dictionary compression from DLL
 2016-08-18 15:05:01 +02:00
Yann Collet
49d105cfcf better warning and error messages in case of dictionary training failure (#292) 2016-08-18 15:02:11 +02:00
Alexander Borzunov
0f6f17a14f Rename ZSTDLIB_API to ZDICTLIB_API in zdict.h 2016-08-18 16:47:06 +05:00
Alexander Borzunov
1f48382b1a Export functions related to dictionary compression from DLL 2016-08-18 16:12:49 +05:00
Yann Collet
e80d15304a Merge pull request #296 from inikep/Other 
Other
 2016-08-18 11:48:48 +02:00
inikep
a7bb322a93 removed never referenced functions 2016-08-18 10:30:21 +02:00
Yann Collet
18442c1482 minor refactoring 2016-08-18 01:40:32 +02:00
Yann Collet
c411902230 fixed g++ conversion warning 2016-08-17 01:50:54 +02:00
Yann Collet
53e17fbd5e updated streaming API 2016-08-17 01:39:22 +02:00
Yann Collet
655393cc72 updated doc for streaming API 2016-08-16 15:11:28 +02:00
Yann Collet
104e5b072d added : streaming decompression API 2016-08-16 15:11:28 +02:00
Yann Collet
5a0c8e2439 new streaming API (compression) 2016-08-16 15:11:27 +02:00
Yann Collet
ba92046031 Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-08-11 22:10:04 +02:00
Yann Collet
e9b414d825 fixed msan warning (#281) 2016-08-11 22:09:09 +02:00
inikep
5f49eba512 added usage of rep[0]-1 for the optimal parser 2016-08-10 15:01:53 +02:00
inikep
98e08cbe34 fixed: tree not updated after finding very long rep matches 2016-08-10 15:00:30 +02:00
inikep
038d1497c9 fixed compilation with Visual Studio 2005 2016-08-10 14:30:10 +02:00
inikep
48849f86f0 fixed compilation with Intel Compiler with Windows 2016-08-10 14:26:35 +02:00
Yann Collet
1ea5622a32 updated xxhash 2016-08-10 09:40:08 +02:00
Yann Collet
666398e7ed added : xxhash namespace enforced from xxhash.h. 
added : xxhash namespace test.
removed : -DXXH_NAMESPACE
 2016-08-10 08:16:51 +02:00
Yann Collet
8ded0b84aa update xxhash to v0.6.2 2016-08-10 07:40:40 +02:00
Yann Collet
280f9a8754 minor comment 2016-08-08 00:44:00 +02:00
Yann Collet
e0b4a2d40f fixed dictionary generation, reported by Bartosz Taudul 2016-08-03 03:36:03 +02:00
Yann Collet
ae40b18d55 bumped library number 2016-08-03 01:59:21 +02:00
Yann Collet
0763905f44 ZSTD_compress_usingCDict() correctly provides original size by default in frame header 
Fixed dictionary examples
 2016-08-03 01:57:57 +02:00
Yann Collet
bf2bc112bb bench : controlled display update when loading lot of files 2016-08-02 23:48:13 +02:00
Yann Collet
346efccc35 fixed doc typo 2016-08-02 14:26:00 +02:00
Yann Collet
f116e87f59 fixed analyzer warning 2016-08-01 19:15:18 +02:00
Yann Collet
9ba929f1d4 Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-08-01 02:26:59 +02:00
Yann Collet
3ca750372d updated doc (#269) 2016-08-01 02:26:20 +02:00
Yann Collet
c55eb18c11 Merge pull request #267 from inikep/dev08 
fixed ZSTD_compressBlock_opt_extDict_generic
 2016-07-31 22:00:16 +02:00
inikep
056df510aa fixed ZSTD_compressBlock_opt_extDict_generic 2016-07-31 20:08:53 +02:00
Yann Collet
917fe188f1 Implemented repOffset "minus 1" on ll==0 2016-07-31 04:01:57 +02:00
Yann Collet
2a2ba3691c Merge pull request #266 from jrmarino/master 
Enable build on FreeBSD ports (includes DragonFly BSD) [dev branch]
 2016-07-31 02:21:26 +02:00
jrmarino
0d07ec0c0c Enable build on FreeBSD ports (includes DragonFly BSD) 
Zstd has been introduced to FreeBSD ports
(http://www.freshports.org/archivers/zstd/) which DragonFly BSD also
uses.  FreeBSD and DragonFly use the install targets (albeit modified in
some cases) so they must be added to the associated Makefile filters.
 2016-07-30 19:11:15 -05:00
Yann Collet
8cebfd1d26 fix attempt on test-zstd-speed 2016-07-31 01:59:23 +02:00
Yann Collet
66f69e58d2 restore decompression speed on fizzle 2016-07-30 15:32:47 +02:00
Yann Collet
3b2bd1d11c zstd_opt uses same tables as zstd_compress 2016-07-30 13:21:41 +02:00
Yann Collet
f714f59c16 fixed visual warning 2016-07-30 12:05:28 +02:00
Yann Collet
761f8dbbd2 back to normal table cell copy 2016-07-30 11:43:53 +02:00
Yann Collet
3c6b808870 minor decompression speed gains 2016-07-30 03:20:47 +02:00
Yann Collet
70a9ff4af3 fixed too large selectivity level, reported by Ilona Papava 2016-07-30 01:09:14 +02:00
Yann Collet
c0ce4f1211 slightly improved compression speed 2016-07-30 00:55:13 +02:00
Yann Collet
ed57d8530a new seqStore 2016-07-29 21:22:17 +02:00
Yann Collet
6b615d32cd Updated API comments, following suggestions by Bryan O'Sullivan 2016-07-29 19:40:37 +02:00
Yann Collet
c00d30fbe4 Merge pull request #264 from inikep/dev08 
Dev08
 2016-07-29 17:42:30 +02:00
inikep
6b68ba2079 zstd_opt.h: fixed checking of rep codes (2) 2016-07-29 16:45:39 +02:00
inikep
59b86fc141 zstd_opt.h: fixed checking of rep codes 2016-07-29 11:00:33 +02:00
Yann Collet
6a82f0f8bf minor comments 2016-07-29 00:55:45 +02:00
Yann Collet
ffa7d0ac1e clarified comment 2016-07-28 21:01:17 +02:00
Yann Collet
4c5bbf64f9 fixed : frame concatenation without checksum 2016-07-28 20:30:25 +02:00
Yann Collet
60ba31c570 zbuff uses ZSTD_compressEnd() 2016-07-28 19:55:09 +02:00
Yann Collet
16e73033ad introduced stage zbf_end 2016-07-28 16:32:34 +02:00
Yann Collet
62470b4bab Changed ZSTD_compressEnd() 2016-07-28 15:29:08 +02:00
Yann Collet
e7bf9156d1 Clarified API comments, from suggestions by ‎Bryan O'Sullivan‎ 2016-07-28 05:00:57 +02:00
Yann Collet
d469a98c01 Clarified API comments, from suggestions by ‎Bryan O'Sullivan‎ 2016-07-28 04:55:03 +02:00
Yann Collet
19c1002e46 applied ZSTD_compressContinueThenEnd() 2016-07-28 01:25:46 +02:00
Yann Collet
5b56739b63 created ZSTD_compressContinueThenEnd() 2016-07-28 01:17:22 +02:00
Yann Collet
c991cc1828 new frame end, 32-bits checksums 2016-07-28 00:55:43 +02:00
Yann Collet
d4180cad9c minor code refactoring 2016-07-27 21:21:36 +02:00
Yann Collet
731ef16fc1 minor code style refactoring 2016-07-27 21:05:12 +02:00
Yann Collet
4b9ca0a6b5 minor example variation 2016-07-27 19:53:19 +02:00
Yann Collet
4110534886 ZSTD_maxCLevel() is promoted to "stable" API (#254, by @FrancescAlted) 2016-07-27 15:09:11 +02:00
Yann Collet
55a8bea0b5 fixed dictionary generation 2016-07-27 14:48:47 +02:00
Yann Collet
c154d9d6a2 better support for large dictionaries (> 128 KB) 2016-07-27 14:37:00 +02:00
Yann Collet
07626dfa51 improved dictbuilder notifications on selectivity 2016-07-27 13:28:46 +02:00
Yann Collet
f796f7ab45 removed fastscan mode 2016-07-27 12:53:54 +02:00
Yann Collet
dd25a27702 added tutorial warning messages for dictBuilder 2016-07-27 12:43:09 +02:00
inikep
003c7a8568 optimal parser: removed ZSTD_REP_INIT 2016-07-27 11:07:13 +02:00
Yann Collet
04cdd8660d Merge pull request #262 from ebiggers/misc_updates 
Miscellaneous updates
 2016-07-27 01:25:45 +02:00
Eric Biggers
0a55e7a0bb ZSTD_decompressFrame(): use remainingSize instead of iend - ip 
Same behavior, but no need to have redundant variables.
 2016-07-26 13:22:27 -07:00
Eric Biggers
aa6c70bf60 ZSTD_decompressFrame(): pass up error code from ZSTD_decodeFrameHeader() 2016-07-26 13:22:27 -07:00
Eric Biggers
e4d0265ea9 Replace remaining references to "direct mode" with "single segment mode" 2016-07-26 13:22:27 -07:00
Yann Collet
d50f9db3ea Improved speed on clang and gcc -O2, thanks to @ebiggers ! (#263) 2016-07-26 21:30:35 +02:00
Yann Collet
7adc2328a3 fixed --test on zero-length files, reported by @amnilsson 2016-07-26 15:49:24 +02:00
inikep
4178f5c289 fixed gcc warning: always_inline function might not be inlinable 2016-07-25 21:17:45 +02:00
inikep
fca90f8f60 legacy decoder for v0.7 format 2016-07-25 17:49:08 +02:00
Yann Collet
cbc5e9dc19 fixes oob read 2016-07-24 18:02:04 +02:00
Yann Collet
38b75ddeb2 removed special case all-1 huffman distribution 2016-07-24 15:35:59 +02:00
Yann Collet
7ed5e33b89 minor comment changes 2016-07-24 14:26:11 +02:00
Yann Collet
10b9c13d07 fixed doc on cLevel default, reported by Oliver Lange 2016-07-24 01:21:53 +02:00
Yann Collet
f8e7b5363f unified encoding types 2016-07-23 16:31:49 +02:00
Yann Collet
571a59034a changed enccoding type order : raw, rle, compressed, repeat-stats 2016-07-23 15:52:05 +02:00
Yann Collet
c2e1a68d81 changed streamNb order to 1-4-4-4 2016-07-22 17:30:52 +02:00
Yann Collet
772d912c2f more complete support for literals repeat mode 2016-07-22 15:04:25 +02:00
Yann Collet
9f2d82d4a4 fixed : big-endian decoding 2016-07-22 14:37:10 +02:00
Yann Collet
32faf6c8e7 fixed conversion warnings 2016-07-22 14:37:09 +02:00
Yann Collet
5e45a5fbb3 force loop-align to 32 for zstd_decompress 2016-07-22 14:37:09 +02:00
Yann Collet
5288ac0cb7 changed filed order 2016-07-22 14:37:09 +02:00
Yann Collet
198e6aac44 Literals header fields use little endian convention 2016-07-22 14:37:09 +02:00
Yann Collet
6fa05a2371 cBlockSize uses little-endian convention 2016-07-22 14:37:09 +02:00
Yann Collet
7bf72bbf5e update header to v0.8 2016-07-22 14:37:09 +02:00
Yann Collet
5894ea8d01 updated cLevels 2016-07-22 14:36:46 +02:00
Yann Collet
d5c5a77990 minor comments clarifications 2016-07-20 13:35:14 +02:00
Yann Collet
572b817be3 Merge pull request #253 from gymdis/heapmode_off_legacy_fix 
Fix compile issue with ZSTD_LEGACY_SUPPORT=1 and ZSTD_HEAPMODE=0
 2016-07-19 13:52:03 +02:00
Christopher Bergqvist
780a9fa857 Fix compile issue with ZSTD_LEGACY_SUPPORT=1 and ZSTD_HEAPMODE=0 2016-07-19 13:25:38 +02:00
Yann Collet
cf05b9d477 ZSTD_getBlockSizeMax() 2016-07-18 16:52:10 +02:00
Yann Collet
16aa38b0e0 minor doc clarifications 2016-07-18 03:52:47 +02:00
Yann Collet
85f3919960 moved zstd.h to /lib 2016-07-17 20:42:21 +02:00
Yann Collet
9375590462 update version to v0.7.5 2016-07-17 16:44:18 +02:00
Yann Collet
e557fd5e92 minor compression level corrections 2016-07-17 16:21:37 +02:00
Yann Collet
d54b2d23b4 minor static assert for 32/64 bits system. Suggested by @ebiggers 2016-07-17 15:53:18 +02:00
Yann Collet
972e5806ee fixed : premature frame end on zero-sized raw block - reported by @ebiggers 2016-07-17 15:39:24 +02:00
luben karavelov
10f999f856 Add legacy support for the low-level streaming API 2016-07-17 01:03:26 +02:00
Yann Collet
6cacd34d44 minor formatting changes 2016-07-15 17:58:13 +02:00
Yann Collet
f6ff53cd4e implemented dictID reserved ranges 2016-07-15 17:03:38 +02:00
Yann Collet
98c8884999 added target zstd in root Makefile 2016-07-15 16:12:38 +02:00
Yann Collet
961b6a0e34 ZSTD_compressBlock() limits block size depending on windowLog parameter 2016-07-15 11:58:49 +02:00
Yann Collet
227cc39e15 improved efficiency for large messages with small dictionaries 2016-07-15 11:27:09 +02:00
Yann Collet
ea2ecdc315 fixed issue with small dictionary 2016-07-14 23:27:31 +02:00
Yann Collet
e9ed5cdc94 fixed minor coverity warning 2016-07-14 21:02:57 +02:00
Yann Collet
b23e1ce319 removed debugging traces 2016-07-14 17:46:38 +02:00
Yann Collet
17508f1a16 fixed a few minor coverity warnings 2016-07-14 17:18:20 +02:00
Yann Collet
8847238cac simplified ZSTD_estimateCCtxSize() 2016-07-14 17:05:38 +02:00
Yann Collet
69c2cdb45c fixed conversion warning 2016-07-14 16:52:45 +02:00
Yann Collet
5e80dd3261 fixed minor coverity warnings 2016-07-13 19:21:57 +02:00
Yann Collet
3c174f4da9 fixed minor coverity warning 2016-07-13 17:25:53 +02:00
Yann Collet
2b1a3638e6 changed macro name to ZSTDCLI_CLEVEL_DEFAULT 2016-07-13 15:16:00 +02:00
Yann Collet
3c242e79d3 updated compression levels table 2016-07-13 14:56:24 +02:00
Yann Collet
fbc69f8649 changed for #245 2016-07-13 13:52:58 +02:00
Yann Collet
eed2081e55 fixed conversion warning 2016-07-12 15:11:40 +02:00
Yann Collet
a43a854cdb updated paramgrill 2016-07-12 13:42:10 +02:00
Yann Collet
73d74a05b9 fixed dfast strategy 2016-07-12 13:03:48 +02:00
Yann Collet
45dc35628c first version of doubleFast 2016-07-12 09:47:31 +02:00
Yann Collet
d158c35e9f added ZSTD_estimateDCtxSize() 2016-07-11 13:46:25 +02:00
Yann Collet
8e0ee681b8 added ZSTD_sizeofDCtx() 2016-07-11 13:09:52 +02:00
Yann Collet
3ae543ce75 added ZSTD_estimateCCtxSize() 2016-07-11 03:12:17 +02:00
Yann Collet
25c506601c promote ZSTD_getDecompressedSize() to stable API 2016-07-10 01:46:18 +02:00
Yann Collet
3b6ae77e15 comment clarification 2016-07-08 23:42:22 +02:00
Yann Collet
722e14bb65 fixed compilation error in decompression module 2016-07-08 19:22:16 +02:00
Yann Collet
bd10607063 updated spec 2016-07-08 19:16:57 +02:00
Yann Collet
c5fb5b7fcd support offset > 128 MB 2016-07-08 13:13:37 +02:00
Yann Collet
ed3845d3fa introduced ZSTD_WINDOWLOG_MAX_32 (#239), suggested by @GregSlazinski 2016-07-08 12:57:10 +02:00
Yann Collet
26f681451f updated doc 2016-07-08 11:45:08 +02:00
Yann Collet
19c27d27f1 simplified legacy functions, no longer need magic number 2016-07-07 14:40:13 +02:00
Yann Collet
e72efeb0a1 removed "error_public.h" dependency from "zstd.h" 2016-07-07 14:17:40 +02:00
Yann Collet
974f52fc5d Added "dictionary decompression" example 2016-07-07 14:08:00 +02:00
Yann Collet
e09d38e921 removed mem.h dependency from zbuff.h (experimental section) 2016-07-07 13:17:37 +02:00
Yann Collet
f323bf7d32 added : ZSTD_getDecompressedSize() 2016-07-07 13:14:21 +02:00
Yann Collet
52c04fe58f removed mem.h dependency from zstd.h (experimental section) 2016-07-07 11:53:18 +02:00
Yann Collet
f246cf5423 ZSTD_decompress_usingDDict() compatible with Legacy mode 2016-07-06 20:32:27 +02:00
Yann Collet
29652e2618 sample set limitation closer to 2 GB 2016-07-06 16:25:46 +02:00
Yann Collet
99b045b70a dictBuilder protection vs huge sample sets (>2 GB) 2016-07-06 16:12:38 +02:00
Yann Collet
445d49d898 fixed conversion warning 2016-07-06 13:27:22 +02:00
Yann Collet
a295b3170f fixed conversion warning 2016-07-06 13:13:12 +02:00
Yann Collet
517e1ba623 fixed dictBuilder issue with HC levels. Reported by Bartosz Taudul. 2016-07-06 12:35:09 +02:00
Yann Collet
fe07eaa972 simplified ZSTD_decodeSequence() 2016-07-06 02:25:44 +02:00
Yann Collet
9ca73364e6 updated spec 2016-07-05 10:53:38 +02:00
Yann Collet
f9cac7a734 Added GNU separator --, to specifies that all following arguments are necessary file names (and not commands). Suggested by @chipturner (#230) 2016-07-04 18:18:24 +02:00
Yann Collet
23f05ccc6b updated specifications 2016-07-04 16:13:11 +02:00
Yann Collet
d916c908e0 updated doc 2016-07-04 00:42:58 +02:00
Yann Collet
698cb63305 Updated specifications 2016-07-03 18:49:35 +02:00
Yann Collet
d57dffbe76 ZSTD_storeSeq takes an U32 as offset type 2016-07-03 01:48:26 +02:00
Yann Collet
302ff036f6 simplified repcodes for lazy_extDict 2016-07-03 01:28:16 +02:00
Yann Collet
9634f67107 fix lazy parser 2016-07-03 01:23:58 +02:00
Yann Collet
92d75667e4 fix for fast mode 2016-07-03 01:10:53 +02:00
Yann Collet
5e734ad09b revert fix 2016-07-02 23:55:34 +02:00
Yann Collet
0d5bf8f06f fixed risk of segfault on very large files (multiple GB) 2016-07-02 21:39:47 +02:00
Yann Collet
2fa9904844 update specification and comments 2016-07-01 20:55:28 +02:00
Yann Collet
c093208ab8 fix : potential leak (#229) 2016-06-30 14:07:30 +02:00
Yann Collet
6c6e1751f6 use ZSTD_getParams() to simplify code 2016-06-27 15:28:45 +02:00
Yann Collet
3d2cd7f816 Introduced ZSTD_getParams() 
bench now uses ZSTD_createCDict_advanced()
 2016-06-27 15:12:26 +02:00
Yann Collet
529d9c7dee updated version to v0.7.2 2016-06-27 10:03:10 +02:00
Yann Collet
d4f4e58ee1 fixed ZSTD_decompressBlock() using multiple blocks 2016-06-27 01:31:35 +02:00
Yann Collet
63b5e7a2ea Improved comments 2016-06-26 17:42:15 +02:00
Yann Collet
3755eb8fea fixed strict-aliasing warning on gcc6 2016-06-22 13:15:53 +02:00
Yann Collet
23042929da Fixed : dictBuilder fails if first sample is too small 2016-06-22 11:05:34 +02:00
Yann Collet
391a128794 fix : segfault in command line during automatic overwrite protection mode 2016-06-21 17:06:25 +02:00
Yann Collet
bda68c253b refactored ZBUFF_compressEnd() for better maintainability 2016-06-21 15:18:11 +02:00
Yann Collet
aa29226b7c fix : ZBUFF_compressEnd() gives right amount remaining to flush, including future epilogue 2016-06-21 14:04:57 +02:00
Yann Collet
f15c1cb00c Fixed : ZBUFF_compressEnd() called multiple times with too small dst buffer (#206) 2016-06-21 13:11:48 +02:00
Yann Collet
a49e066b26 clarified comments on ZSTD_compressContinue() 2016-06-21 11:54:03 +02:00
Yann Collet
d4f38d0dcd updated library to v0.7.1 2016-06-21 10:15:43 +02:00
Yann Collet
22d76322ce minor refactor 2016-06-21 08:01:51 +02:00
Yann Collet
a436a529bc minor : fast_extDict does no longer skip first byte 2016-06-20 23:34:04 +02:00
Yann Collet
4623d11571 new correction, less extreme replacement value 2016-06-20 19:15:37 +02:00
Yann Collet
5477cc25f7 fixed corruption error related to inter-blocks rep-offset 2016-06-20 18:31:25 +02:00
Yann Collet
e4811ba761 Modified : ZSTD_createDDict() accepts dictionary < 8 bytes in pure content mode (reported by @chipturner) 2016-06-19 23:06:54 +02:00
Yann Collet
06d9a73b48 minor refactor, using WILDCOPY_OVERLENGTH macro instead of hard-coded 8 2016-06-19 14:27:21 +02:00
Yann Collet
19cab46f2f Joined seqStore initialization at dispatch point 2016-06-17 12:54:52 +02:00
Yann Collet
510cff3570 minor comment change 2016-06-16 16:39:55 +02:00
Yann Collet
4948f270b3 make room for reserved "information bit" in frame header 2016-06-16 15:38:51 +02:00
Yann Collet
23ba41533a Fixed zstd_opt encoding error with repeat-offsets 2016-06-16 13:20:46 +02:00
Yann Collet
80d033fb43 fixed ptr arithmetic warning 2016-06-16 01:41:50 +02:00
Yann Collet
ad39b7a718 zdict stores standard rep-offset. 
It can use custom ones, but the proper formula and impact on statistics is not done yet.
 2016-06-16 01:14:41 +02:00
Yann Collet
736d419289 strengthened dict loading on decompresson side 2016-06-16 01:05:04 +02:00
Yann Collet
8e36a9c169 decoder restores repOffsets from dictionary 2016-06-16 01:05:04 +02:00
Yann Collet
52a0622beb RepsCodes are saved into Dict 
(uncomplete : need decompression to regenerate them)
 2016-06-16 01:05:04 +02:00
Yann Collet
efd0b4993a fixed fuzzer error (inter-block repeated offsets) 2016-06-16 00:53:56 +02:00
Yann Collet
9b998e4d08 Fixed decompression of literals in dictionary mode 2016-06-15 23:11:20 +02:00
Yann Collet
d059092897 fixed conversion warnings 2016-06-14 15:34:24 +02:00
Yann Collet
45c03c564f fixed corruption with inter-blocks repeated offsets 2016-06-14 13:46:11 +02:00
Yann Collet
4266c0a2fd adding inter-blocks rep-offsets 2016-06-14 01:49:25 +02:00
Yann Collet
43dfe01919 Check repIndex for validity 2016-06-13 21:43:06 +02:00
Yann Collet
18c8f79f3e fixed gcc warning on uninitialized structure variable 2016-06-12 22:51:52 +02:00
Yann Collet
cd98f93cff Fixed decompression issue with invalid data 2016-06-11 23:26:22 +02:00
Yann Collet
37fece22e8 enable repeat-entropic-stats mode 2016-06-11 02:52:42 +02:00
Yann Collet
d60a5bf900 Literal decompression builds Huffman tables within shared space (for later re-use) 2016-06-11 02:35:31 +02:00
Yann Collet
237ad4beb3 Added single-stream decompression variant using external DTable 2016-06-11 01:46:03 +02:00
Yann Collet
289bbd52e5 Updated huff0 2016-06-11 01:31:54 +02:00
Yann Collet
1869f7966e Merge pull request #205 from inikep/dev 
legacy decoder for v0.6
 2016-06-10 17:13:07 +02:00
Yann Collet
0974f681a4 completed .gitignore 2016-06-10 14:44:16 +02:00
Yann Collet
9dd12742f3 litBlockType_t is an enum 2016-06-10 00:12:26 +02:00
inikep
4923222412 fixed warnings from Travis 2016-06-09 20:03:30 +02:00
inikep
4000945a1d project updated for legacy decoder zstd_v06.c 2016-06-09 18:12:06 +02:00
inikep
bf853d5510 added legacy decoder for v0.6 format 2016-06-09 17:59:18 +02:00
Yann Collet
662a541431 updated huff0 - now generates a common HUF_DTable type for all decoding tables 2016-06-08 11:11:02 +02:00
Yann Collet
302fb53a76 Removed ZSTD_*_usingPrepared?Ctx() declaration from public space 2016-06-07 12:16:49 +02:00
Yann Collet
81e13ef7cf first implementation of the new dictionary API (untested) 2016-06-07 00:51:51 +02:00
Yann Collet
9d504ae85b Added decoding of RLE blocks 2016-06-06 19:52:35 +02:00
Yann Collet
2cc72f1fd3 fixed initialization issue in bench 2016-06-06 17:50:07 +02:00
Yann Collet
e3d529403d fixed initialization mismatch in ZSTD_copyCCtx() 2016-06-06 11:07:33 +02:00
Yann Collet
142acbdea7 fixed minor visual conversion warning 2016-06-06 00:46:56 +02:00
Yann Collet
673f0d7cdc new frame format, allowing custom window size 2016-06-06 00:26:38 +02:00
Yann Collet
89703d20fb reduced dependencies 2016-06-05 01:50:33 +02:00
Yann Collet
51778b7cca updated README following merging of *_static.h 2016-06-05 01:38:10 +02:00
Yann Collet
a91ca620cf removed HUF_readStats() from public space 2016-06-05 01:33:55 +02:00
Yann Collet
d0e2cd15cb Merged fse_static into fse.h . Now requires FSE_STATIC_LINKING_ONLY macro. 2016-06-05 00:58:01 +02:00
Yann Collet
130fe11394 merged huf_static.h into huf.h . Requires HUF_STATIC_LINKING_ONLY macro. 2016-06-05 00:42:28 +02:00
Yann Collet
dc048d18d3 minor comment (detailing an #include motivation) 2016-06-05 00:32:23 +02:00
Yann Collet
49bb0041af removed ZSTD_highbit() from zstd_internal.h, as it is only used by zstd_compress.c 2016-06-04 20:17:38 +02:00
Yann Collet
d3b7f8d21f Merged zstd_static.h into zstd.h . Now requires ZSTD_STATIC_LINKING_ONLY macro 2016-06-04 19:47:02 +02:00
Yann Collet
ac110a1f21 Removed ZBUFF internal util function from public area 2016-06-04 19:16:49 +02:00
Yann Collet
5347aee8f7 merged zbuff_static.h into zbuff.h . Now requires ZBUFF_STATIC_LINKING_ONLY macro 2016-06-04 19:12:48 +02:00
Yann Collet
e69b8ccceb merged zdict_static.h into zdict.h. Now requires ZDICT_STATIC_LINKING_ONLY macro. 2016-06-04 18:56:23 +02:00
Yann Collet
198d127b35 minor comment change (unfinished description of new header format) 2016-06-04 18:40:55 +02:00
Yann Collet
f4f5affdf7 restore ZBUFF full-block-size, for better performance on small input 2016-06-03 23:09:28 +02:00
Yann Collet
ab7b6f1ece Merge pull request #198 from inikep/dev070 
Dev070
 2016-06-03 21:37:49 +02:00
inikep
3640396b1a fixed: deallocation of structures in case of error in ZBUFF_createCCtx and ZBUFF_createDCtx 2016-06-03 16:36:50 +02:00
Yann Collet
fe48775868 minor decoder code refactoring 2016-06-03 15:41:51 +02:00
inikep
2a74609b90 zlibWrapper: ZWRAP_createCCtx and ZWRAP_freeCCtx use custom memory allocation functions 2016-06-03 14:53:51 +02:00
inikep
3763c77f6b defaultCustomNULL replaced with defaultCustomMem 2016-06-03 13:28:20 +02:00
inikep
36fac00149 removed calloc calls from lib/ 2016-06-03 13:23:04 +02:00
inikep
db2f540414 added defaultCustomNULL 2016-06-03 12:56:56 +02:00
inikep
b74a468fad Merge remote-tracking branch 'refs/remotes/Cyan4973/dev070' into dev070 2016-06-02 22:09:09 +02:00
Yann Collet
923938edde Added -Wdeclaration-after-statement compilation flag 2016-06-02 17:56:00 +02:00
inikep
ff9114aee3 zlibWrapper: added support for custom memory allocation functions 2016-06-02 16:52:36 +02:00
inikep
c4807f4d2f default custom allocation functions moved to zstd_internal.h 2016-06-02 15:11:39 +02:00
inikep
2866951558 opaque parameter for custom memory allocation functions 2016-06-02 13:04:18 +02:00
inikep
9242816b56 fparamsPtr->windowLog==0 means that a frame is skippable 2016-06-01 18:47:04 +02:00
Yann Collet
70d1301d6e Changed ZSTD_adjustCParams() prototype 
`ZSTD_adjustCParams()` is now automatically invoked at the end of `ZSTD_getCParams()`
 2016-06-01 18:45:34 +02:00
Yann Collet
83c3f4427c upgraded zbufftest to also test advanced frame parameters no/checksum no/dictID 2016-06-01 17:44:53 +02:00
inikep
13f42d9085 VS2010 project: reverted zstdlib.rc 2016-06-01 14:44:31 +02:00
inikep
5c2771710d Merge remote-tracking branch 'refs/remotes/Cyan4973/dev070' into dev070 
# Conflicts:
#	.gitignore
#	lib/decompress/zstd_decompress.c
#	programs/zbufftest.c
 2016-06-01 09:16:11 +02:00
Yann Collet
202844ebd0 fixed zbufftest : 2016-06-01 00:44:36 +02:00
Yann Collet
8e3a36a6db decompression validates frame content checksum 2016-06-01 00:18:28 +02:00
inikep
a6b942018d Merge remote-tracking branch 'refs/remotes/origin/dev' into dev070 
# Conflicts:
#	.travis.yml
#	Makefile
#	lib/common/zstd_static.h
#	programs/Makefile
#	projects/VS2008/zstd/zstd.vcproj
#	projects/VS2008/zstdlib/zstdlib.vcproj
#	projects/cmake/lib/CMakeLists.txt
#	projects/cmake/programs/CMakeLists.txt
 2016-06-01 00:07:09 +02:00
Yann Collet
f2a3b6e7b4 added : frame content checksum 2016-05-31 22:23:45 +02:00
inikep
43aa9fe8b3 fixed skippable frame 2016-05-31 19:36:51 +02:00
inikep
f772bf54a5 support for skippable frames 2016-05-31 12:43:46 +02:00
Giuseppe Ottaviano
370b751e24 Expose function to add entropy tables to pre-built dictionary. 
In some cases a custom dictionary building algorithm tailored for a specific
input can be more effective than the one produced by `ZDICT_trainFromBuffer`,
but with the current API it's not possible encode the entropy tables into the
custom-built dictionary.

This commit extracts the logic to add entropy tables to a dictionary from
`ZDICT_trainFromBuffer` and exposes it as a function
`ZDICT_addEntropyTablesFromBuffer`.
 2016-05-30 19:50:09 -07:00
Yann Collet
290aaa7521 Added : ability to manually select the dictionary ID of a newly created dictionary 2016-05-30 21:18:52 +02:00
Yann Collet
30009521d7 fuzzer tests dictBuilder. 
Added : ability to not store dictID during compression; decompression doesn't check dictID then
 2016-05-30 16:17:33 +02:00
Yann Collet
c0a9bf3c2e minor code refactoring 2016-05-30 04:48:32 +02:00
Yann Collet
c46fb924df added dictionary ID (incomplete) 2016-05-29 05:01:04 +02:00
Yann Collet
f51e0660f4 Simplified list of *.c files 2016-05-29 01:39:19 +02:00
Yann Collet
0c5e8b17ad moved xxhash to lib/common 2016-05-29 01:06:30 +02:00
inikep
957823f56f zstdcli: -r (operate recursively on directories) works with dictBuilder and compression 2016-05-25 15:30:55 +02:00
inikep
e75909ef93 ZBUFF_createCCtx_advanced moved to zbuff_static.h 2016-05-25 11:31:16 +02:00
inikep
02c244bf78 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev 2016-05-24 17:15:04 +02:00
inikep
fb5df613cf zstd_stats.h included only in debug mode 2016-05-24 15:36:37 +02:00
inikep
71dd67cf57 fullbench: tests for ZBUFF_createCCtx_advanced and ZBUFF_createDCtx_advanced 2016-05-24 10:57:14 +02:00
inikep
13ba880b49 added ZBUFF_createCCtx_advanced and ZBUFF_createDCtx_advanced 2016-05-23 17:04:23 +02:00
inikep
107e243195 added ZSTD_createDCtx_advanced 2016-05-23 16:24:52 +02:00
inikep
50e82c015d added ZSTD_createCCtx_advanced 2016-05-23 15:49:09 +02:00
inikep
35b891c5b0 zstd_stats.h updated to v0.6 2016-05-20 19:42:20 +02:00
Yann Collet
f22a0d653d huff0 dynamic reduction 2016-05-20 14:36:36 +02:00
Yann Collet
45f84ab7cc all programs use ZSTD_VERSION_STRING from zstd.h to display their version 2016-05-20 12:34:40 +02:00
Yann Collet
de4d4da52d changed version to 0.6.2 
removed make-controlled version (now used from zstd.h)
removed std=c99 by default
added : c90/gno90/c99/gnu99/c11 tests
 2016-05-20 12:20:12 +02:00
Yann Collet
7591a7f224 minor refactoring 2016-05-20 11:44:43 +02:00
Yann Collet
236d94fa9a reverted default compression level to 1 2016-05-18 12:06:33 +02:00
Yann Collet
78aa05228a fixed Visual warning in debug mode 2016-05-17 14:30:19 +02:00
Yann Collet
c8e9c4046f Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-05-17 12:02:35 +02:00
Yann Collet
ea4daa308d added ZSTDv05_getFrameParams() to "zstd_v05.h" 2016-05-17 12:01:55 +02:00
klemens
268775dae8 spelling-fixes, as of lintian.debian.org 2016-05-16 22:23:24 +02:00
Yann Collet
d9b3cca4d3 restore openVMS support (detected by @thatsafunnyname) 2016-05-13 12:43:36 +02:00
inikep
63ecd747de added common/entropy_common.c 2016-05-13 11:27:56 +02:00
Yann Collet
ccd6e86d45 improved decoding speed by inlining x_isError() functions 2016-05-12 15:55:26 +02:00
Yann Collet
16871680e3 minor refactor : more accurate variable scope 2016-05-12 14:27:00 +02:00
Yann Collet
b19d5d76ec restored inline for better decompression speed 2016-05-12 14:06:31 +02:00
Yann Collet
1ceb5a9295 restored DStream init 
due to strange speed regressions and warnins on gcc 5.3
 2016-05-12 13:50:13 +02:00
Yann Collet
ce27e85209 fixed clang conversion warning 2016-05-11 18:55:27 +02:00
Yann Collet
1032fbe714 update FSE library 2016-05-11 18:30:24 +02:00
Yann Collet
249ae0ca99 better big-endian / little-endian r/w support 2016-05-11 16:38:18 +02:00
Yann Collet
3329d98df9 fixed minor asan warning in legacy decoder 2016-05-11 11:07:54 +02:00
Yann Collet
c75e4c25e8 Added : ZSTD_getErrorString(), to get error string from error enum (#168) 2016-05-10 17:47:11 +02:00
Yann Collet
b3a9938d0c Merge pull request #182 from inikep/dev 
support for Visual Studio 2012, 2013, 2015
 2016-05-10 17:18:12 +02:00
Yann Collet
bd39d54576 zbuff supports byte-by-byte decompression scenarios 2016-05-10 14:14:19 +02:00
inikep
ed9a08538c Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev 
# Conflicts:
#	lib/common/util.h
#	programs/paramgrill.c
#	visual/2013/fullbench/fullbench.vcxproj.filters
#	visual/2013/fuzzer/fuzzer.vcxproj.filters
 2016-05-10 13:20:01 +02:00
Yann Collet
918b934a26 minor comment update 2016-05-10 07:00:07 +02:00
inikep
349fcf7ea2 fixed Linux compilation 2016-05-09 18:04:07 +02:00
inikep
aaaf923a17 common/util.h moved to programs/util.h 
decompress/fse_decompress.c moved to common/
 2016-05-09 16:19:25 +02:00
inikep
d44ec48b1c detection of snprintf for Visual <= 2012 2016-05-09 14:37:41 +02:00
Yann Collet
59b6ba7677 Merge pull request #180 from thatsafunnyname/patch-1 
No stdint.h on OpenVMS.
 2016-05-09 11:56:42 +02:00
Yann Collet
4ff73a2eff added : midipix support (#181) 2016-05-09 11:44:00 +02:00
Yann Collet
d275236e6a Fixed Windows API, as detected by @KrzysFR (#171) 2016-05-09 08:19:27 +02:00
Yann Collet
f6ca09b5ff Reduced console display on loading lots of files with zstd --train. Reported by @KrzysFR, see #177 2016-05-09 04:44:45 +02:00
Yann Collet
e1a69a88c0 Expose Advanced Streaming API for v0.5.x legacy 2016-05-08 09:13:40 +02:00
Yann Collet
a5dacdf551 updated fse (minor refactor) 2016-05-06 16:55:27 +02:00
Yann Collet
18dedece91 Support legacy ZSTD_decompress_usingDict() (starting v0.5+) 2016-05-06 16:43:23 +02:00
Yann Collet
8283a2f0aa Fixed decoding corruption error with quad-symbols huffman on legacy decoders 2016-05-06 01:51:31 +02:00
inikep
13c8424ea0 code cleaning 2016-05-05 13:58:56 +02:00
inikep
0bd0faec32 fixed compatibility issues 2016-05-05 13:10:57 +02:00
Yann Collet
6d1d25299a fixed Huff0 quad-symbols decoder (#173) 2016-05-05 12:41:36 +02:00
inikep
9c22e57bfb Compiler Options moved to util.h 2016-05-05 11:53:42 +02:00
Yann Collet
ddb8ebd5b3 Stricter gnu90 tests 2016-05-05 04:59:53 +02:00
Yann Collet
3f8ed50bf1 Added gnu90 test 
work-around : disabled automatic selection of huff0 quad-decoder (see #173)
 2016-05-05 03:01:13 +02:00
inikep
3163403855 UTIL_prepareFileList 2016-05-05 00:25:38 +02:00
thatsafunnyname
b0e5427d1a No stdint.h on OpenVMS. 
No stdint.h on OpenVMS, it is inttypes.h instead, some projects use HAVE_STDINT_H.
 2016-05-04 15:43:42 +01:00
Yann Collet
6f9c056662 Fixed BMI behavior on x86 and x32 targets, reported by @tycho (#178) 2016-05-01 10:26:30 +02:00
inikep
55d047aa92 getTotalFileSize moved to common/util.h 2016-04-28 16:50:13 +02:00
inikep
d5ff2c3d9a ordering of #include 2016-04-28 14:40:45 +02:00
inikep
83c76b4594 bench.c: time functions moved to common/util.h 2016-04-28 13:16:01 +02:00
inikep
69fcd7c0ae getFileSize moved to common/util.h 2016-04-28 12:23:33 +02:00
inikep
1007a1fe3d get rid of some Visual C++ warnings 2016-04-25 15:23:09 +02:00
inikep
6cb083fe3d fixed 64-bit compilation with Visual C++ 2016-04-25 14:42:15 +02:00
inikep
ea4ee3eee0 added compatibility with Visual C++ 2012 2016-04-25 13:09:06 +02:00
inikep
a8138fd767 updated lib/README.md part 2 2016-04-25 11:36:44 +02:00
inikep
49794316c2 updated lib/README.md 2016-04-25 11:31:28 +02:00
inikep
f066816998 fix for g++ compilation 2016-04-22 18:54:05 +02:00
inikep
db3964382c introduced ZSTD_NODECOMPRESS to link only compressor 2016-04-22 18:22:30 +02:00
inikep
a1febea01d Error functions moved to common/zstd_common.c 2016-04-22 17:14:25 +02:00
inikep
3c7c3527d0 introduced ZSTD_NOCOMPRESS to generate decompressor only 2016-04-22 13:59:05 +02:00
inikep
23a0889301 separation of lib/ into common/, compress/, decompress/, dictBuilder/, legacy/ 2016-04-22 12:43:18 +02:00
inikep
4699b3be03 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev 2016-04-21 12:30:58 +02:00
inikep
38654988f3 minor speed improvements 2 
bench.c: block size has to be bigger than 32 bytes
zstdcli.c: support for e.g. -B16k -B16m
 2016-04-21 12:18:47 +02:00
inikep
ef51941822 zst_opt.h: minor compression speed improvement 2016-04-21 11:08:43 +02:00
Yann Collet
e02808ffc5 bumped lib version to 0.6.1 
added ZSTD_VERSION_STRING
 2016-04-20 22:46:16 +02:00
inikep
c5e1d295ee bench.c: force at least one compression and decompression loop 
fix for -i0 with small files
 2016-04-19 09:37:59 +02:00
inikep
2c5eeea635 introduced ZSTD_DEAFULT_CLEVEL for (compressionLevel<=0) 2016-04-15 13:44:46 +02:00
inikep
4b3c5eed99 removed ZSTD_compressBegin_targetSrcSize 2016-04-14 13:43:51 +02:00
inikep
c70c90e8a3 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev 2016-04-13 10:48:26 +02:00
Yann Collet
16748729c9 Updated DICT_MAGIC, since internal dictionary format has changed (length encoding) 2016-04-12 23:26:23 +02:00
Yann Collet
c5b6d9741a Added : zbuff.c to library; reported by Yipei Zhang 2016-04-12 21:46:18 +02:00
Yann Collet
389648cc6d Fixed : contentSize + copyCCtx() 2016-04-12 19:13:08 +02:00
Yann Collet
541dc7c7d4 Added : fuzzer test, checking contentLength value after copyCCtx() usage 2016-04-12 18:00:20 +02:00
Yann Collet
6236eba991 removed headerBuffer 2016-04-12 15:52:33 +02:00
Yann Collet
0085cd3138 created ZSTD_writeFrameHeader() 2016-04-12 14:14:10 +02:00
Yann Collet
abb5c653f1 minor speed improvement for ZSTD_copyCCtx() 2016-04-11 20:42:31 +02:00
Yann Collet
887e7da785 Fixed copyCCtx() on uninitialized cctx (reported by Oliver Lange) 2016-04-11 20:12:27 +02:00
inikep
8844e21dff Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev 2016-04-11 16:49:27 +02:00
inikep
f7d210b2e9 cache literal prices for ZSTD_btopt 2016-04-11 16:35:13 +02:00
Yann Collet
582933fde5 fixed round-trip corruption, reported by Oliver Lange 2016-04-11 16:25:56 +02:00
Yann Collet
125d81774f fixed uninitialization error, reported by Maciej Adamczyk 2016-04-11 14:22:56 +02:00
Yann Collet
d673d4cf66 fixed uninitialization error, reported by Maciej Adamczyk 2016-04-11 14:10:23 +02:00
Yann Collet
ccbba296e1 Fix infinite loop with invalid data (fuzzer test) 2016-04-10 14:42:26 +02:00
Yann Collet
793c649ead updated cLevels 2016-04-09 20:32:00 +02:00
Yann Collet
b144bc3e1c Fixed uninitialized warning, reported by Maciej Adamczyk 2016-04-09 17:26:22 +02:00
Yann Collet
9990986bdc fixed minor warnings when compiling with gcc -g (aka -std=gnu90) 2016-04-09 16:17:18 +02:00
Yann Collet
029267ab3f Integrated Legacy v0.5.x decoder (provided by @inikep) 2016-04-09 09:42:27 +02:00
Yann Collet
7eff39f7ea fixed decoding error 2016-04-09 01:51:36 +02:00
Yann Collet
ea63bb7b5e converted fuzzer timer to clock_t 2016-04-08 15:25:32 +02:00
Yann Collet
78267d1abe updated cLevel for block <= 256K 2016-04-08 12:36:19 +02:00
Yann Collet
72cc704514 Merge branch 'dev060' of github.com:Cyan4973/zstd into dev060 2016-04-08 02:02:31 +02:00
Yann Collet
0dbf2874ee faster level 1 at 256 KB 2016-04-08 02:02:12 +02:00
Yann Collet
04a4fd638b Merge pull request #164 from inikep/dev 
only repcode 0 for lazy, lazy2 and btlazy2 strategies
 2016-04-07 20:46:13 +02:00
inikep
afe1f7994b Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev 
# Conflicts:
#	lib/zstd_compress.c
 2016-04-07 19:50:03 +02:00
Yann Collet
9e8b09a7bd Fixed memory initialization issue, reported by Maciej Adamczyk 2016-04-07 19:35:23 +02:00
inikep
2bf79f9bff use extDict only when needed 2016-04-07 19:24:33 +02:00
inikep
64d7bcb0dc only repcode 0 for lazy, lazy2 and btlazy2 strategies 2016-04-07 19:14:09 +02:00
inikep
a69640cca1 Revert "removed void ZSTD_compressBlock_greedy" 
This reverts commit 9296400a69.
 2016-04-07 18:22:29 +02:00
Yann Collet
d7bee33caf Fixed make install (reported by Oliver Lange) 2016-04-07 18:16:42 +02:00
Yann Collet
5d39357bc4 minor compression speed optimization 2016-04-07 17:19:00 +02:00
inikep
9296400a69 removed void ZSTD_compressBlock_greedy 2016-04-07 16:58:04 +02:00
Yann Collet
485371600a Implemented default coding table for lengths and offset 2016-04-07 15:24:29 +02:00
Yann Collet
79459da14b Merge pull request #163 from inikep/dev 
3 repcodes and MINMATCH=3
 2016-04-07 14:37:30 +02:00
inikep
81a345b0ce minor speed improvements 2016-04-07 11:35:17 +02:00
inikep
e47f33f980 reordered code 2016-04-06 20:58:00 +02:00
inikep
2db1eb78e9 reordered functions 2016-04-06 17:14:19 +02:00
inikep
e5b6e9e551 fixed support for searchLength=4 2016-04-06 13:15:38 +02:00
inikep
75716851d4 MINMATCH=3 for lazy strategy 2016-04-06 12:34:42 +02:00
inikep
7bc19b6b3e MINMATCH=3 2016-04-06 09:46:01 +02:00
inikep
f8a339d274 working ZSTD_compressBlock_opt_extDict_generic 2016-04-05 23:58:51 +02:00
inikep
83417bbbd2 code formatting 2016-04-05 21:30:03 +02:00
inikep
5ce00aee37 ZSTD_compressBlock_opt_extDict_generic in progress 2016-04-05 21:03:43 +02:00
inikep
faa8d8aae1 introduced ZSTD_compressBlock_greedy 2016-04-05 19:01:10 +02:00
inikep
908fcb384a 3 repcodes with ZSTD_compressBlock_lazy_extDict_generic 2016-04-05 18:16:38 +02:00
inikep
a4385aa23b ZSTD_REP_NUM=3 only 2016-04-05 17:11:56 +02:00
inikep
d5359b2e55 Merge remote-tracking branch 'refs/remotes/origin/repcodes' into dev 2016-04-05 17:07:36 +02:00
inikep
ba73d1bb6b support for ZSTD_REP_NUM=3 2016-04-05 10:30:05 +02:00
inikep
74e3a7ba4c removed kSlotNew 2016-04-05 10:18:37 +02:00
inikep
b3532793a9 experiments with ZSTD_decodeSequence 2016-04-05 10:08:25 +02:00
Yann Collet
46bdb6267e Merge pull request #161 from inikep/dev 
Dev
 2016-04-05 09:59:59 +02:00
inikep
2ec2698ef9 working 4 reps 2016-04-05 09:34:13 +02:00
inikep
19140cd80e zstd_stats.h moved to .debug/ 2016-04-05 08:52:57 +02:00
inikep
d6f208badf improved logging 2016-04-04 21:15:23 +02:00
inikep
78e52044de Merge remote-tracking branch 'refs/remotes/origin/dev' into repcodes 
# Conflicts:
#	lib/zstd_compress.c
#	lib/zstd_decompress.c
#	lib/zstd_internal.h
#	lib/zstd_opt.h
#	programs/bench.c
 2016-04-04 16:28:40 +02:00
inikep
cb70c8ab90 fixed -Werror,-Wconversion warning 2016-04-04 15:43:45 +02:00
inikep
0c7456c5b7 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev 
# Conflicts:
#	lib/zstd_compress.c
#	lib/zstd_static.h
 2016-04-04 14:54:53 +02:00
Yann Collet
b59bf963a8 fixed Visual conversion warnings 2016-04-04 14:53:16 +02:00
inikep
227dd4bc74 changed price estimation for optimal parser 2016-04-04 14:34:02 +02:00
Yann Collet
8a57b928ba changed contentLog -> chainLog 2016-04-04 13:49:18 +02:00
Yann Collet
1005fc1276 reduced memory usage of very hich compression levels in 32-bits mode 2016-04-04 13:28:28 +02:00
inikep
19bd48f516 Merge from Cyan4973/dev060 2016-04-04 12:10:00 +02:00
Yann Collet
1535414927 Fixed 32-bits windowLog limitation 
Fixed some macro errors in zbufftest
 2016-04-04 04:22:53 +02:00
Yann Collet
5a88a70f97 fixed buffer sync error in zbuff 2016-04-03 20:24:25 +02:00
Yann Collet
ef36390e4a refined exception for hashlog < min 2016-04-02 00:46:40 +02:00
Yann Collet
40358d01aa Fix : rare bugs when hashLog < min in bt mode 2016-04-02 00:40:09 +02:00
Yann Collet
1d803d8deb fixed zbuff error 775-3 (internal buffer management) 2016-04-01 18:57:41 +02:00
Yann Collet
efb18309ef fixed minor visual 32-64bits conversion warning 2016-04-01 18:54:13 +02:00
Yann Collet
afa60e0f14 fixed decoding error, reported by Maciej Adamczyk 2016-04-01 15:49:41 +02:00
Yann Collet
27caf2afca minor variable isolation 2016-04-01 15:48:48 +02:00
Yann Collet
9bb87e5995 fixed harmless comparison warning 2016-03-30 21:28:15 +02:00
Yann Collet
51d500425f fixed paramgrill 2016-03-30 20:42:19 +02:00
Yann Collet
dd6466a211 fixed clang conversion warning 2016-03-30 20:06:26 +02:00
Yann Collet
3b71925c2d separate params into compressionParams and frameParams 2016-03-30 19:48:05 +02:00
Yann Collet
21588e370b changed validateParams() into checkParams() + adjustParams() 2016-03-30 16:50:44 +02:00
Yann Collet
b44be74244 Fixed zdict 
more dictionary compression tests
 2016-03-26 20:52:14 +01:00
Yann Collet
74bd11954b fixed ARM compatibility 2016-03-26 17:50:26 +01:00
Yann Collet
b9151406de fixed 32-bits compatibility 2016-03-26 17:18:11 +01:00
inikep
97c88e716c fix for -Wshorten-64-to-32 warning in ZSTD_statsUpdatePrices 2016-03-25 13:39:14 +01:00
Yann Collet
433a5cce7e improved decompression speed (680) 2016-03-25 11:43:48 +01:00
inikep
5cc4efdaf8 created zstd_stats.h 2016-03-25 10:52:25 +01:00
Yann Collet
646693e3be support for alternate offset (fusion) 2016-03-24 02:42:15 +01:00
Yann Collet
b21ce15efe minor variables isolation 2016-03-24 02:02:32 +01:00
Yann Collet
7cbe79ab25 fixed write overflow found by fuzzer 2016-03-23 22:31:57 +01:00
Yann Collet
72d706a020 fixed crash at -O3 with customized block size (wrong alignment) 2016-03-23 20:44:12 +01:00
inikep
eaba91a161 added ZSTD_compressBegin_targetSrcSize 2016-03-23 20:30:26 +01:00
Yann Collet
862a85976f Added BMI instructions 2016-03-23 18:45:23 +01:00
inikep
7adceef974 hashLog3 added to ZSTD_CCtx 2016-03-23 15:53:38 +01:00
Yann Collet
6cf45dac81 shared const BIT_mask 2016-03-23 14:41:54 +01:00
Yann Collet
3c017867de minor optimization (DSpeed 665) 2016-03-23 14:09:51 +01:00
Yann Collet
2512597576 minor reordering (DSpeed 665) 2016-03-23 14:00:09 +01:00
Yann Collet
afab02098a improved decoding speed (660) 2016-03-23 13:57:49 +01:00
inikep
472638c861 added mili_sleep and setHighPriority 2016-03-23 12:28:28 +01:00
Yann Collet
9c34df93b6 new decodeSequence, merging parts 2016-03-23 01:54:25 +01:00
Yann Collet
add08d6f61 minor variation - DSpeed at 640 2016-03-23 01:32:41 +01:00
Yann Collet
be391438ff first working version with both encoder and decode alternate LL + ML coding scheme. 
decompression speed highly impacted
 2016-03-22 23:19:28 +01:00
inikep
2872b6f01f Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev 
# Conflicts:
#	lib/zstd_compress.c
#	programs/bench.c
 2016-03-22 14:38:34 +01:00
Yann Collet
fadda6c875 first prototype with ML encoding scheme (but not yet decoding scheme) 2016-03-22 12:14:26 +01:00
inikep
2b942881ac improved 4reps in ZSTD_compressBlock_lazy_generic 
kSlotNew = 0
 2016-03-22 11:56:22 +01:00
Yann Collet
1dcf930f72 Merge branch 'dev060' into alternateLLcode 2016-03-22 00:29:57 +01:00
inikep
f2fa0e1198 final repcodes for ZSTD_btopt 2016-03-21 20:14:03 +01:00
inikep
378be62a9e better repcodes 2016-03-21 19:48:54 +01:00
inikep
fc1d7bd93b changed update logic of repcodes 2016-03-21 19:02:16 +01:00
inikep
e7ec8f2b9b improved ratio 2016-03-21 17:23:02 +01:00
Yann Collet
be4605d851 improved decompression speed 2016-03-21 14:29:26 +01:00
Yann Collet
e91477c171 minor variable isolation 2016-03-21 14:22:38 +01:00
Yann Collet
b0aec17a90 code simplification (but reduce decompression speed ...) 2016-03-21 13:24:16 +01:00
Yann Collet
6c62b7dfc8 support default LL distribution 2016-03-21 00:39:19 +01:00
Yann Collet
d64f435f63 handles litLength >= 65535 2016-03-21 00:07:42 +01:00
Yann Collet
597847a2ae first emulation 2016-03-20 19:14:22 +01:00
Yann Collet
ecabfe3777 Improved index overflow protection 2016-03-20 16:20:06 +01:00
Yann Collet
5244738046 minor code refactoring 2016-03-20 16:00:00 +01:00
Yann Collet
de406eebcd minor code refactor 2016-03-20 15:46:10 +01:00
Yann Collet
516ba88022 updated huff0 2016-03-20 05:40:39 +01:00
Yann Collet
a5c2c08c68 code refactoring 2016-03-20 01:09:18 +01:00
Yann Collet
70e4577179 minor code refactoring 2016-03-19 18:08:32 +01:00
Yann Collet
c6eea2b2f2 Improved speed for small files 2016-03-19 17:18:00 +01:00
Yann Collet
e74215e3c7 performance fix for small files 2016-03-19 16:09:09 +01:00
inikep
1269e2f7db working kSlotNew for ZSTD_btopt 2016-03-19 15:32:21 +01:00
Yann Collet
bb604486f1 fixed clang warning 2016-03-19 15:18:42 +01:00
Yann Collet
37f3d1b2ab minor refactor 2016-03-19 15:11:42 +01:00
Yann Collet
90a03a95a1 minor performance fix (decompression) 2016-03-19 14:47:31 +01:00
Yann Collet
01e5b95199 code refactor 2016-03-19 14:14:31 +01:00
Yann Collet
d1d210f3fb minor code refactor 2016-03-19 12:12:07 +01:00
Yann Collet
4db09efde4 simplifications 2016-03-18 22:23:49 +01:00
Yann Collet
39c596ca58 minor simplifications 2016-03-18 21:40:56 +01:00
inikep
fade9b7ae3 ZSTD_btopt with 4 repcodes 2016-03-18 16:10:35 +01:00
inikep
1c1e6916e4 improved timer accuracy 2016-03-18 12:49:44 +01:00
Yann Collet
a910dc8238 minor code refactoring 2016-03-18 12:37:45 +01:00
inikep
5b15865413 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into repcodes 
# Conflicts:
#	lib/zstd_decompress.c
#	programs/bench.c
 2016-03-18 11:14:58 +01:00
inikep
eca2ec0b3b introduced ZSTD_REP_MOVE and ZSTD_REP_INIT 2016-03-18 11:03:43 +01:00
inikep
c56e32e782 4 repcodes when litLen==0 2016-03-17 19:53:38 +01:00
inikep
3e743052cd kSlotNew 2016-03-17 17:24:17 +01:00
Yann Collet
3fd164e656 minor refactoring 2016-03-17 13:43:32 +01:00
Yann Collet
b40287aaef Improved decompression speed (by @inikep) 2016-03-17 12:11:16 +01:00
inikep
7d1742825b support for ZSTD_REP_NUM == 1 and ZSTD_REP_NUM == 4 2016-03-17 11:21:00 +01:00
inikep
5945308983 working 4 reps 2016-03-16 15:35:14 +01:00
Yann Collet
569b81adb1 changed BLOCKSIZE into ZSTD_BLOCKSIZE_MAX 2016-03-16 15:26:51 +01:00
inikep
62d694795d ZSTD_compressBlock_lazy supports more reps 2016-03-16 12:57:07 +01:00
inikep
0ac4a551b9 ZSTD_REP_NUM removed from ZSTD_storeSeq 2016-03-16 12:03:07 +01:00
inikep
df164e6d35 introduced U32 rep[ZSTD_REP_NUM] 2016-03-16 11:56:59 +01:00
inikep
4569667c85 ZSTD_REP_NUM 2016-03-16 11:40:09 +01:00
Yann Collet
6cb41356e8 update doc / comments 2016-03-15 15:47:38 +01:00
Yann Collet
346bffbc68 Added frame content size test within fuzzer 2016-03-15 15:24:52 +01:00
Yann Collet
09b21ee2f6 fix fileio bug with new zbuff 
simplified zbuff decoding
 2016-03-15 12:56:03 +01:00
inikep
d700a1a8b9 ZSTD_setAdditionalParam 2016-03-15 12:18:44 +01:00
Yann Collet
1c2c2bcaff minor refactoring 2016-03-15 01:33:36 +01:00
Yann Collet
d1b26849e5 Frame content size supported 2016-03-15 01:24:33 +01:00
inikep
5fdd0b46d3 added additionalParam 2016-03-14 19:51:11 +01:00
Yann Collet
fb7973580f minor comments update 2016-03-13 11:08:40 +01:00
Yann Collet
03ea59b17b Decompression can decode frame content size 2016-03-12 01:25:40 +01:00
Yann Collet
0e491c01fe created ZSTD_frameParams 2016-03-11 21:58:04 +01:00
inikep
cc6c283633 removed hashLog3 from ZSTD_parameters 2016-03-10 15:17:24 +01:00
inikep
260a9e3886 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev 2016-03-10 15:04:12 +01:00
inikep
b0c640d934 small ratio improvement with MINMATCH=3 2016-03-10 13:18:36 +01:00
Yann Collet
da5be72fda update compression level parameters 2016-03-10 11:01:44 +01:00
Yann Collet
dcc000f0eb minor doc clarifications 2016-03-10 01:09:41 +01:00
Yann Collet
e3193c4081 update compresson levels 2016-03-09 16:57:09 +01:00
Yann Collet
4f013a2281 Merge pull request #142 from inikep/dev 
Dev
 2016-03-09 16:29:42 +01:00
inikep
b5a519f43c improved compression speed 
introduced cache in ZSTD_setLog2Prices
 2016-03-09 15:45:01 +01:00
Yann Collet
95cd0c2e26 frame format can support window sizes up to 128 MB 2016-03-08 18:24:21 +01:00
Yann Collet
ff14b351f5 Merge pull request #141 from inikep/dev 
fix update for zstd_opt
 2016-03-08 14:41:04 +01:00
Yann Collet
d3a6660ad1 Update clevel table for <= 128K 2016-03-08 14:12:46 +01:00
Yann Collet
2e91dde43e improved memory size evaluation by paramgrill 2016-03-08 12:22:11 +01:00
inikep
48413765b4 fixed update of hashTable for matches with length of 4+ 2016-03-08 11:03:55 +01:00
Yann Collet
dd54bbc184 Fixed large NbSeq > 32 K 
Added a test in Fuzzer to check NbSeq > 32 K
 2016-03-08 02:35:34 +01:00
Yann Collet
370b08e840 fix compression ratio for blocks <= 16 KB 2016-03-08 00:03:59 +01:00
Yann Collet
fb7ab8b2b8 Merge pull request #140 from inikep/dev 
mml3
 2016-03-07 18:06:48 +01:00
inikep
4f7f79ef9d fixed update of nextToUpdate 2016-03-07 16:14:58 +01:00
inikep
e2446b0e36 added support for MINMATCH=3 on big endian architecture 
changed ZSTD_HASHLOG3_MIN to 2 (4 bytes for MINMACH > 3)
 2016-03-07 10:07:08 +01:00
inikep
89c9e1a847 added missing "case 3:" in ZSTD_BtGetAllMatches 2016-03-06 23:21:52 +01:00
Yann Collet
1df2594bfe new compression level scale for <= 16KB 2016-03-05 18:43:21 +01:00
inikep
01060bc477 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev 
# Conflicts:
#	lib/zstd_compress.c
 2016-03-04 20:10:09 +01:00
inikep
f3c6503e55 resolve conflict in zstd_internal.h 2016-03-04 20:04:25 +01:00
inikep
e29caf7d1c code cleanup 2016-03-04 19:52:23 +01:00
inikep
805d2a7465 removed zstd_opt_internal.h 2016-03-04 19:31:57 +01:00
Yann Collet
944d0d2285 updated fse version 2016-03-04 19:26:59 +01:00
inikep
4ab9c913a5 MEM_readMINMATCH replaced with inline function 2016-03-04 19:17:31 +01:00
Yann Collet
4c01580362 finer searchLength validation 2016-03-04 17:07:54 +01:00
Yann Collet
d409db68d3 Support for nbSeq > 32767 
tamed -Wstrict-aliasing warning
 2016-03-04 14:45:31 +01:00
Yann Collet
1435335342 Merge pull request #138 from inikep/dev 
reduce stack usage with optimal parsing mode
 2016-03-03 02:08:34 +01:00
inikep
5cccd77c55 fix clang -Wcast-align warning (part 2) 2016-03-02 20:37:49 +01:00
inikep
51bb9a0064 fix clang -Wcast-align warning 2016-03-02 19:17:13 +01:00
inikep
87d4f3daa4 priceTable moved to heap 2016-03-02 15:56:24 +01:00
inikep
338533f741 improved price function initialization 2016-03-02 14:36:41 +01:00
Yann Collet
a997654670 Merge pull request #136 from inikep/dev 
merging mml3 mode
 2016-03-01 17:31:03 +01:00
inikep
a4dde25498 fixed repcode before lowLimit 2016-03-01 14:14:35 +01:00
inikep
2d55563b92 better logs 2016-02-29 22:07:40 +01:00
inikep
f647d99d61 fixed bug with 3 or more succesive rep codes 2016-02-29 12:33:08 +01:00
inikep
f414647c6b fixed ZSTD_copyCCtx 2016-02-25 22:31:07 +01:00
inikep
59493e8669 fixed paramgrill 2016-02-25 19:15:08 +01:00
inikep
afa9c9f5ab turn off RTF optimization 2016-02-25 16:47:57 +01:00
Yann Collet
ee0229287c changed version to v0.6.0 2016-02-25 14:53:35 +01:00
inikep
02137f8c42 more accurate gain function 2016-02-24 18:09:36 +01:00
inikep
ee55628c9d combined price function 2016-02-24 14:40:30 +01:00
Yann Collet
3152a8c87a finer dictionary completion 2016-02-23 21:28:59 +01:00
inikep
4a981f7937 improved ZSTD_GETPRICE 2016-02-23 19:08:20 +01:00
inikep
e0010e9baf priceFunc 2016-02-23 16:25:04 +01:00
inikep
15174b0cfb statistics of encoded sequences 2016-02-23 12:41:56 +01:00
inikep
9f754d23dc dummy levels 22-25 2016-02-22 17:00:04 +01:00
inikep
6b3739c8e5 MINMATCH=3 or 4 selected in ZSTD frame 2016-02-22 15:53:42 +01:00
inikep
84f43e2359 support for searchLength from 3 to 7 for optimal parser 2016-02-22 11:34:07 +01:00
inikep
c950b78ce2 removed backward match extension 2016-02-22 10:11:39 +01:00
inikep
d3b8d7a4e5 removed ZSTD_opt parser 2016-02-22 10:06:17 +01:00
inikep
6291c54006 litlen bounded to 128 bytes 2016-02-19 18:24:14 +01:00
Yann Collet
f8482131d7 better compatibility with Visual <= 2012 2016-02-19 17:33:43 +01:00
Yann Collet
191b52a8dc silence array subscript warning 2016-02-19 15:16:15 +01:00
inikep
fcafb26b2e speed optimiaztions 2016-02-19 11:59:44 +01:00
inikep
c3a9a9ca56 ZSTD_rescaleFreqs and removed minml 2016-02-19 11:05:25 +01:00
inikep
cc52a97214 added hashLog3 2016-02-19 10:09:35 +01:00
Yann Collet
4c64d51425 modified level 18 2016-02-18 01:01:42 +01:00
Yann Collet
d56a4192f6 Added : README within /lib directory, to help file selection 2016-02-17 17:47:29 +01:00
Yann Collet
6062b15a07 performance fix for hash chain mode 2016-02-16 17:41:03 +01:00
Yann Collet
accfd80c37 Added : option -t to test compressed files integrity 2016-02-15 19:33:16 +01:00
Yann Collet
0068be94d8 Faster literals cost evaluation (suggested by @inikep) 2016-02-15 18:42:13 +01:00
Yann Collet
e93add0439 Removed if (matchIndex >= current) return 0; as recommended by @inikep 2016-02-15 17:44:14 +01:00
Yann Collet
b8a6f6824b refactored fix to be more generic and applicable to both btlazy2 and btopt 2016-02-15 17:06:29 +01:00
Yann Collet
b37e0a9241 performance tweak : bt search w/ long repeated segments 2016-02-15 14:07:07 +01:00
Yann Collet
cefef8cde7 Change : zstd_opt_bt => zstd_btopt 2016-02-15 07:21:54 +01:00
Yann Collet
0d78a33374 updated CLevel confs for 256K 2016-02-15 06:49:09 +01:00
Yann Collet
422fd563f3 modified : compression Levels for blocks <= 16KB 2016-02-13 15:07:24 +01:00
Yann Collet
38fba56483 Changed compression level settings 2016-02-13 11:20:23 +01:00
Yann Collet
e8bd73df80 Fixed : error when compressing one file 2016-02-13 03:09:48 +01:00
Yann Collet
23e8aee224 Introduced new compression level settings 
Changed position of parameter TargetLength
 2016-02-13 02:08:46 +01:00
Yann Collet
82f4d7da03 Fixed Visual solution 2016-02-12 22:04:49 +01:00
Yann Collet
6f3acbac0d notificationLevel into ZDICT_param_t 2016-02-12 20:19:48 +01:00
Yann Collet
09ab681328 fixed libzstd install 2016-02-12 19:00:30 +01:00
Yann Collet
62ae5fb0d2 zstd_buffered => zbuff 2016-02-12 18:59:11 +01:00
Yann Collet
f4c9d75370 dictBuilder => zdict 2016-02-12 18:45:02 +01:00
Yann Collet
f062436fa6 cli : added ability to compress multiple files into stdout (-c) 2016-02-12 15:56:46 +01:00
Yann Collet
71eafdd23f Integrated dictionary builder into zstd cli 2016-02-12 02:31:57 +01:00
Yann Collet
fb92a78ac1 Fixed clang warnings 2016-02-12 00:32:38 +01:00
Yann Collet
7d36028078 Integrate dictBuilder into libzstd 2016-02-12 00:10:25 +01:00
Yann Collet
82260ddd8a fixed asan read error in ZSTD_HcGetAllMatches 2016-02-11 07:14:25 +01:00
Yann Collet
04b12d8bcb paramgrill can select manually targetLength 
new compression levels
 2016-02-11 06:23:24 +01:00
Yann Collet
bd828d9cd1 changed : sufficientLength => targetLength 2016-02-11 04:38:55 +01:00
Yann Collet
4488661678 minor cosmetics 2016-02-11 04:17:50 +01:00
Yann Collet
490aa68b18 restored fast zstd_opt variant 2016-02-10 17:18:24 +01:00
Yann Collet
b79a0b34f3 paramgrill supports sufficientLength 
stronger variation for ZSTD_opt
 2016-02-10 17:07:37 +01:00
Yann Collet
62ae262771 restored currentMl>0 condition, as suggested by @inikep 2016-02-10 15:44:07 +01:00
Yann Collet
3b63f7fbb0 change zstd_opt.c into zstd_opt.h, since it's included 2016-02-10 15:05:12 +01:00
Yann Collet
0db4115f35 fixed debug print macros on Windows 2016-02-10 14:50:22 +01:00
Yann Collet
2f94d2f166 minor refactor 2016-02-10 14:26:30 +01:00
Yann Collet
768c6bc705 fix : inlining difficulties with gcc 4.6 (travis) 2016-02-10 14:01:49 +01:00
Yann Collet
70e8c389f1 Refactoring for portability 2016-02-10 13:37:52 +01:00
Yann Collet
302177a0c8 Merge pull request #125 from inikep/dev 
Dev
 2016-02-10 00:57:00 +01:00
inikep
ce6730a740 fix for ZSTD_HcGetAllMatches_generic 2016-02-09 23:26:30 +01:00
inikep
be77f3342d Revert "Revert "ZSTD_compressBlock_opt_extDict_generic"" 
This reverts commit 2b714dfd26.
 2016-02-09 23:00:41 +01:00
inikep
2b714dfd26 Revert "ZSTD_compressBlock_opt_extDict_generic" 
This reverts commit b498fc1c99.
 2016-02-09 22:47:51 +01:00
inikep
b498fc1c99 ZSTD_compressBlock_opt_extDict_generic 2016-02-09 22:23:48 +01:00
inikep
e75621f98d dumpsEnd 2016-02-09 21:12:23 +01:00
inikep
31314ae0d1 int to U32 2016-02-09 20:47:39 +01:00
Yann Collet
6bff748e43 Fixed minor warnings in legacy modes 2016-02-09 17:55:01 +01:00
Yann Collet
4f0a3931c8 added missing patches 2016-02-07 04:00:27 +01:00
inikep
da730ff092 a bugfix for backward matches 2016-02-06 15:18:26 +01:00
inikep
dbade7c263 remove ZSTD_compressBlock_opt2_generic 2016-02-05 20:29:31 +01:00
inikep
f2fee4c316 added params.sufficientLength 2016-02-05 19:45:25 +01:00
inikep
5b0749715d backward match extension 2016-02-05 19:25:40 +01:00
inikep
2712b69582 Merge remote-tracking branch 'refs/remotes/origin/v05x' into dev 2016-02-05 09:22:10 +01:00
inikep
3379c5df36 ZSTD_OPT_DEBUG 2016-02-05 09:21:20 +01:00
inikep
f967753bc2 ssize_t -> int 2016-02-05 09:07:22 +01:00
Yann Collet
982ffc773d changed enum type name to ZSTD_ErrorCode 2016-02-05 02:33:10 +01:00
inikep
55a444764b sufficient_len = 32 2016-02-04 22:22:47 +01:00
inikep
34608aaa29 ssize_t 2016-02-04 22:18:02 +01:00
inikep
17712afdbc Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into v05x 2016-02-04 21:49:45 +01:00
inikep
b3df71e538 code cleaning 2016-02-04 21:45:26 +01:00
inikep
4fa4098628 last match will start at a minimum distance of 8 from oend 2016-02-04 21:16:07 +01:00
inikep
08f500ca10 cur_min 2016-02-04 20:53:54 +01:00
inikep
6f07186745 clean code 2016-02-04 17:57:50 +01:00
inikep
59c9146258 40099096 2016-02-04 17:48:33 +01:00
inikep
baff8d6c1f gitignore 2016-02-04 16:02:05 +01:00
inikep
40dd4739e3 removed llen 2016-02-04 15:52:35 +01:00
inikep
c9483f0982 clean code 2016-02-04 15:28:55 +01:00
Yann Collet
953ce72f4a minor comments refactoring 2016-02-04 15:28:14 +01:00
inikep
158677c16a new copy rep 2016-02-04 15:25:37 +01:00
inikep
d1ff7bf89a cur_rep 2016-02-04 14:57:11 +01:00
inikep
eb72f5992a i != litlen 2016-02-04 14:55:23 +01:00
inikep
997fb064a6 anchor 2016-02-04 13:01:56 +01:00
inikep
75be1a412c fixes 2016-02-04 11:48:06 +01:00
inikep
cb5cd16d6b 40392170 integer 2016-02-04 11:19:06 +01:00
inikep
27e1c6a51f ZSTD_resetFreqs 2016-02-04 11:11:08 +01:00
inikep
4196923dc9 40370570 ZSTD_getPriceReal 2016-02-04 10:04:33 +01:00
inikep
70b054559b MaxLit 2016-02-03 22:56:55 +01:00
inikep
a98149df59 ZSTD_getLiteralPrice 2 2016-02-03 21:26:18 +01:00
inikep
dbbedfbd95 ZSTD_getLiteralPrice 2016-02-03 19:53:29 +01:00
inikep
892776efd6 ZSTD_get_price2 2016-02-03 19:27:44 +01:00
inikep
1292509987 ZSTD_get_price 2016-02-03 19:13:28 +01:00
inikep
749460d9ab ZSTD_updatePrice 2016-02-03 19:08:48 +01:00
inikep
3bfcfc7bb0 ZSTD_updatePrice 2016-02-03 18:47:30 +01:00
inikep
53721f134c Merge remote-tracking branch 'refs/remotes/Cyan4973/v05x' into v05x 
# Conflicts:
#	lib/zstd_compress.c
#	lib/zstd_static.h
 2016-02-03 17:29:40 +01:00
inikep
78e5ea3a32 ZSTD_getPrice 2016-02-03 17:25:42 +01:00
inikep
d02506c186 40699855 2016-02-03 16:54:23 +01:00
inikep
85cc6b509b minml 2016-02-03 15:58:08 +01:00
inikep
7f6680e658 clean some code 2016-02-03 13:30:34 +01:00
inikep
4cb2009821 40901629 2016-02-03 13:25:34 +01:00
inikep
1b8b1f9d53 auto check of matches 2016-02-03 13:17:28 +01:00
inikep
e9f30eae4b 40970636 2016-02-03 12:53:07 +01:00
Yann Collet
b010b3b627 Fixed decoding error (afl) 2016-02-03 12:39:34 +01:00
Yann Collet
72bff50ecf Updated error API, following suggestions by @nemequ (#116) 2016-02-03 12:06:24 +01:00
Yann Collet
ae7aa06650 lots of minor refactorings 2016-02-03 02:46:46 +01:00
Yann Collet
7d968c7bdd move _usingDict() to stable API (zstd.h) 2016-02-03 02:11:32 +01:00
Yann Collet
464fa996e9 added v0.4 legacy support 2016-02-03 01:09:46 +01:00
inikep
5391a718a4 fixed 2016-02-03 00:04:14 +01:00
inikep
55bc9a868a 41374379 2016-02-02 19:47:09 +01:00
inikep
5d4131067b update 2016-02-02 17:41:21 +01:00
Yann Collet
7d8e6bddb9 changed long length format 2016-02-02 17:30:37 +01:00
inikep
d6e8cc5f0c 41403384 2016-02-02 16:43:31 +01:00
Yann Collet
7a6343fb1b fixed encoding bugs 2016-02-02 16:00:50 +01:00
inikep
e1b8e53e84 better logging 2016-02-02 15:41:16 +01:00
Yann Collet
2ce4923238 modified buffer management rules 2016-02-02 14:36:49 +01:00
inikep
59140c5480 turn off: last match must start at a minimum distance of 8 from oend 2016-02-02 12:36:09 +01:00
inikep
ef2706ed12 in progress 2016-01-31 15:32:01 +01:00
inikep
6b0715b8fd no "start" 2016-01-31 15:17:50 +01:00
inikep
d4c3e8c595 fixed logs 2016-01-31 14:28:26 +01:00
inikep
c0dc4c361d best_off 2016-01-31 12:36:41 +01:00
inikep
9bfb1a3357 working 2016-01-31 12:17:16 +01:00
inikep
ee4be5e981 ZSTD_compressBlock_opt_generic 2016-01-31 11:57:01 +01:00
inikep
e2bfe2407a GetAllMatches 2016-01-31 11:26:05 +01:00
inikep
c71568fd44 ZSTD_opt 2016-01-31 11:26:04 +01:00
Yann Collet
35f7de52c8 fixed Visual warnings 2016-01-31 02:51:03 +01:00
Yann Collet
61e16ce07c minor optimization for small files 2016-01-31 02:04:15 +01:00
Yann Collet
e93d6ced17 minor compression gain 2016-01-31 00:58:06 +01:00
Yann Collet
fd416f1eaf added : ZSTD_compress_usingPreparedCCtx() 2016-01-30 03:14:15 +01:00
Yann Collet
863ec40f1e minor compression speed improvement 2016-01-28 17:56:33 +01:00
Yann Collet
9cadd0853c add basic tests 2016-01-28 15:39:52 +01:00
Yann Collet
4ddb1f5566 optimized counting of small segments 2016-01-28 03:24:53 +01:00
Yann Collet
fb810d6053 changed dictionary format 2016-01-28 00:21:13 +01:00
Yann Collet
1c8e194297 modified streaming compression API 2016-01-26 16:31:22 +01:00
Yann Collet
7b51a2948f modified streaming decompression API 2016-01-26 15:58:49 +01:00
Yann Collet
b923f65076 introduced dictionary format 2016-01-26 03:14:20 +01:00
Yann Collet
bc4c8aa4b7 fixed new MIN_CBLOCK_SIZE 2016-01-25 17:26:01 +01:00
Yann Collet
3742219b4e fixed decompression bugs 2016-01-25 16:54:05 +01:00
Yann Collet
a1249dc8da fix visual warnings 2016-01-25 04:22:03 +01:00
Yann Collet
afe0709964 stream control 2016-01-25 04:10:46 +01:00
Yann Collet
59d1f79768 variable litblock header size 2016-01-23 19:28:41 +01:00
Yann Collet
a768a301ba fixed VS project 2016-01-21 16:04:35 +01:00
Yann Collet
ffec740d37 fixed visual and clang errors 2016-01-21 15:50:11 +01:00
Yann Collet
977f1f3600 modified error system, following suggestions by @nemequ 2016-01-21 15:38:47 +01:00
Yann Collet
74ce8e812c fixed VS 2015 'ERROR already defined', reported by Rich Geldreich 2016-01-21 14:32:41 +01:00
Yann Collet
7beaa05d81 fixed bt bug 2016-01-21 11:57:45 +01:00
Yann Collet
1840d6af33 bench pay attention to variable block sizes 2016-01-20 15:39:06 +01:00
Yann Collet
fba6aed175 changed : ZSTD_LEGACY_SUPPORT set to 0 by default within zstd_decompress.c (#113) 2016-01-18 12:03:54 +01:00
Yann Collet
a87278a4f0 small compression speed 2016-01-17 00:12:55 +01:00
Yann Collet
3a3b72f25a Implemented ZSTD_HEAPMODE for zstd_decompress.c 
Reduce cmake version requirement to 2.8.7
 2016-01-11 12:56:11 +01:00
Yann Collet
523b594233 fixed read error 2016-01-09 02:10:40 +01:00
Yann Collet
b012510d9f dictionary with block API 2016-01-09 02:00:10 +01:00
Yann Collet
bf42c8e5d8 block-level API 2016-01-09 01:08:23 +01:00
Yann Collet
c64c100658 fixed : unused variable 2016-01-08 17:54:02 +01:00
Yann Collet
600962711d added duplication tests 2016-01-08 17:27:50 +01:00
Yann Collet
6e1c4c6c65 fixed context copy 2016-01-07 23:07:44 +01:00
Yann Collet
ecd651bd91 added ZSTD_duplicateCCtx() 2016-01-07 15:35:18 +01:00
Yann Collet
494c786a3b fixed outstanding strict aliasing warnings 2016-01-06 12:54:02 +01:00
Yann Collet
1fdd82312d Fixed : strict aliasing warnings in legacy decoders 2016-01-06 12:35:42 +01:00
Yann Collet
218bd31942 minor strict aliasing warning 2016-01-06 02:19:55 +01:00
Yann Collet
3b994cb465 updated fse 2016-01-06 01:58:37 +01:00
Yann Collet
96ffa424eb minor fast mode improvement 2016-01-02 01:16:28 +01:00
Yann Collet
2cc12cb3cb minor refactoring 2016-01-01 07:47:58 +01:00
Yann Collet
1358f91ec4 minor refactoring 2016-01-01 07:29:39 +01:00
Yann Collet
72e84cfd0f minor simplification 2015-12-31 19:08:44 +01:00
Yann Collet
722504c2a0 update version to 0.4.6 2015-12-31 18:32:15 +01:00
Yann Collet
5f9f98768a Makefile version automatically follows zstd.h 2015-12-30 19:52:08 +01:00
Yann Collet
48da164f14 fixed clang warnings 2015-12-29 23:40:02 +01:00
Yann Collet
ee3f4515c1 faster hc mode with repetitive data (experimental) 2015-12-29 22:26:09 +01:00
Yann Collet
d94efbf95c fix fast mode on Windows 2015-12-29 14:29:08 +01:00
Yann Collet
1ac364d60b Merge pull request #99 from KrzysFR/dev 
Update version to 0.4.5 in zstdlib resources
 2015-12-21 10:19:26 +01:00
Yann Collet
ed73ae8c67 fix #101 2015-12-19 19:51:42 +01:00
Christophe Chevalier
8972f13b4a Added missing ZSTDLIB_API for ZSTD_decompress_usingDict and ZSTD_decompressDCtx 2015-12-18 11:20:37 +01:00
Yann Collet
31683c0b28 benchmark can use dictionary 2015-12-18 01:26:48 +01:00
Yann Collet
fdcad6d3e1 added ZSTD_compress_usingDict() 2015-12-17 23:50:15 +01:00
Yann Collet
deb078b9b1 added : -m : decompress multiple files 2015-12-17 20:30:14 +01:00
Yann Collet
f54f57023c update params for 256K 2015-12-16 19:38:54 +01:00
Yann Collet
0700585fb9 fixed asan warning 2015-12-12 12:54:48 +01:00
Yann Collet
188b6ed6c7 fixed g++ warning 2015-12-12 12:44:44 +01:00
Yann Collet
3a061a4c9f extended static dictionary to buffered mode 2015-12-12 11:17:42 +01:00
Yann Collet
6c3e2e7855 fixed : high compression modes for Windows 32 bits 2015-12-11 10:44:07 +01:00
Yann Collet
9f5ab1a4f5 fix 32-bits windows decoding on high-compression archives 2015-12-11 00:27:41 +01:00
Christophe Chevalier
2abb04d90d Added ZSTDLIB_API to new ZSTD_maxCLevel method 2015-12-09 23:55:23 +01:00
Christophe Chevalier
7b05324a71 Only define ZSTDLIB_API when building a Windows DLL 2015-12-09 17:12:12 +01:00
Christophe Chevalier
844baf4fc5 Merge remote-tracking branch 'upstream/dev' into zstdlib_api 2015-12-09 15:45:57 +01:00
Yann Collet
d608088ca3 added : ZSTD_maxCLevel() 
added : 256KB blocks mode
 2015-12-09 09:05:22 +01:00
Yann Collet
0cde77bc5e fixed comment (reported by @annulen) 2015-12-08 14:47:46 +01:00
Christophe Chevalier
d2199e7250 Added ZSTDLIB_API definition to zstd_buffered.h which does not seem to include zstd.h 2015-12-07 19:01:41 +01:00
Christophe Chevalier
c6e845398a Add ZSTDLIB_API macro to prefix all exported methods with__declspec(dllexport) when building a DLL (on windows) 2015-12-07 17:48:53 +01:00
Yann Collet
4bfe415fa7 validated external dictionary 2015-12-06 13:18:37 +01:00
Yann Collet
e47c4e5f8e strengthened bufferless streaming decompression 2015-12-05 09:23:53 +01:00
Yann Collet
417890cec2 strengthened streaming bufferless compression 2015-12-04 17:16:37 +01:00
Yann Collet
6a2f03295c update version to 0.4.2 2015-12-02 15:05:22 +01:00
Yann Collet
120230b895 compress uses less memory for small blocks 2015-12-02 14:00:45 +01:00
Yann Collet
a308259afa new : parameters for <= 16KB 2015-12-02 13:38:48 +01:00
Peter Harris
f06e238d6f Fix compression on Big Endian CPUs 2015-12-01 14:58:24 -05:00
Yann Collet
c620b48489 revert install to normal zstd due to travis limitation (likely gcc 4.6) 2015-12-01 01:56:02 +01:00
Yann Collet
44287a3bf4 refactored decompress (#82) 2015-11-30 23:13:56 +01:00
Yann Collet
aead1bf86b changed to ZSTD_LEGACY_SUPPORT=0, as suggested within #80 2015-11-30 14:18:47 +01:00
Yann Collet
daa1c65500 fixed #81 : removed dead file (reported by @inikep) 2015-11-30 12:25:48 +01:00
Yann Collet
a85a8dd94a Fixed #80 (reported by @luben) 
Added : tests for ZSTD_LEGACY=disable build mode
extended ZSTD_LEGACY=disable to dll build
 2015-11-30 11:53:11 +01:00
Yann Collet
375727310c fixed compression bug 2015-11-29 03:17:04 +01:00
Yann Collet
7fe531e7ec fixed address space overflow (32-bits mode) 2015-11-29 02:38:09 +01:00
Yann Collet
52732b505d fixed huff0 decoder (32-bits mode) 2015-11-28 22:03:08 +01:00
Yann Collet
bf7aa3c14e fixed decoder error (rollbuffer mode) 2015-11-28 18:19:44 +01:00
Yann Collet
ad50c59bb7 fixed decoder error (32-bits mode, malicious input) 2015-11-28 17:09:28 +01:00
Yann Collet
00fd7a2110 protect 32-bits mode 2015-11-28 16:03:22 +01:00
Yann Collet
53fbf01370 minor level change 2015-11-28 14:08:01 +01:00
Yann Collet
800fa6c378 stronger tests for zbuff decompression 2015-11-27 14:30:23 +01:00
Yann Collet
31d1806a77 zbuff properly supports small dst buffers 2015-11-27 14:07:36 +01:00
Yann Collet
944042ab7d coalesce block+header size (buffered mode) 2015-11-27 13:26:38 +01:00
Yann Collet
d248aa1c62 fixed hang bug (infinite loop) in decoder, buffered mode, malicious input 2015-11-26 22:39:29 +01:00
Yann Collet
661b94fb5d fixed potential warning on undefine, reported by Maciej Adamczyk 2015-11-26 16:42:06 +01:00
Yann Collet
29a58c94f6 fixed lib makefile 2015-11-26 16:25:56 +01:00
Yann Collet
26fa696713 minor API change 2015-11-26 16:07:08 +01:00
Yann Collet
29a2c838c5 support for legacy 0.3 format 2015-11-26 16:02:04 +01:00
Yann Collet
c2ce890ac2 fixed bug in legacy decoder v0.2, reported by Maciej Adamczyk 2015-11-26 14:12:23 +01:00
Yann Collet
26415d3ff2 fixed leak 2015-11-26 12:43:28 +01:00
Yann Collet
6bcdeac954 fixed asan warning 2015-11-26 11:43:00 +01:00
Yann Collet
5f2ec63852 fixed decompression bug (buffered mode) 2015-11-26 10:32:17 +01:00
Yann Collet
91a97960f7 Added zbufftest 2015-11-26 09:59:49 +01:00
Yann Collet
e4fdad55dc fixed minor decompression bug in buffered mode 2015-11-25 21:09:17 +01:00
Yann Collet
d3cb690156 fixed leak issue 2015-11-25 15:26:55 +01:00
Yann Collet
377b9e8b94 fixed : potential malloc error 2015-11-25 15:04:37 +01:00
Yann Collet
7b2c31043c fixed : potential malloc error 2015-11-25 15:02:46 +01:00
Yann Collet
88fcd2916e Added : zstd buffered API 2015-11-25 14:42:45 +01:00
Yann Collet
c36521571e fixed large buffers bug 2015-11-24 14:06:07 +01:00
Yann Collet
225179dd05 more tests (bt modes) 2015-11-23 16:52:22 +01:00
Yann Collet
239cc289d3 fixed asan bugs 2015-11-23 16:17:21 +01:00
Yann Collet
03526e18fa preliminary rollbuffer support for bt mode 2015-11-23 15:29:15 +01:00
Yann Collet
c1e52f0f05 hc simplication 2015-11-23 14:37:59 +01:00
Yann Collet
06eade5102 comments 2015-11-23 14:23:47 +01:00
Yann Collet
5054ee0cc0 rollbuffer refactor 2015-11-23 13:34:21 +01:00
Yann Collet
d7233d6e14 more compression tests 2015-11-22 14:40:51 +01:00
Yann Collet
287b7d9607 small hc speed improvement 2015-11-22 13:24:05 +01:00
Yann Collet
428619ccd7 small cLevel update 2015-11-22 12:46:30 +01:00
Yann Collet
a85c77b9d3 fixed roll buffer lazy2 mode 2015-11-22 12:22:04 +01:00
Yann Collet
b7fc88eec7 fixed roll buffer lazy mode 2015-11-22 03:12:28 +01:00
Yann Collet
734aa92ed5 fixed roll buffer greedy mode 2015-11-22 03:01:33 +01:00
Yann Collet
9a24e5986b fixed roll buffer in fast mode 2015-11-22 02:53:43 +01:00
Yann Collet
007c1c6ff0 minor cr improvement 2015-11-22 02:42:28 +01:00
Yann Collet
7a23179779 small fast compression improvement 2015-11-21 15:27:35 +01:00
Yann Collet
402fdcf1a3 fixed asan 2015-11-20 12:46:08 +01:00
Yann Collet
55aa7f94e3 modified format 2015-11-20 12:04:52 +01:00
Yann Collet
743402cb23 modified format 2015-11-20 12:03:53 +01:00
Yann Collet
7dfd56be4a restored fix small blocks 2015-11-19 17:46:29 +01:00
Yann Collet
c95f899725 refactored greedy 2015-11-19 17:28:35 +01:00
Yann Collet
b3a2af970e refactored decompression 2015-11-19 17:13:19 +01:00
Yann Collet
110cc14bab fixed bug #73 notified by @nemequ 2015-11-19 12:02:28 +01:00
Yann Collet
dcac00e7a6 update library builder 2015-11-19 11:23:44 +01:00
Yann Collet
b2549846ba more tests 2015-11-18 11:29:32 +01:00
Yann Collet
94b9d8ec8d fix compression bug in rollbuffer mode 2015-11-17 16:14:55 +01:00
Yann Collet
138e89c9c4 fix compression bug (extDict mode) 2015-11-17 14:26:54 +01:00
Yann Collet
786eb7d42e fixed repcode bug 2015-11-17 12:23:53 +01:00
Yann Collet
82368cfa1b comment 2015-11-16 19:10:56 +01:00
Yann Collet
eaa5f75b99 fixed decompression bug 2015-11-15 11:13:41 +01:00
Yann Collet
93a823ccb7 fixed asan warnings 2015-11-13 15:08:43 +01:00
Yann Collet
89db5e001d rolling buffer preliminary support for fast mode 2015-11-13 11:27:46 +01:00
Yann Collet
0f366c61ed fixed streaming decoder 2015-11-12 16:19:30 +01:00
Yann Collet
5b78d2f20c basic rolling buffer support in decoder 2015-11-12 15:36:05 +01:00
Yann Collet
14983e7aed refactored compression code 2015-11-11 21:38:21 +01:00
Yann Collet
7f6e91fa98 fixed library compilation 2015-11-11 14:39:50 +01:00
Yann Collet
5be2dd25f2 Changed Library source tree 
- no more zstdhc ; zstd.h is enough
- ZSTD_compress() now needs a compressionLevel
- separated zstd_compress.c and zstd_decompress.c
- updated zstdcli, fullbench, fuzzer with new API
 2015-11-11 13:43:58 +01:00
Yann Collet
530918b409 level tuning 2015-11-10 10:52:58 +01:00
Yann Collet
9f432926b4 fileio can use small blocks params 2015-11-09 17:42:17 +01:00
Yann Collet
43e0cd512a small blocks params 2015-11-09 16:38:17 +01:00
Yann Collet
4baee50728 fix bug 2015-11-09 03:19:33 +01:00
Yann Collet
d1ade5a61d fix 32-bits mode 2015-11-08 15:49:20 +01:00
Yann Collet
e9eba608c2 simplified bt 2015-11-08 15:08:03 +01:00
Yann Collet
31d00ba4b3 fix bt bug 2015-11-08 14:11:29 +01:00
Yann Collet
72a8e751f4 level tuning 2015-11-07 22:24:46 +01:00
Yann Collet
f48e35c206 minor bt improvements 2015-11-07 01:13:31 +01:00
Yann Collet
a81d9ac42d minor bt CSpeed 2015-11-06 19:03:59 +01:00
Yann Collet
628065cf98 revert 2015-11-06 18:44:54 +01:00
Yann Collet
9b8e453713 minor CRatio improvement 2015-11-06 18:28:02 +01:00
Yann Collet
47cfa9a985 minor CRatio improvement 2015-11-06 17:13:46 +01:00
Yann Collet
eb283710dd minor greedy CRatio improvement 2015-11-06 16:33:11 +01:00
Yann Collet
fc2afcfa64 small Cspeed increase 2015-11-06 15:40:14 +01:00
Yann Collet
444873f494 minor greedy mod 2015-11-06 11:48:56 +01:00
Yann Collet
805a52a773 fixed asan warning 2015-11-06 10:52:17 +01:00
Yann Collet
8ad12ec343 level tuning 2015-11-06 10:19:00 +01:00
Yann Collet
f12c1304a8 integrate fast mode 2015-11-05 18:16:59 +01:00
Yann Collet
1f44b3f778 added fast mode 2015-11-05 17:32:18 +01:00
Yann Collet
47b68908b2 rename lazy2 2015-11-05 15:14:17 +01:00
Yann Collet
9036105b62 simplified source 2015-11-05 15:03:12 +01:00
Yann Collet
5106a76dc9 merged strats 2015-11-05 15:00:24 +01:00
Yann Collet
43ae161fe9 level tuning 2015-11-05 11:58:02 +01:00
Yann Collet
92309cb37e level tuning 2015-11-05 00:48:37 +01:00
Yann Collet
3137d1a059 fix fuzzer32 litCSize limit condition 2015-11-04 23:36:36 +01:00
Yann Collet
50c5cdb44c fixed issues reported by Maciej Adamczyk 2015-11-04 20:35:33 +01:00
Yann Collet
3e3582719c Fixed issue #62, reported by @luben 2015-11-04 18:19:39 +01:00
Yann Collet
e8455f51ea fix clang warning 2015-11-04 17:41:20 +01:00
Yann Collet
b241e9deb7 small compression improvement 2015-11-04 13:57:24 +01:00
Yann Collet
59d7063fbc fix and level tuning 2015-11-04 12:05:27 +01:00
Yann Collet
96b9f0ba4d btlazy2 2015-11-04 03:52:54 +01:00
Yann Collet
588d1e5fa0 Fixed asan issue reported by Maciej Adamczyk 2015-11-03 10:48:42 +01:00
Yann Collet
050efba81b level tuning 2015-11-03 09:49:30 +01:00
Yann Collet
3564487614 lazydeep 2015-11-02 16:14:46 +01:00
Yann Collet
7671f393be v0.3.2 2015-11-02 12:17:39 +01:00
Yann Collet
4e049bca8c Fixed Visual Studio issue #60, reported by @mailagentrus 2015-11-02 11:09:54 +01:00
Yann Collet
56e1092cb0 level tuning 2015-11-02 01:02:43 +01:00
Yann Collet
9b11b46f8a Fixed speed regression 2015-11-01 12:40:22 +01:00
Yann Collet
be2010ea1b lazy strategy 2015-10-31 12:57:14 +01:00
Yann Collet
4b100f43b0 level tuning 2015-10-30 15:49:48 +01:00
Yann Collet
ec43ba4a95 updated visual studio projects 2015-10-30 11:51:26 +01:00
Yann Collet
aa074051d6 decoding support for legacy v0.2 format 2015-10-30 11:21:50 +01:00
Yann Collet
fc04bcc6d7 fixed minor gcc warning 2015-10-30 07:11:07 +01:00
Yann Collet
8f9d49c092 fixed asan warning 2015-10-30 07:00:37 +01:00
Yann Collet
4114f95ce9 fix Visual Studio projects 2015-10-30 06:40:22 +01:00
Yann Collet
44fe99116e fixed hc alloc 2015-10-29 22:02:40 +01:00
Yann Collet
712def905b one less hc malloc 2015-10-29 18:41:45 +01:00
Yann Collet
3d9cf7acf2 fix include 2015-10-29 17:15:14 +01:00
Yann Collet
71bcdb5c1f sync paramgrill with HC starting at level 2 2015-10-29 17:08:03 +01:00
Yann Collet
2acb5d3d48 HC modes can be selected from command line 2015-10-29 16:49:43 +01:00
Yann Collet
76c5c2a4a0 minor tuning 2015-10-28 19:07:05 +01:00
Yann Collet
81e49e6654 fine tune & messages 2015-10-28 15:56:48 +01:00
Yann Collet
b315bc86e1 srcLog 2015-10-27 13:12:25 +01:00
Yann Collet
2c6992e1dc blocksize & T 2015-10-27 12:18:00 +01:00
Yann Collet
d55a7e0e4d simplified chain 2015-10-27 03:06:40 +01:00
Yann Collet
8b91abec18 can test param conf 2015-10-27 02:59:12 +01:00
Yann Collet
342892c3ae repcode+1 2015-10-26 17:44:04 +01:00
Yann Collet
786f5b554a skip slow conf faster 2015-10-26 15:45:58 +01:00
Yann Collet
3a787da419 Fixed : minor clang warning 2015-10-25 19:42:03 +01:00
Yann Collet
083fcc82fc HC params 2015-10-25 14:06:35 +01:00
Yann Collet
53fff6cd9d Individual memlog values 2015-10-24 13:48:37 +01:00
Yann Collet
ed0a781246 repfirst 2015-10-23 19:25:06 +01:00
Yann Collet
83fae977e6 Added asan tests 2015-10-23 15:21:53 +01:00
Yann Collet
fc60883d42 fixed issue #49, reported by Hanno Böck (@hannob) 2015-10-23 12:45:02 +01:00
Yann Collet
eeb8ba18d1 uncoupled maxdlog and chainlog 2015-10-22 16:55:40 +01:00
Yann Collet
f3eca25322 quick first zstdhc version 2015-10-22 15:31:46 +01:00
Yann Collet
353c5d26cf unified ctx naming convention 2015-10-21 14:39:26 +01:00
Yann Collet
f42803e5b4 restored ZSTD_compressCCtx() def 2015-10-21 13:31:46 +01:00
Yann Collet
8eb50b8acf Fixed minor gcc5+ warning 2015-10-21 10:01:09 +01:00
Yann Collet
7010c27a13 Added advanced compression functions 2015-10-21 09:07:25 +01:00
Yann Collet
c8a7254192 clang optimizer bug workaround 2015-10-20 03:01:38 +01:00
Yann Collet
4caefaf62f refactored literal segment 2015-10-19 19:25:44 +01:00
Yann Collet
1fbee1f483 Fixed sanitizer warning 2015-10-19 15:32:47 +01:00
Yann Collet
4d1a0ef515 minor speed improvement 2015-10-19 03:12:23 +01:00
Yann Collet
c7ae9ffde3 small decompression speed improvement 2015-10-19 02:36:34 +01:00
Yann Collet
107c5755f4 small decompression speed improvement 2015-10-19 02:14:43 +01:00
Yann Collet
50174c77fd missed updated files 2015-10-18 22:26:26 +01:00
Yann Collet
b1f3f4b565 update huff0 2015-10-18 22:18:32 +01:00
Yann Collet
8f86c700cd Fix uninitialized warnings reported by @nemequ 2015-10-14 17:53:44 +01:00
Yann Collet
59aac5f467 Clarified comments 2015-10-14 16:28:19 +01:00
xaphier
77ee44c7b6 Remove one malloc/free from compression 
By making the buffer part of the zstd context structure, one malloc/free
can be removed from the compression.
 2015-10-10 12:14:51 +02:00
Yann Collet
be50aaa0ee Added : Frame concatenation ability 2015-09-10 23:26:09 +01:00
Yann Collet
5abd8203cb Fix : decompression i/o detects flush write errors 2015-08-27 03:16:04 +01:00
Yann Collet
866a3c3158 Fixed minor visual warning 2015-08-26 23:10:09 +01:00
Yann Collet
352ab88aea fixed minor issue on decoding invalid data 2015-08-26 22:45:23 +01:00
Yann Collet
6b5198fe3a Updated FSE 2015-08-26 19:22:01 +01:00
Yann Collet
bf50409e05 Refactor Makefile and .travis tests 2015-08-26 00:19:06 +01:00
Yann Collet
f44b2b02b4 Zstd cli : --version and --help write to stdout 2015-08-25 23:32:45 +01:00
Yann Collet
c705567522 Fixed : compression error on some specific distribution 2015-08-25 22:56:42 +01:00
Yann Collet
1885029ba1 Fixed a few issues found by AFL (American Fuzzy Lop) 2015-08-24 20:17:11 +01:00
Yann Collet
fee8e240c7 Fixed decoding error #11 (reported by @magv) 2015-08-24 15:47:04 +01:00
Yann Collet
d5d9bc3f82 Added : ZSTD decompression : ability to resist to noisy/faulty data source 2015-08-23 23:13:49 +01:00
Yann Collet
d02114e0e1 updated fse 2015-08-21 03:59:31 +01:00
Yann Collet
997f9ee2ef Added : fuzzer tests : decompressing noisy src 2015-08-21 02:44:20 +01:00
Yann Collet
7d20acd340 Fix : decoder issue in exceptionnal circumstances (dst buffer too small) 2015-08-20 15:55:50 +01:00
Yann Collet
602834f794 Fixed : bug in compression in specific conditions (too small dst size) 2015-08-20 07:46:10 +01:00
Yann Collet
23743530e0 Updated fse 2015-08-19 23:53:56 +01:00
Yann Collet
f4ce8913a3 Added more strigent tests : compresson into too small buffer 2015-08-11 14:18:45 +01:00
Yann Collet
f90314610d Minor decompression simplification 2015-08-08 02:54:26 +01:00
Yann Collet
8a46c159c5 minor compress sequence refactor 2015-08-08 02:16:11 +01:00
Yann Collet
e9853b2cdb Fixed : ZSTD_compress() can attempt compression on a too small buffer 2015-08-07 19:07:32 +01:00
Yann Collet
96f3b2c0f7 zstd_compress() no longer requires maxDstSize >= ZSTD_compressBound(srcSize) 2015-08-07 15:50:42 +01:00
Yann Collet
a787550d1c Updated FSE lib 2015-08-07 15:21:00 +01:00
Yann Collet
77c82b680b updated fse 2015-08-02 01:19:09 +01:00
Yann Collet
138db2187d Fixed some minor sanitizer warnings within huff0 2015-07-27 20:19:21 +01:00
Yann Collet
fb8296f55c Updated fse (faster huff0 32-bits decoding) 2015-07-27 19:34:27 +01:00
Yann Collet
e8c6bb1e42 Integrated huff0 (breaking format change) 2015-07-26 00:23:57 +01:00
Yann Collet
968f275981 Changed allocation strategy to reduce stack usage of ZSTD_compressSequences() 2015-07-07 16:54:25 -08:00
Yann Collet
858db1291f Fixed : ZSTD_compress() corrently prevents buffer overflow 2015-07-07 01:02:04 -08:00
Yann Collet
dc6c1d920a Fixed clang warning 2015-07-07 00:44:22 -08:00
Yann Collet
674d91b3da Added sanitize=undefine test 2015-07-07 00:36:49 -08:00
Yann Collet
17867ced03 Fixed minor clang warnings 2015-07-07 00:14:27 -08:00
Yann Collet
7083b8a54a Travis CI : added ARM compilation tests 2015-07-06 23:58:59 -08:00
Yann Collet
ee28a0def4 Fixed issue #34 and #31 2015-07-06 23:45:35 -08:00
Yann Collet
94f998b1fc Fixed a few visual analyzer warnings 2015-07-04 23:10:40 -08:00
Yann Collet
7393c5a51d Advanced API change : Cctx & DCtx are now incomplete types for stricter type checking 2015-07-04 18:20:42 -08:00
Yann Collet
1efa31fad8 Updated FSE lib 2015-07-04 15:56:41 -08:00
Yann Collet
078a9a2804 Added Wcast-qual compilation flag 
Updated xxHash
Removed log traces
 2015-06-20 19:37:53 -08:00
Yann Collet
213089c078 Updated FSE 
Added clang and g++ tests
 2015-06-18 07:43:16 -08:00
Yann Collet
bbfa7d77c9 Fixed issue #28 2015-03-23 23:15:22 +01:00
Yann Collet
ad68a0e5f7 "type redef fix" for gcc <= 4.4 2015-02-26 00:29:15 +01:00
Yann Collet
26aa1ec946 updated FSE 2015-02-24 09:05:58 +01:00
Yann Collet
a3c75bad5a Updated FSE 
Faster speed on barely compressible data
 2015-02-21 03:31:59 +01:00
Yann Collet
00be3437a8 Some comments, to explain streaming decompression API 2015-02-20 18:53:31 +01:00
Yann Collet
1db6f22f34 Removed useless constants 2015-02-17 13:38:44 +01:00
Yann Collet
c5d46b5c90 New streaming API behavior, to solve issue 19 (https://github.com/Cyan4973/zstd/issues/19) 2015-02-16 18:06:26 +01:00
Yann Collet
fb98fd0bd4 datagen refactoring 2015-02-10 18:15:20 +01:00
Yann Collet
e30c39cd1b internal refactoring - better separation between match & entropy 2015-02-09 03:50:11 +01:00
Yann Collet
2ddf7e9a28 Fixed : FSE issue with corner case distributions 2015-02-08 20:26:47 +01:00
Yann Collet
d6914bfd5a fix minor Visual warning 2015-02-01 12:09:46 +01:00
Yann Collet
aacace37f7 Fixed a few clang warnings 2015-02-01 11:57:30 +01:00
Yann Collet
b0802817c0 avx2 mode solely detected using __AVX2__ macro 2015-02-01 10:13:22 +01:00
Yann Collet
e5c25e0ebc minor fix for Visual 2015-01-31 20:10:53 +01:00
Yann Collet
936e678dda Fixed : issue 8 regarding huge input buffer (> 4 GB) 2015-01-31 15:09:51 +01:00
Yann Collet
439eb77088 Normalize all the line endings 2015-01-31 10:52:59 +01:00
Yann Collet
1cc58def2d Restored cast for C++ (fse) 2015-01-29 07:13:54 +01:00
Yann Collet
6434adcf54 Fix : no AVX2 for Visual 2008 and older 2015-01-29 07:00:43 +01:00
Yann Collet
565b81d0ba fixed : corner case (FSE) 2015-01-29 06:51:30 +01:00
Yann Collet
e98ea20c37 Fix for VS2005 (AVX2 intrinsics) 2015-01-28 01:23:14 +01:00
r-lyeh
0da7941a72 Bugfixed Visual Studio errors when compiled as C++ sources 2015-01-26 10:24:04 +01:00
Yann Collet
fb81417323 Fixed : minor warnings under Visual 32 bits 2015-01-25 13:19:12 +01:00
Yann Collet
759433d679 Updated fullbench tests 2015-01-24 13:31:55 +01:00
Yann Collet
04d8e15e57 fixed : file permissions 2015-01-24 11:22:13 +01:00
Yann Collet
4856a00164 Initial release 2015-01-24 01:58:16 +01:00