4226 Commits

Author	SHA1	Message	Date
Yann Collet	5c2f2ebfdb	zstdmt via compress_generic: reduce opportunity to free/create mtctx ... `zstreamtest --newapi` (and `--opaqueapi`) create and destroy way too many threads resulting in failure of tsan tests, and potentially connected to the qemu flaky tests. This is because, at each test, the nb of threads can be changed (random). The `--no-big-tests` directive reduce this choice to 1/2 threads, in order to limit memory usage, especially for qemu and 32-bits builds. Unfortunately, swapping between 1 and 2 threads is enough to constantly create/destroy new mtctx. This patch takes advantage of the following property : via compress_generic, no internal mtctx is needed for nbThreads < 2. As a consequence, when nbThreads == 2, the currently active mtctx is necessarily good. This dramatically reduces the nb of thread creations when invoking `zstreamtest --newapi --no-big-tests` (only when parent cctx itself is created, which is randomized to 1/256 tests). Expected outcome : - at a minimum : tsan tests shall now work continuously without exploding the thread counter - at best : flaky qemu tests on `zstreamtest --newapi --no-big-tests` may stop being flaky, due to less stress from constant thread creation/destruction Real world impact : minimal, I don't expect users to constantly change `nbThreads` between each invocation. If `nbThreads` remains stable, existing implementation re-uses existing mtctx. Also : `zstreamtest --newapi` but without `--no-big-tests` doesn't benefit as much, since this test can select a random `nbThreads` value between 1 and 4. The current patch only reduces opportunity to free/create mtctx (for example : 2->1->2 doesn't need a new mtctx) but doesn't completely eliminate it, since `nbThreads` can still change between 2/3/4. A more complete solution could be to only use 2 out of 4 allocated threads, thus keeping the pool at a constant size. This would require a larger change to `POOL_*` api though.	2017-12-16 12:48:13 -08:00
Yann Collet	3cbfac1cdb	updated levels 15-20 ... taking advantage of `btopt` improved speed to tune parameters. Levels 16-19 are stronger than previous release, making the graph more favorable. In theory, I should also update small-size tables, but I got lazy on that one ...	2017-12-14 23:29:00 -08:00
Yann Collet	2cff66b62f	version bump to v1.3.3	2017-12-14 16:11:20 -08:00
Yann Collet	8c41a9cb1e	Merge pull request #951 from facebook/lastBlock ... saves 3-bytes on small input with streaming API	2017-12-14 15:39:50 -08:00
Yann Collet	a0ac8c895c	Merge pull request #950 from facebook/srcSizeAdaptation ... fix adaptation on srcSize	2017-12-14 14:48:31 -08:00
Yann Collet	281f06e01f	saves 3-bytes on small input with streaming API ... zstd streaming API was adding a null-block at end of frame for small input. Reason is : on small input, a single block is enough. ZSTD_CStream would size its input buffer to expect a single block of this size, automatically triggering a flush on reaching this size. Unfortunately, that last byte was generally received before the "end" directive (at least in `fileio`). The later "end" directive would force the creation of a 3-bytes last block to indicate end of frame. The solution is to not flush automatically, which is btw the expected behavior. It happens in this case because blocksize is defined with exactly the same size as input. Just adding one-byte is enough to stop triggering the automatic flush. I initially looked at another solution, solving the problem directly in the compression context. But it felt awkward. Now, the underlying compression API `ZSTD_compressContinue()` would take the decision the close a frame on reaching its expected end (`pledgedSrcSize`). This feels awkward, a responsability over-reach, beyond the definition of this API. ZSTD_compressContinue() is clearly documented as a guaranteed flush, with ZSTD_compressEnd() generating a guaranteed end. I faced similar issue when trying to port a similar mechanism at the higher streaming layer. Having ZSTD_CStream end a frame automatically on reaching `pledgedSrcSize` can surprise the caller, since it did not explicitly requested an end of frame. The only sensible action remaining after that is to end the frame with no additional input. This adds additional logic in the ZSTD_CStream state to check this condition. Plus some potential confusion on the meaning of ZSTD_endStream() with no additional input (ending confirmation ? new 0-size frame ?) In the end, just enlarging input buffer by 1 byte feels the least intrusive change. It's also a contract remaining inside the streaming layer, so the logic is contained in this part of the code. The patch also introduces a new test checking that size of small frame is as expected, without additional 3-bytes null block.	2017-12-14 11:47:02 -08:00
Yann Collet	c005df136f	Merge pull request #947 from facebook/fix944 ... Fix #944	2017-12-14 10:01:52 -08:00
Yann Collet	2e97a6d464	fixed minor declaration-after-statement warning	2017-12-13 18:50:05 -08:00
Yann Collet	5432ef6921	fixes adaptation on srcSize ... This patch restores capability for each file to receive adapted compression parameters depending on its size. The bug breaking this feature was relatively silly : setting a parameter with a value "0" is supposed to be a no-op. Unfortunately, it would pin down compression parameters as if they were manually set, preventing later automatic adaptation. Unfortunately, I'm currently short of a test case that could check this situation and trigger an error. Compression parameters selection between tableID 0,1,2,3 is largely internal, leaving no trace to outside world, not even in frame header.	2017-12-13 17:45:26 -08:00
Yann Collet	d23eb9a098	zstreamtest : added missing CHECK_Z()	2017-12-13 15:35:49 -08:00
Nick Terrell	22727a7467	Fix cdict compressor repcodes	2017-12-13 11:31:20 -08:00
Yann Collet	e28305fcca	fix #944 : ZSTDMT with large files and dictionary now works correctly ... windowLog is now enforced from provided compression parameters, instead of being copied blindly from `cdict` where it could be smaller. also : - fix a minor bug in zstreamtest --mt : advanced parameters must be set before init - changed advanced parameter name to ZSTDMT_jobSize	2017-12-12 18:04:58 -08:00
Yann Collet	03832b7aa5	re-added test case ... messing with revert ... :(	2017-12-12 14:01:54 -08:00
Yann Collet	8a104fda05	Revert "Created a test case which reliably reproduces bug #944" ... This reverts commit 5098d1fbe2.	2017-12-12 12:51:49 -08:00
Yann Collet	5098d1fbe2	Created a test case which reliably reproduces bug #944 ... in zstreamtest.	2017-12-12 12:48:31 -08:00
Yann Collet	ac8e022806	Merge pull request #943 from facebook/fix942 ... Fix #942	2017-12-08 13:53:08 -05:00
Yann Collet	dfc697e967	comment clarification	2017-12-08 12:16:49 -05:00
Yann Collet	c029ee1f0b	ZSTD_initCStream_srcSize() considers "0" to mean "unknown" ... to not break existing programs relying on this behavior. Might be changed to mean "empty" in the future.	2017-12-07 17:13:10 -05:00
Yann Collet	3aa2b27a89	fix #942 : streaming interface does not compress after ZSTD_initCStream() ... While the final result is still, technically, a frame, the resulting frame expands initial data instead of compressing it. This is because the streaming API creates a tiny 1-byte buffer for input, because it believes input is empty (0-bytes), because in the past, 0 used to mean "unknown" instead. This patch fixes the issue. Todo : add a test which traps the issue.	2017-12-07 02:52:50 -05:00
Yann Collet	c173dbd6e7	no longer supported starting C++17	2017-12-04 18:00:53 -08:00
Yann Collet	7e05ef851a	Merge branch 'dev' into qemu32panic	2017-12-03 11:14:36 -08:00
Yann Collet	5e1f34b7e4	setParameter : no side-effect on setting a compression parameter ... last such side-effect was modifying cctx->loadedDictEnd on setting forceWindow. It is no a useless operation, so it's removed. No side-effect left when setting a compression parameter.	2017-12-01 21:17:09 -08:00
Yann Collet	78290874a5	fixed Visual warning on minor interface discrepancy	2017-11-29 17:01:14 -08:00
Yann Collet	d3c59edac9	removed long-range-mode tests from zstreamtest --no-big-tests	2017-11-29 16:42:20 -08:00
Yann Collet	998a93b784	simplified ZSTD_CCtx_setParametersUsingCCtxParams() ... Any ZSTD_CCtx_setParameter() shall just write the requested parameter, without further action. Any action shall be taken at parameter application only (during init). It makes it possible to just copy CCtxParams from external container to internal state, and get rid of the more complex code which was trying to compensate for missing actions.	2017-11-29 16:13:05 -08:00
Yann Collet	f98ee994c4	zstd_opt: added comments, as requested by @terrelln	2017-11-29 15:19:00 -08:00
Yann Collet	bc42bc3b1d	removed one invocation of SET_PRICE() macro	2017-11-28 16:08:56 -08:00
Yann Collet	0a0a212934	zstd_opt: changed cost formula ... There was a flaw in the formula which compared literal cost with match cost : at a given position, a non-null literal suite is going to be part of next sequence, while if position ends a previous match, to immediately start another match, next sequence will have a litlength of zero. A litlength of zero has a non-null cost. It follows that literals cost should be compared to match cost + litlength==0. Not doing so gave a structural advantage to matches, which would be selected more often. I believe that's what led to the creation of the strange heuristic which added a complex cost to matches. The heuristic was actually compensating. It was probably created through multiple trials, settling for best outcome on a given scenario (I suspect silesia.tar). The problem with this heuristic is that it's hard to understand, and unfortunately, any future change in the parser would impact the way it should be calculated and its effects. The "proper" formula makes it possible to remove this heuristic. Now, the problem is : in a head to head comparison, it's sometimes better, sometimes worse. Note that all differences are small (< 0.01 ratio). In general, the newer formula is better for smaller files (for example, calgary.tar and enwik7). I suspect that's because starting statistics are pretty poor (another area of improvement). However, for silesia.tar specifically, it's worse at level 22 (while being better at level 17, so even compression level has an impact ...). It's a pity that zstd -22 gets worse on silesia.tar. That being said, I like that the new code gets rid of strange variables, which were introducing complexity for any future evolution (faster variants being in mind). Therefore, in spite of this detrimental side effect, I tend to be in favor of it.	2017-11-28 14:07:03 -08:00
Yann Collet	b71405dc51	removed a bunch of code related to cached literal price ... optState was used both to evaluate price and to cache cost of previously calculated literals. This created a strong dependency, forcing parser to request cost in a strict order. This limitation is forbids future parser with skipping capabilities. After this patch, caching literals price still exists, but is now explicit, in a stack structure.	2017-11-28 12:32:24 -08:00
Yann Collet	03f30d9dcb	separate rawLiterals, fullLiterals and match costs ... removed one SET_PRICE() macro invocation	2017-11-28 12:14:46 -08:00
Yann Collet	eee87cd6f2	btopt: minor refactor : removed one SET_PRICE() macro invocation ... direct assignment makes operation cleaner. Also allows some (very minor) optimization (non-measurable)	2017-11-27 17:18:57 -08:00
Yann Collet	e9d1987fd7	btopt: minor speed optimization ... matchPrice is always right at beginning	2017-11-27 17:01:51 -08:00
Yann Collet	bd88f633ac	zstreamtest : in -T#s, s considered a suffix meaning "seconds" ... avoid unintentionnally triggering `seedset`, so that seed gets automatically determined when not set.	2017-11-27 12:15:23 -08:00
Yann Collet	f8d5c478af	fixed comment, reported by @gyscos	2017-11-21 10:36:14 -08:00
Yann Collet	4154aec679	fixed comment, as suggested by @terrelln	2017-11-21 10:26:17 -08:00
Yann Collet	899f2a29f6	strategy ZSTD_btopt pinned to (0) variant (faster one)	2017-11-20 11:53:20 -08:00
Yann Collet	3f457264d1	slightly improved compression speed	2017-11-19 14:40:21 -08:00
Yann Collet	42c1e64270	slightly improved ratio at -22 ... merging of repcode search into btsearch introduced a small compression ratio regressio at max level : 1.3.2 : 52728769 after repMerge patch : 52760789 (+32020) A few minor changes have produced this difference. They can be hard to spot. This patch buys back about half of the difference, by no longer inserting position at hc3 when a long match is found there. It feels strangely counter-intuitive, but works : after this patch : 52742555 (-18234)	2017-11-19 14:00:55 -08:00
Yann Collet	99435dbbab	minor : search early-out on sufficient_len for hc3 and rep ... very very small speed and ratio increases	2017-11-19 12:58:04 -08:00
Yann Collet	d100670045	btopt0 : a bit faster and weaker	2017-11-19 10:38:02 -08:00
Yann Collet	e6da37c430	created (hidden) new strategy btopt0 ... about ~+10% faster but losing ~0.01 compression ratio (note : amplitude vary a lot depending on files, but direction remains the same)	2017-11-19 10:21:21 -08:00
Yann Collet	e717a5b0dd	zstd_opt: minor speed optimization ... Calculate reference log2sums only once per serie of sequence (as opposed to once per sequence) Also: improved code comments	2017-11-18 16:24:02 -08:00
Yann Collet	d11661c3ec	fix ZSTD_COMPRESSBOUND() macro ... It was using macro `KB`, which is not defined in `zstd.h`.	2017-11-18 11:16:39 -08:00
Yann Collet	a4a20a4b2f	fix un-initialized memory warning ... harmless, but cleaner	2017-11-17 15:51:52 -08:00
Yann Collet	23767e950a	fix one UB pointer arithmetic in encoder ... Instead of calculating distance between 2 memory objects, which is UB, we extract the offset from object 1, and transfer it into object 2.	2017-11-17 13:24:51 -08:00
Yann Collet	cdade555ee	fixed one UB pointer arithmetic	2017-11-17 11:40:08 -08:00
Yann Collet	11e58d9ba4	fixed minor warning ... warning: void function returning a value (even if the return value is void)	2017-11-16 15:21:30 -08:00
Yann Collet	15768cabb5	fixed some complex scenarios ... Fixed : multithreading to compress some small data with dictionary Fixed : ZSTD_initCStream_usingCDict() Improved streaming memory usage when pledgedSrcSize is known.	2017-11-16 15:18:18 -08:00
Yann Collet	05dffe43a7	Fixed Btree update ... ZSTD_updateTree() expected to be followed by a Bt match finder, which would update zc->nextToUpdate. With the new optimal match finder, it's not necessarily the case : a match might be found during repcode or hash3, and stops there because it reaches sufficient_len, without even entering the binary tree. Previous policy was to nonetheless update zc->nextToUpdate, but the current position would not be inserted, creating "holes" in the btree, aka positions that will no longer be searched. Now, when current position is not inserted, zc->nextToUpdate is not update, expecting ZSTD_updateTree() to fill the tree later on. Solution selected is that ZSTD_updateTree() takes care of properly setting zc->nextToUpdate, so that it no longer depends on a future function to do this job. It took time to get there, as the issue started with a memory sanitizer error. The pb would have been easier to spot with a proper `assert()`. So this patch add a few of them. Additionnally, I discovered that `make test` does not enable `assert()` during CLI tests. This patch enables them. Unfortunately, these `assert()` triggered other (unrelated) bugs during CLI tests, mostly within zstdmt. So this patch also fixes them. - Changed packed structure for gcc memory access : memory sanitizer would complain that a read "might" reach out-of-bound position on the ground that the `union` is larger than the type accessed. Now, to avoid this issue, each type is independent. - ZSTD_CCtxParams_setParameter() : @return provides the value of parameter, clamped/fixed appropriately. - ZSTDMT : changed constant name to ZSTDMT_JOBSIZE_MIN - ZSTDMT : multithreading is automatically disabled when srcSize <= ZSTDMT_JOBSIZE_MIN, since only one thread will be used in this case (saves memory and runtime). - ZSTDMT : nbThreads is automatically clamped on setting the value.	2017-11-16 12:18:56 -08:00
Yann Collet	dfc14579f5	removed wrong assertion	2017-11-15 15:35:56 -08:00
Yann Collet	c55e35b2fc	removed a few specialized traces	2017-11-15 15:04:53 -08:00
Yann Collet	61c2d70c86	shortened repcode match finder implementation	2017-11-15 14:37:40 -08:00
Yann Collet	d7e9805028	fixed corruption issue	2017-11-15 13:44:24 -08:00
Yann Collet	046ea53bef	still fighting data corruption ... due to messed up tree. Seems to happen when reaching end of buffer.	2017-11-15 11:29:24 -08:00
Yann Collet	4202b2e8a6	merged rep search into btMatchSearch ... but there is a tree corruption somewhere ... bug hunt ongoing	2017-11-14 20:38:52 -08:00
Yann Collet	9a11f70dc3	merged repcode search into BT match search ... this version has same speed as branch `opt` which is itself 5-10% slower than branch `dev` (no identified reason) It does not compress exactly the same as `opt` or `dev`, maybe because it doesn't stop search after repcodes, leading to sometimes better compression, sometimes worse (by a small margin). warning : _extDict path does not work for the time being This means that benchmark module works, but file module will fail with large files (and high compression level). Objective is to fuse _extDict path into current one, in order to have a single parser to maintain.	2017-11-13 02:23:48 -08:00
Yann Collet	eb47705b18	reduced scope of multiple variables ... renamed some variables for better understanding	2017-11-10 08:31:12 -08:00
Yann Collet	100d8ad6be	lib/compress: created ZSTD_LLcode() and ZSTD_MLcode() ... transform length into code. Since transformation is needed in several places throughout the code, better write the logic in one place.	2017-11-08 12:43:05 -08:00
Yann Collet	5aa0352742	zstd_opt: simplified ZSTD_getPrice() and ZSTD_updatePrice() interface ... ZSTD_getPrice() and ZSTD_updatePrice() accept normal matchlength as argument instead of matchlength-MINMATCH, which makes them easier / more logical to use and read. Conversion is simply done internally.	2017-11-08 12:23:27 -08:00
Yann Collet	bf730e2044	zstd_opt: refactor code for improved readability ... renamed variables to be more meaningful reduced scope of multiple variables removed some useless var attribution	2017-11-08 12:07:39 -08:00
Yann Collet	4191efa993	zstd_opt: ensure sufficient_len < ZSTD_OPT_NUM to simplify some tests	2017-11-08 11:24:00 -08:00
Yann Collet	ee441d5d2b	renamed zstd_compress.h into zstd_compress_internal.h ... to emphasize the fact that all definitions it contains must remain private, accross lib/compress modules.	2017-11-07 16:15:23 -08:00
Yann Collet	8b6aecf2cb	moved a few structures from zstd_internal.h to zstd_compress.h ... which is a more precise scope	2017-11-07 16:03:14 -08:00
Yann Collet	aec56a52fb	Merge pull request #908 from facebook/ubsan ... Modified one pointer arithmetic expression to a more conformant way.	2017-11-07 11:45:34 -08:00
Yann Collet	d0ffd398d2	Merge pull request #906 from facebook/fixAutoPledge ... fix : ZSTD_compress_generic(,,,ZSTD_e_end) automatically sets pledgedSrcSize	2017-11-02 10:14:20 -07:00
Yann Collet	150354c5fe	minor refactor ... added some traces and assert related to hunting a potential ubsan error in 32-bits more (it ends up being a compiler-side issue : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82802). Modified one pointer arithmetic expression for a more conformant way.	2017-11-01 16:57:48 -07:00
Yann Collet	428e8b3bf4	fix : ZSTD_compress_generic(,,,ZSTD_e_end) automatically sets pledgedSrcSize ... as per documentation, on ZSTD_setPledgedSrcSize() : > If all data is provided and consumed in a single round, > this value (pledgedSrcSize) is overriden by srcSize instead. This wasn't applied before compression level is transformed into compression parameters. As a consequence, small input missed compression parameters adaptation. It seems to work fine now : compression was compared with ZSTD_compress_advanced(), results were the same.	2017-11-01 13:15:23 -07:00
Nick Terrell	1fc4f593da	Allow skippable frames of any size	2017-11-01 13:07:26 -07:00
Yann Collet	61e5a1adfc	removed direct call to malloc() from pool.c	2017-10-31 17:43:24 -07:00
Yann Collet	f73e15de33	Merge pull request #903 from terrelln/empty-input ... [libzstd] Fix parameter selection for empty input	2017-10-28 17:28:08 -07:00
Nick Terrell	86b8134cad	[libzstd] Fix parameter selection for empty input ... ZSTD_compress() and friends would treat an empty input as an unknown size when selecting parameters. Thus, they would drastically overallocate the context. Tell ZSTD_getParams() that the source size is 1 when it is empty.	2017-10-25 17:24:15 -07:00
Nick Terrell	b495140f67	Update BUCK files ... * Correct XXH namespace (Fixes #901) * Multithreading always enabled * GZIP/LZ4/LZMA always enabled * Legacy support always fully enabled	2017-10-25 12:47:57 -07:00
Yann Collet	97dccbbb2b	fixed zbufftest ... preserve "pledgedSrcSize=0" means "unknown" in init_advanced()	2017-10-19 14:06:02 -07:00
Yann Collet	ca1a9ebac5	fixed zlib wrapper ... it was invoking ZSTD_initCStream_advanced() with pledgedSrcSize==0 and contentSizeFlag=1 which means "empty" while the intention was to mean "unknown". The contentSizeFlag==1 is new, it is a consequence of setting this value to 1 by default. The solution selected here is to pass ZSTD_CONTENTSIZE_UNKNOWN to mean "unknown". So contentSizeFlag remains set (it wasn't in previous versions).	2017-10-18 11:22:23 -07:00
Yann Collet	1ff8a8c109	Merge pull request #891 from facebook/contentSize ... Content size	2017-10-17 17:24:51 -07:00
Yann Collet	32c9f715ae	fixed : Visual build compressing stdin with multi-threading enabled fails ... It was multiple reasons stacked : - Visual use a different code path, because ZSTD_NEWAPI is not defined - fileio.c sends `0` as `pledgedSrcSize` to mean `ZSTD_CONTENTSIZE_UNKNOWN` (fixed) - ZSTDMT_resetCCtx() interpreted `0` as "empty" instead of "unknown" (fixed)	2017-10-17 14:07:43 -07:00
Yann Collet	13bfe885aa	edited ZSTD_initCStream_advanced() comment	2017-10-16 14:06:22 -07:00
Nick Terrell	7f961ba6cd	Don't allow default tables to repeat ... It isn't useful in any case to repeat default tables. Saves a few bytes on Silesia, since we don't trigger the dictionary heuristic. Before: 211988480 => 73651998 bytes After: 211988480 => 73651721 bytes	2017-10-16 11:37:56 -07:00
Yann Collet	fc8d293460	dictionary compression use correct file size estimation ... when determining compression parameters to compress one file only. For multiple files, it still "bets" that files are going to be small. There was also a bug recently added in ZSTD_CCtx_loadDictionary_advanced() making it incapable to use pledgedSrcSize to determine compression parameters.	2017-10-14 01:21:43 -07:00
Yann Collet	5eed8e7a55	changed API comments to invite using macro ZSTD_CONTENTSIZE_UNKNOWN ... to mean "pledgedSrcSize is not known at init time" instead of `0`. Note that, a few prototypes created and documented with `0` to mean "unknown" still interpret "0" as unknown, to avoid breaking 3rd party applications which depend on this behavior. But this value is no longer recommended to mean "unknown". In some future version, it might be possible to switch "0" to mean "empty", as is already the case for several prototypes. The advantage is that pledgedSrcSize field would have same behavior accross entire API, making it easier to reason about. Note that all concerned prototypes belong to the "experimental" API section. srcSize is controlled at end of compression, so if someone uses "0" to mean "unknown" while it effectively means "empty", this is immediately caught by the compression function, which generates an error code : ZSTD_ERROR_srcSize_wrong	2017-10-14 00:32:06 -07:00
Yann Collet	beb9b4b398	fixed ZSTDMT_initCStream() when contentSizeFlag==1 by default ... and a wrong test in zstreamtest --mt	2017-10-13 19:09:30 -07:00
Yann Collet	213ef3b510	fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag, ... and a stream fuzzer test, which was incorrect (relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)	2017-10-13 19:01:58 -07:00
Yann Collet	3c1e3f8ec9	contentSizeFlag enabled by default would also fail for streaming and MT operations ... fixed	2017-10-13 18:32:06 -07:00
Yann Collet	fb44516641	ensure fParams.contentSizeFlag starts at 1 ... such default was failing for ZSTD_compressBegin/ZSTD_compressContinue fixed too	2017-10-13 17:39:13 -07:00
Yann Collet	dd18d73e7e	fileio: content size is enabled by default	2017-10-13 16:32:18 -07:00
Nick Terrell	ced6e6189c	Add DEBUGLOG() that prints FSE encoding types	2017-10-13 14:55:23 -07:00
Nick Terrell	24ac2dbd2a	Fix invalid use of dictionary offcode table ... Fixes #888.	2017-10-13 12:47:03 -07:00
Yann Collet	a9e5705077	minor code formatting ... added a trace during sequence encoding	2017-10-13 02:36:16 -07:00
Yann Collet	7f6a783862	fixed a small error in decodeCorpus ... a compressed block must be strictly smaller than its decompressed size.	2017-10-07 15:19:52 -07:00
Nick Terrell	a86a7097ec	Ensure dictionary Huff table can encode any symbol ... * Ensure that the dictionary Huffman CTable has maxSymbolValue 255. * Fix a stack buffer overflow during compression dictionary loading.	2017-10-03 13:22:13 -07:00
Yann Collet	67478f4cb0	fixed minor conversion warnings for printf ... in debug mode	2017-10-02 17:28:57 -07:00
Yann Collet	9b166d2291	Merge branch 'dev' of github.com:facebook/zstd into dev	2017-10-02 16:34:26 -07:00
Yann Collet	3b27ed41fd	Merge branch 'srcSize' into dev	2017-10-02 16:34:14 -07:00
Yann Collet	7e00df4a49	bumped version number ... and updated NEWS in anticipation for release	2017-10-02 16:27:25 -07:00
Yann Collet	004fd34fd9	Merge pull request #876 from facebook/srcSize ... CLI Fix : srcSize written in frame headers when compressing multiple files	2017-10-02 15:02:05 -07:00
Nick Terrell	86e83e926f	[libzstd] Set CLEVEL_CUSTOM correctly ... In `ZSTD_compressBegin_advanced()`, `ZSTD_parameters` are used to set the compression parameters, but the level didn't get set to `CLEVEL_CUSTOM`, so `ZSTD_compressBlock()` used the wrong parameters when checking the source size.	2017-10-02 13:43:30 -07:00
Yann Collet	5db19b8685	added comment on ZSTD_COMPRESSBOUND() ... as requested by @terrelln	2017-10-01 11:32:38 -07:00
Yann Collet	6e930c13d1	Merge branch 'dev' into compressBound	2017-10-01 11:24:02 -07:00
Yann Collet	76ac0b2d99	macro compatible with scenario where windowSize = 1024 (minimum)	2017-09-30 15:34:44 -07:00
Yann Collet	dc404119e5	ZSTD_adjustCParams_internal : minor optimization	2017-09-30 15:02:40 -07:00
Nick Terrell	c5d6dde502	Don't size -= 1 in ZSTD_adjustCParams() ... The window size could end up too small if the source size is 2^n + 1. Credit to OSS-Fuzz	2017-09-30 14:20:06 -07:00
Yann Collet	ee1ed78fcb	fix proper naming on FSE_createCTable() arguments in fse.h	2017-09-30 11:08:50 -07:00
Yann Collet	5b10345b26	added ZSTD_COMPRESSBOUND() as a macro ... ZSTD_compressBound() works fine, but is only useful for dynamic allocation. For static allocation, only a macro can provide the amount during compilation time.	2017-09-29 23:17:41 -07:00
Yann Collet	8afb151c9b	cli: fixed wrong initialization in MT mode ... It's not good to mix old and new API ZSTD_resetCStream() doesn't just set pledgedSrcSize : it also sets the CCtx for a single thread compression. Problem is, when 2+ threads are defined in cctx->requestedParams, ZSTD_compress_generic() will want to start MT compression, since initialization is supposed to have already happened (thanks to ZSTD_resetCStream()) except that the underlying ZSTDMT_CCtx* object is not created, resulting in a segfault. This is an invalid construction (correct one is to use ZSTD_CCtx_setPledgedSrcSize()). I haven't found a nice way to mitigate this impact if someone makes the same mistake. At some point, removing the old API to keep only the new API within fileio.c will limit these risks.	2017-09-29 22:14:37 -07:00
Yann Collet	fbd5ab7027	minor fix : no longer use fake srcSize during resource creation ... srcSize is read and provided at each file, not at resource creation. This used to be useful with older API, because it could not re-adapt parameters between sessions. At some point, it will be better to remove the old code, and only keep the new_api. It works fine by now.	2017-09-29 19:40:27 -07:00
Yann Collet	db1668a43b	fix : srcSize written in frame header when multiple files compressed ... This information used to be disabled when nbFiles>1. It was badly initialized later in the code, resulting in an error.	2017-09-29 18:05:18 -07:00
Yann Collet	7c9669f272	Merge pull request #873 from facebook/shorterTests ... Leaner tests	2017-09-29 17:26:46 -07:00
Yann Collet	1416bc0f07	erase existence of a buffer when it's sent out of the pool ... In some complex scenario, the buffer would be freed because it's too large, another buffer would be allocated, but fail, trigger an error, and the general buffer pool would then be freed, where the definition of the already freed buffer would be found (beyond total index, but still), and freed again, resulting in double-free error.	2017-09-29 16:27:47 -07:00
Yann Collet	e963800e27	zstdmt : fixed : buffer dst0 wasn't properly set to null after usage ... now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources() wether previous compression was completed correctly or not.	2017-09-28 23:01:31 -07:00
Yann Collet	754ae5cc0b	removed ZSTDMT_waitForAllJobsCompleted() from ZSTDMT_freeCCtx() ... as per @terrelln comment	2017-09-28 20:45:31 -07:00
Yann Collet	86b4fe5b45	adjustCParams : restored previous behavior ... unknowns srcSize presumed small if there is a dictionary (dictSize>0) and presumed large otherwise.	2017-09-28 18:14:28 -07:00
Yann Collet	b93598d6a4	zstdmt : reduced maximum nb of threads ... to avoid memory address space issues on 32-bits systems (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876416#17)	2017-09-28 13:49:12 -07:00
Yann Collet	e4ec427720	Merge branch 'dev' into shorterTests ... fixed conflicts	2017-09-28 12:19:28 -07:00
Yann Collet	8074261d00	zstdmt : move on when not enough memory for a new input buffer ... just continue operations without input forward progress, instead of an error that stops current compression session.	2017-09-28 11:46:19 -07:00
Yann Collet	2cd15dd9a4	fixed minor Visual conversion warning	2017-09-28 02:33:41 -07:00
Yann Collet	377abcc02c	zstdmt : better behavior when freeing a context right after a memory allocation error ... wait for all jobs to be completed, so that freeing can happen safely	2017-09-28 02:23:44 -07:00
Yann Collet	d6770f80af	minor : rewrite unit tests using CHECK_Z macro	2017-09-28 02:14:48 -07:00
Yann Collet	9b5b47ac93	ensure adjustCParams adjust hLog and cLog even without srcSize ... It would previously exit when srcSize is unknown. But in the case of custom parameters, hLog and cLog can still be too large in comparison with windowLog. Reduces maximum memory allocated during zstreamtest --newapi	2017-09-28 01:25:40 -07:00
Yann Collet	54a827fff0	Merge branch 'dev' into newFormats ... Fixed conflicts in zstdmt_compress.c	2017-09-27 16:39:40 -07:00
Yann Collet	e45a2aea9b	Merge pull request #869 from terrelln/dev ... [libzstd] pthread function prefixed with ZSTD_	2017-09-27 16:35:08 -07:00
Nick Terrell	b555b7ef41	[libzstd][opt] Simplify repcode logic	2017-09-27 15:30:12 -07:00
Yann Collet	ea1f50bf73	removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal() ... It does not feel "right" from a dependency perspective. ZSTD_initDCtx_internal() is triggered once, on DCtx creation, while ZSTD_decompressBegin() is invoked at the beginning of each new frame, and is also a user-facing prototype. Downside : a DCtx must be init before first usage ! This was always the intention by the way, and is documented as such. This stage is automatically done within ZSTD_decompress() and variants, and also within ZSTD_decompressStream(). Only ZSTD_decompressContinue() is impacted, it must be preceded by a ZSTD_decompressBegin(), as detailed in doc. A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.	2017-09-27 13:51:05 -07:00
Yann Collet	c994932788	fixed ZSTD_format_e value validation	2017-09-27 12:22:22 -07:00
Nick Terrell	6c41adfb28	[libzstd] pthread function prefixed with ZSTD_ ... * `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]` * Fix up `lib/common/threading.[hc]` * `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`	2017-09-27 11:48:48 -07:00
Yann Collet	ecf1778e23	updated ZSTD_format_e value validation ... also updated manual	2017-09-27 11:19:21 -07:00
Yann Collet	9416195221	changed error code when pos<=size condition is not respected ... Now pointing towards src_size or dst_size, instead of error_GENERIC.	2017-09-27 10:35:56 -07:00
Yann Collet	d56a350402	removed unsupported formats	2017-09-27 10:29:31 -07:00
Yann Collet	ca306c1c84	fixed a bug in zstreamtest ... decoder output buffer would receive a wrong size. In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size. In this version, this condition is actively checked, and the function returns an error code if this condition is not respected. This check could also be done with an assert(), but since this is a user-facing interface, it seems better to keep this check at runtime.	2017-09-27 00:39:41 -07:00
Yann Collet	cd53ac831b	fixed DCtx initialization error ... now relying on initialization of dctx->format first	2017-09-26 18:26:09 -07:00
Yann Collet	4791561c4a	silence minor gcc warning -Wempty-body ... also silence fuzz test artefacts	2017-09-26 17:57:38 -07:00
Yann Collet	c0dd960363	switch assert() position	2017-09-26 15:36:57 -07:00
Yann Collet	319c699991	created ZSTD_startingInputLength() ... as suggested by @terrelln	2017-09-26 15:36:14 -07:00
Yann Collet	8d1e97ea9c	minor fixes following @terrelln comments	2017-09-26 15:06:30 -07:00
Yann Collet	df4e9bba25	fixed constant errors for gcc in c99 mode ... C standard does not consider a `static const int` as a constant. This is a problem for initializer, and ZSTD_STATIC_ASSERT(). Replaced by macro values	2017-09-26 14:31:06 -07:00
Yann Collet	9f0b8dfbe9	Merge branch 'dev' into newFormats	2017-09-26 14:22:39 -07:00
Nick Terrell	c233bdbaee	Increase maximum window size ... * Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail on my Mac. * Maximum window size in 64-bit mode is 2GB, since that is the largest power of 2 that works with the overflow prevention. * Allow `--long=windowLog` to set the window log, along with `--zstd=wlog=#`. These options also set the window size during decompression, but don't override `--memory=#` if it is set. * Present a helpful error message when the window size is too large during decompression. * The long range matcher defaults to a hash log 7 less than the window log, which keeps it at 20 for window log 27. * Keep the default long range matcher window size and the default maximum window size at 27 for the API and CLI. * Add tests that use the maximum window size and hash size for compression and decompression.	2017-09-26 14:00:01 -07:00
Yann Collet	586df82a78	Merge pull request #862 from terrelln/static ... [zstd] Backport kernel patch from @ColinIanKing	2017-09-25 17:02:40 -07:00
Yann Collet	52a1d1c6dc	added ZSTD_DCtx_reset()	2017-09-25 16:56:48 -07:00
Yann Collet	5d8fdd1641	Merge pull request #855 from terrelln/maxoff ... [libzstd] Increase MaxOff	2017-09-25 16:34:29 -07:00
Nick Terrell	76cb38d085	[zstd] Backport kernel patch from @ColinIanKing ... * Make the U32 table in `FSE_normalizeCount()` static. * Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com. * Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED). * Benchmarks showed no difference in speed.	2017-09-25 16:18:23 -07:00
Yann Collet	f2a913862c	added ZSTD_decompress_generic_simpleArgs()	2017-09-25 15:46:34 -07:00
Yann Collet	6ee05a02b8	added ZSTD_decompress_generic() ... same as ZSTD_decompressStream(), just for a similar feeling as the compression side, which uses ZSTD_compress_generic()	2017-09-25 15:41:48 -07:00
Yann Collet	b8d4a3887f	introduced constant ZSTD_frameIdSize ... within zstd_internal.h This is the size of magic number. Avoids using `4` directly in source code, which is a bit less meaningful.	2017-09-25 15:26:18 -07:00
Yann Collet	044fb4c057	implemented magic-less frame decoder	2017-09-25 15:12:09 -07:00
Yann Collet	62568c9a42	added capability to generate magic-less frames ... decoder not implemented yet	2017-09-25 14:26:26 -07:00
Nick Terrell	bbe77212ef	[libzstd] Increase MaxOff	2017-09-25 13:36:18 -07:00
Yann Collet	96f0cde31a	minor function rename ... ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams _usingX is clear. _advanced feels redundant	2017-09-24 16:47:02 -07:00
Yann Collet	7c3dea42ce	added prototypes for advanced parameters for decompression API ... required to decode custom formats	2017-09-24 15:57:29 -07:00
Yann Collet	e60f48c549	Merge branch 'dev' into newFormats	2017-09-24 14:33:37 -07:00
Yann Collet	8977224b9b	Merge pull request #859 from terrelln/31 ... Prepare for ZSTD_WINDOWLOG_MAX == 31	2017-09-22 09:01:39 -07:00
Nick Terrell	d6abb28951	Prepare for ZSTD_WINDOWLOG_MAX == 31	2017-09-21 17:18:41 -07:00
Yann Collet	cd3115b284	added control from frame content size at end of decompression ... adding check at end of single-pass ZSTD_decompressFrame(). Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3	2017-09-21 16:21:10 -07:00
Yann Collet	645563583e	Merge branch 'dev' into newFormats	2017-09-21 16:08:06 -07:00
Yann Collet	f97c2dbd39	created ZSTD_format declaration	2017-09-21 16:07:29 -07:00
Yann Collet	da74aabc00	Merge pull request #850 from terrelln/fse-optimal ... [fse] Fix FSE_optimalTableLog() for srcSize==1	2017-09-19 14:59:21 -07:00
Yann Collet	c399ab4804	Merge pull request #849 from terrelln/30 ... [bitstream] Allow adding 31 bits at a time	2017-09-19 14:25:10 -07:00
Nick Terrell	74718d7e43	[bitstream] Allow adding 31 bits at a time	2017-09-19 13:57:33 -07:00
Nick Terrell	6c9ed76676	[ldm] Fix corner case where minMatch < 8 ... There is a potential read buffer overflow when minMatch < 8. fix-fuzz-failure	2017-09-19 13:49:37 -07:00
Nick Terrell	18442a31ff	[libzstd] Fix bad window size assert ... The window size is not validated or used in the one-pass API, so there shouldn't be an assert based on it. fix-fuzz-failure	2017-09-19 13:47:59 -07:00
Yann Collet	cb8b471e8b	Merge branch 'dev' of github.com:facebook/zstd into dev	2017-09-18 14:48:23 -07:00
Yann Collet	7d1ff3817b	fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream() ... previous result was over-estimated by counting streaming buffers twice	2017-09-18 14:47:34 -07:00
Nick Terrell	cae3e3c652	[fse] Fix FSE_optimalTableLog() for srcSize==1	2017-09-18 14:11:18 -07:00
Yann Collet	72a80515ec	Merge pull request #848 from terrelln/fparams ... [block] Don't use fParams in ZSTD_decompressBlock()	2017-09-18 13:48:31 -07:00
Yann Collet	539b91ee9b	minor : added assert in bt	2017-09-16 23:41:58 -07:00
Nick Terrell	5f22479517	[block] Don't use fParams in ZSTD_decompressBlock()	2017-09-15 17:37:20 -07:00
Yann Collet	77c137b3ae	minor comment refactor	2017-09-14 15:12:57 -07:00
Yann Collet	335780c427	fixed too strong alignment assert in ZSTD_initStaticCCtx() ... 64-bits fields are only 32-bits aligned on 32-bits CPU	2017-09-13 16:35:29 -07:00
Yann Collet	f1571dad8f	Merge pull request #838 from stellamplau/ldm-mergeDev ... Add long distance matcher	2017-09-13 13:24:08 -07:00
Yann Collet	4120a7fd5a	Merge pull request #837 from facebook/libzstd-nomt ... makes it possible to compile libzstd in single-thread mode without zs…	2017-09-12 17:13:17 -07:00
Yann Collet	3306bcb0e6	fix #820 : GCC v3.x 32-bits doesn't define 64-bits intrinsic ... resulting in undefined symbol error. Push the requirement to GCC 4 for now. Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead. __LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits) Might be better, but seems also more complex, hence more prone to side effects. Keeping the simple solution for now (just rely on __GNUC__)	2017-09-11 15:17:31 -07:00
Stella Lau	eb3327c10a	Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev	2017-09-11 15:00:01 -07:00
Stella Lau	f902bf9676	Merge branch 'ldm-integrate' into ldm-mergeDev	2017-09-11 14:55:29 -07:00
Yann Collet	f325ee4e84	fixed pass-through warning	2017-09-11 14:37:03 -07:00
Stella Lau	0d1b54db61	Explicitly cast raw numerals when left-shifting	2017-09-11 14:28:18 -07:00
Yann Collet	0d6ecc72a3	makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c (#819)	2017-09-11 14:09:34 -07:00
Yann Collet	ce31004f20	fix following suggestions by @terrelln	2017-09-11 13:12:52 -07:00
Yann Collet	b3f33ccfb3	use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream() ... Use same definition as public one minor : reduce allocated buffer size in some cases (when frameContentSize is known and == windowSize)	2017-09-09 14:37:28 -07:00
Yann Collet	058ed2ad33	ZSTD_decodingBufferSize_min() ... supporting function for bufferless streaming API (ZSTD_decompressContinue()) makes it possible to correctly size a round buffer for decoding using this API. also : added field blockSizeMax within ZSTD_frameHeader, as it's a necessary information to know when to restart at beginning of decoding buffer.	2017-09-09 01:03:29 -07:00
Yann Collet	3128e03be6	updated license header ... to clarify dual-license meaning as "or"	2017-09-08 00:09:23 -07:00
Stella Lau	360428c5d9	Move ldm functions to their own file	2017-09-06 18:09:26 -07:00
Yann Collet	baa37c3362	programs/Makefile : better support for GNU conventions ... see https://www.gnu.org/prep/standards/html_node/Command-Variables.html	2017-09-06 16:53:59 -07:00
Yann Collet	3a12531a3d	lib/Makefile : better support for GNU conventions ... see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html	2017-09-06 16:35:49 -07:00
Yann Collet	1c7b914cdf	update README on BUCK file	2017-09-06 16:23:39 -07:00
Yann Collet	36374cc3b4	update and clarify lib/README	2017-09-06 16:15:18 -07:00
Stella Lau	2b99d696de	Remove debug code	2017-09-06 15:57:26 -07:00
Stella Lau	eeff55dfa8	Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev	2017-09-06 15:56:32 -07:00
Yann Collet	ad0046244f	Merge pull request #831 from terrelln/split-compress ... Split parsers out of zstd_compress.c	2017-09-06 10:01:27 -07:00
Stella Lau	9e4060200b	Add tests and fix pointer alignment	2017-09-06 09:14:05 -07:00
Stella Lau	c706de5395	Rename and add short ldm parameters in cli	2017-09-05 21:11:18 -07:00
Stella Lau	98b85426f1	Fix setting of nextToUpdate at end of ldm matcher	2017-09-05 20:41:37 -07:00
Nick Terrell	721726d688	Split parsers out of zstd_compress.c	2017-09-05 17:10:25 -07:00
Stella Lau	08d33fe1c9	Fix parameter handling in copyCCtx with cdict	2017-09-05 15:50:20 -07:00
Stella Lau	fd0071da29	Fix parameter handling with ZSTD_copyCCtx	2017-09-05 15:34:17 -07:00
Stella Lau	643d28c701	Add ldm options to 'man zstd'	2017-09-05 11:27:15 -07:00
Nick Terrell	423b133568	[POOL] Allow free on NULL when multithreading is disabled	2017-09-05 11:18:13 -07:00
Stella Lau	67d4a6161c	Add ldmBucketSizeLog param	2017-09-02 21:55:29 -07:00
Stella Lau	a1f04d518d	Move hashEveryLog to cctxParams and update cli	2017-09-01 15:05:47 -07:00
Stella Lau	767a0b3be1	Move ldm hashLog, bucketLog, and mml to cctxParams	2017-09-01 12:24:59 -07:00
Yann Collet	8a5c0c98ae	restored 32-bits decoder ability to decode long offsets (>32 MB, levels 21+)	2017-09-01 11:56:57 -07:00
Yann Collet	36aa8b5999	improved decoding speed	2017-09-01 11:40:59 -07:00
Stella Lau	17d8e0bdcc	Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate	2017-09-01 10:19:38 -07:00
Stella Lau	8081becadc	Add long distance matching as a CCtxParam	2017-09-01 09:18:58 -07:00
Yann Collet	d963daa6a9	fixed minor warning (empty translation unit)	2017-09-01 00:12:07 -07:00
Yann Collet	3704507774	fixed decompression bug reported by @Etsukata (#828)	2017-09-01 00:05:37 -07:00
Yann Collet	369c29dd1a	fixed impact of merge conflict for longRange	2017-08-31 18:25:56 -07:00
Yann Collet	d7ad99b2ab	Merge branch 'longRangeMatcher' into dev	2017-08-31 18:08:37 -07:00
Stella Lau	6a546efb8c	Add long distance matcher ... Move last literals section to ZSTD_block_internal	2017-08-31 12:53:19 -07:00
Yann Collet	b0cb081dc8	last batch of header files changed to reflect new license (#825) ... only remains to update contrib/linux-kernel (@terrelln)	2017-08-31 12:20:50 -07:00
Yann Collet	e21384fffb	fixed more file headers after license change (#825)	2017-08-31 12:11:57 -07:00
Yann Collet	e9dc204f42	fixed a bunch of headers after license change (#825)	2017-08-31 11:24:54 -07:00
Stella Lau	90a31bfa16	Pass dictMode to ZSTDMT_initCStream; fix nits ... - Return error code in estimate{CCtx,CStream}Size functions	2017-08-30 16:19:07 -07:00
Stella Lau	ee65701720	Minor fixes; remove formatting only changes	2017-08-29 20:27:35 -07:00
Stella Lau	a6e20e1bd7	Add test for raw content starting with dict header	2017-08-29 18:36:18 -07:00
Stella Lau	623e3cd40b	Use ZSTD_dm_rawContent in zstdmt_compress	2017-08-29 18:04:32 -07:00
Stella Lau	82d636b76a	Rename applyCCtxParams()	2017-08-29 18:03:06 -07:00
Stella Lau	4e835720bf	Delay creation of ZSTDMT_CCtx	2017-08-29 17:58:32 -07:00
Stella Lau	c7a18b7c21	Localize 'dictMode' from cctx to function param	2017-08-29 15:52:24 -07:00
Yann Collet	d6ddb879da	Merge pull request #817 from terrelln/pool-custom-alloc ... [pool] Accept custom allocators	2017-08-29 13:05:39 -07:00
Stella Lau	c88fb9267f	Replace 'byReference' with enum	2017-08-29 11:55:02 -07:00
Nick Terrell	9822f97721	[error] Don't guard undef X with ifdef X	2017-08-29 11:54:38 -07:00
Stella Lau	b5b9275e67	Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced()	2017-08-29 10:49:29 -07:00
Stella Lau	0e56a84a1e	Fix getting cParams from CCtxParams	2017-08-28 19:25:17 -07:00
Nick Terrell	02033be08c	[pool] Visual Studios disallows empty structs	2017-08-28 17:19:01 -07:00
Nick Terrell	7c365eb02c	[threading] Fix ERROR macro after including windows.h	2017-08-28 16:25:02 -07:00
Bernhard M. Wiedemann	cf689b84f9	Sort input file list ... in order to make builds reproducible in spite of indeterministic filesystem readdir order. See https://reproducible-builds.org/ for why this is good.	2017-08-26 17:08:00 +02:00
Stella Lau	024098a47d	Fix parameter retrieval from cdict	2017-08-25 17:58:28 -07:00
Stella Lau	2adde898c8	Fix typo with ZSTDMT_parameter	2017-08-25 16:13:40 -07:00
Stella Lau	18224608ff	Remove ZSTD_setCCtxParameter()	2017-08-25 13:58:41 -07:00
Stella Lau	0744592d38	Add function initializing cctxParams from clevel	2017-08-25 13:36:47 -07:00
Stella Lau	9911153723	Move jobSize and overlapLog in zstdmt to cctxParams	2017-08-25 13:14:51 -07:00
Stella Lau	de5193422d	Distinguish between jobParams and cctxParams in zstdmt	2017-08-25 11:36:17 -07:00
Stella Lau	eb7bbab36a	Remove ZSTD_p_refDictContent and dictContentByRef	2017-08-25 11:11:45 -07:00
Nick Terrell	db3f5372df	[zstdmt] Use POOL_create_advanced()	2017-08-24 18:12:28 -07:00
Nick Terrell	de6c6bce85	Fix zstd_internal.h for C++ mode	2017-08-24 18:09:50 -07:00
Nick Terrell	26dc040a7b	[pool] Accept custom allocators	2017-08-24 17:01:41 -07:00
Nick Terrell	89dc856cae	[pool] Fix formatting	2017-08-24 16:48:32 -07:00
Stella Lau	15fdeb9e41	Enforce nbThreads<=1 for estimateCCtxSize	2017-08-24 16:28:49 -07:00
Nick Terrell	376f435914	[dictBuilder] Set default compression level to 3	2017-08-24 16:21:05 -07:00
Stella Lau	2fbf0285b2	Fix interaction with ZSTD_setCCtxParameter() and cleanup	2017-08-24 11:25:41 -07:00
Stella Lau	fd9bf42516	Fix forceWindow and dictMode setting for zstdmt jobs	2017-08-23 19:16:57 -07:00
Stella Lau	bf3108fb50	Ensure zstdmt uses 'job version' of cctx parameters	2017-08-23 17:03:31 -07:00
Stella Lau	1c81f725ff	Remove duplicated testing code	2017-08-23 15:47:15 -07:00
Stella Lau	64ce49426b	Fix cstream compression level	2017-08-23 12:30:47 -07:00
Stella Lau	5bc2c1e982	Add prototype support for customMem with cctxParams	2017-08-23 12:03:30 -07:00
Yann Collet	e9ce1208a1	Merge pull request #812 from facebook/longRangeFix ... fixed extraordinary scenario where all fields use maximum nbBits	2017-08-23 11:35:28 -07:00
Yann Collet	74cde5a4d8	Merge pull request #813 from stellamplau/highbit32fix ... Fix undefined behavior when srcSize==1	2017-08-23 11:31:06 -07:00
Stella Lau	6f1a21c7e9	Remove formatting-only changes	2017-08-23 10:24:19 -07:00
Dmitriy Titarenko	20f715d709	Fix displayLevel overflow	2017-08-23 15:56:15 +05:00
Stella Lau	11303778d0	Add function to make cctxParams from ZSTD_parameters	2017-08-22 14:53:13 -07:00
Yann Collet	bd9c8ca146	Merge pull request #811 from terrelln/segmentSize ... [cover] Fix end condition for small dictionary	2017-08-22 14:36:30 -07:00
Stella Lau	23fc0e41fa	Remove 'opaque' naming from internal functions	2017-08-22 14:24:47 -07:00
Stella Lau	8fd1636776	Remove unused functions	2017-08-22 13:33:58 -07:00
Yann Collet	6b2b6a9bd5	fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously ... can only happen if windowLog>=27 (level 22 --ultra)	2017-08-22 12:09:21 -07:00
Stella Lau	e50ed1fa3a	Fix undefined behavior when srcSize==1	2017-08-22 11:55:42 -07:00
Stella Lau	60e1bc617c	Explicitly create a job cctxParam for multithreading	2017-08-21 15:39:37 -07:00
Stella Lau	5b956f4753	Comment out CCtx_param versions of CDict functions	2017-08-21 14:49:16 -07:00
Nick Terrell	29c2d9a4d0	[cover] Turn down notification for ZDICT subroutines	2017-08-21 14:28:31 -07:00
Nick Terrell	98de3f6847	[cover] Add dictionary size to compressed size	2017-08-21 14:23:17 -07:00
Yann Collet	78c3d16bf4	Merge pull request #809 from terrelln/dev ... [cover] Fix divide by zero	2017-08-21 13:33:19 -07:00
Nick Terrell	9a54a315aa	[cover] Convert score to U32 and check for zero	2017-08-21 13:30:07 -07:00
Stella Lau	fd8a25786e	Check parameters are valid in initCCtxParams	2017-08-21 13:23:35 -07:00
Stella Lau	1c0dbe81b1	Add documentation for CCtx_params	2017-08-21 13:18:00 -07:00
Nick Terrell	d49eb40c03	[cover] Stop when segmentSize is less than d	2017-08-21 13:10:03 -07:00
Stella Lau	939f954285	Pass ZSTD_CCtx_params as const ptr when possible	2017-08-21 12:57:18 -07:00
Stella Lau	73c73bf16a	Reduce code duplication in zstreamtest	2017-08-21 12:41:19 -07:00
Stella Lau	560b34f6d2	Return error code when initializing NULL cctxParams	2017-08-21 11:52:26 -07:00
Stella Lau	25be09c6b4	Set some parameters to zero before initializing cdict	2017-08-21 11:35:46 -07:00
Yann Collet	232d62b637	fixed a few headers that were too hastily copy/pasted during last license change	2017-08-21 11:24:32 -07:00
Nick Terrell	f306d400c0	[cover] Fix divide by zero	2017-08-21 11:12:11 -07:00
Stella Lau	502031ca10	Use cctxParam version of createCDict internally	2017-08-21 11:00:44 -07:00
Stella Lau	91b30dbe84	Remove test parameter	2017-08-21 10:09:06 -07:00
Stella Lau	f181f33bdf	Disable tests and refactor	2017-08-21 01:59:08 -07:00
Stella Lau	023b24e6d4	Add cctx param tests	2017-08-20 22:55:07 -07:00
Yann Collet	7db552676e	reduced pool queue to 0 to save memory ... fixed : pool performance when jobs are fires fast and queueSize==0	2017-08-19 15:07:54 -07:00
Stella Lau	6cee6e07e5	Add internal createCDict function	2017-08-18 22:48:31 -07:00
Stella Lau	d775519296	Add cctxParam versions of internal functions	2017-08-18 17:37:58 -07:00
Yann Collet	32fb407c9d	updated a bunch of headers ... for the new license	2017-08-18 16:52:05 -07:00
Stella Lau	63b8c98531	Pass cctx parameters to MTCtx	2017-08-18 16:17:24 -07:00
Stella Lau	399ae013d4	Add function to apply cctx params	2017-08-18 13:01:55 -07:00
Stella Lau	81d89d82a6	Move nbThreads to cctx params	2017-08-18 12:08:57 -07:00
Stella Lau	2300c58a6f	Move dictContentByRef to cctx params	2017-08-18 12:03:16 -07:00
Stella Lau	b6cb2ed8cb	Move dictMode to cctxParams	2017-08-18 11:43:31 -07:00
Stella Lau	97e27affcb	Move compression level to cctx params	2017-08-18 11:20:08 -07:00
Stella Lau	c0221124d5	Add function to set opaque parameters	2017-08-17 19:30:22 -07:00
Stella Lau	4169f49171	Add initialization/allocation functions for opaque params	2017-08-17 18:45:04 -07:00
Stella Lau	ade95b8bed	Add opaque interfaces for static initialization	2017-08-17 18:13:08 -07:00
Stella Lau	699f11b4f7	Create opaque parameter structure	2017-08-17 17:33:46 -07:00
Yann Collet	f9e6590715	Merge pull request #796 from terrelln/is-error ... [FSE][HUF] Inline error checks	2017-08-15 12:37:28 -07:00
Yann Collet	2dbcfc6994	Merge pull request #794 from terrelln/force-inline ... [libzstd] Fix FORCE_INLINE macro	2017-08-15 12:03:44 -07:00
Nick Terrell	07c6ff588e	[FSE][HUF] Inline error checks ... Caught by Clang's optimization remarks.	2017-08-15 11:23:28 -07:00
Nick Terrell	565e925eb7	[libzstd] Fix FORCE_INLINE macro	2017-08-14 21:12:05 -07:00
Roman Gershman	b9d4f4fb74	Fix ZSTD_estimateDStreamSize function after ZSTD_DStream and ZSTD_DCtx were merged	2017-08-13 13:29:42 +03:00
Nick Terrell	9ba97182d1	[CI] Add gcc7build test	2017-08-08 13:28:56 -07:00
Yann Collet	d9f2893eb9	Merge pull request #782 from terrelln/dstSizeTooSmall ... Fix compression failure on incompressible data	2017-08-07 14:52:02 -07:00
Yann Collet	8049556928	Merge pull request #778 from terrelln/bad-huff ... [libzstd] Fix bug in Huffman decompresser	2017-08-07 14:05:58 -07:00
Nick Terrell	abe12b3399	[libzstd] Fix bug in Huffman decompresser ... The zstd format specification doesn't enforce that Huffman compressed literals (including the table) have to be smaller than the uncompressed literals. The compressor will never Huffman compress literals if the compressed size is larger than the uncompressed size. The decompresser doesn't accept Huffman compressed literals with 4 streams whose compressed size is at least as large as the uncompressed size. * Make the decompresser accept Huffman compressed literals whose size increases. * Add a test case that exposes the bug. The compressed file has to be statically generated, since the compressor won't normally produce files that expose the bug.	2017-08-07 12:37:48 -07:00
Nick Terrell	308047eb5d	Fix compression failure on incompressible data ... If the destination buffer is the minimum allowed size in `ZSTD_compressSequences()` (2^17), then if the block isn't compressible compression might fail with `dstSize_tooSmall`, when it should instead emit a raw uncompressed block. Additionally, `ZSTD_compressLiterals()` implicitly called `ZSTD_noCompressLiterals()` if Huffman compression failed. Make that explicit.	2017-08-07 11:45:24 -07:00
Stella Lau	73ba58955f	Signal after finishing job when queueSize=0	2017-08-01 20:12:06 -07:00
Stella Lau	1d76da1d87	Replace marker with queueEmpty variable and update pool.h comment	2017-08-01 12:30:16 -07:00
Stella Lau	5adceeed01	Allow queueSize=0 in pool.c and update poolTests	2017-07-31 10:10:16 -07:00
Yann Collet	e1222544be	Merge pull request #753 from paulcruz74/adapt-approach-3 ... adaptive compression v1	2017-07-27 10:00:10 -07:00
Nick Terrell	ae20d413da	[libzstd] Fix CHECK_V_F macros	2017-07-25 12:52:01 -07:00
Yann Collet	a90b16e150	Visual blind fix 2	2017-07-20 15:57:55 -07:00
Yann Collet	b4d460f32c	pool.c : blindfix for Visual warnings	2017-07-20 01:13:14 -07:00
Yann Collet	3974d2b38a	blind fix for Windows Multithreading module ... adds a fake 0 return value for mutex/cond init	2017-07-19 13:33:21 -07:00
Paul Cruz	6945b3c43d	removed previous version of completion for compression	2017-07-19 11:51:50 -07:00
Yann Collet	b71363b967	check pthread_*_init() success condition	2017-07-19 01:05:40 -07:00
Nick Terrell	cc1522351f	[libzstd] Fix bug in Huffman encoding ... Summary: Huffman encoding with a bad dictionary can encode worse than the HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible input, and even if we did, the dictionaries Huffman table could be ill suited to compressing actual data. The fast optimization doesn't seem to improve compression speed, even when I hard coded fast = 1, the speed didn't improve over hard coding it to 0. Benchmarks: $ ./zstd.dev -b1e5 Benchmarking levels from 1 to 5 1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s 2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s 3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s 4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s 5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s $ ./zstd -b1e5 Benchmarking levels from 1 to 5 1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s 2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s 3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s 4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s 5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s $ ./zstd.dev -b1e5 ../silesia.tar Benchmarking levels from 1 to 5 1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s 2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s 3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s 4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s 5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s $ ./zstd -b1e5 ../silesia.tar Benchmarking levels from 1 to 5 1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s 2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s 3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s 4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s 5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s Test Plan: I added a test case to the fuzzer which crashed with ASAN before the patch and succeeded after.	2017-07-18 13:20:40 -07:00
Yann Collet	77d67fb167	Merge pull request #766 from terrelln/real-block-split ... [libzstd] Pull optimal parser state out of seqStore_t	2017-07-18 08:26:24 -07:00
Yann Collet	14c83b05c7	Merge pull request #765 from terrelln/real-block-split ... [libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()	2017-07-17 19:25:55 -07:00
Nick Terrell	7a28b9e4a3	[libzstd] Pull optimal parser state out of seqStore_t	2017-07-17 15:29:11 -07:00
Yann Collet	3381bf4b84	Merge pull request #764 from terrelln/real-block-split ... [libzstd] Refactor ZSTD_compressSequences()	2017-07-17 14:46:01 -07:00
Nick Terrell	e198230645	[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()	2017-07-17 12:27:24 -07:00
Nick Terrell	634f012420	[libzstd] Refactor ZSTD_compressSequences()	2017-07-17 11:36:11 -07:00
Paul Cruz	50ce4eaeb6	added error detection for pthread initialization, added compression completion measurement, fixed const values	2017-07-17 10:12:44 -07:00
Yann Collet	3b0cff3c33	fixed clang's -Wdocumentation	2017-07-13 18:58:30 -07:00
Yann Collet	2bd6440be0	pinned down error code enum values ... Note : all error codes are changed by this new version, but it's expected to be the last change for existing codes. Codes are now grouped by category, and receive a manually attributed value. The objective is to guarantee that error code values will not change in the future when introducing new codes. Intentionnal empty spaces and ranges are defined in order to keep room for potential new codes.	2017-07-13 17:12:16 -07:00
Nick Terrell	830ef4152a	[libzstd] Increase granularity of FSECTable repeat mode	2017-07-13 12:45:39 -07:00
Yann Collet	d985319337	Merge pull request #759 from terrelln/real-block-split ... [libzstd] Pull CTables into sub-structure	2017-07-13 10:24:19 -07:00
Yann Collet	3a60efd3a9	policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) ... Previously, ZSTDMT would refuse to create the compressor. Also : increased ZSTDMT_NBTHREADS_MAX to 256, updated doc, and added relevant test	2017-07-13 10:17:23 -07:00
Yann Collet	132e6efd76	switched ZSTDMT_compress_advanced() last argument to overlapLog ... overlapRLog (== 9 - overlapLog) was a bit "strange" as all other public entry points use overlapLog	2017-07-13 02:22:58 -07:00
Yann Collet	4e77f7761d	clarified comment on ZSTD_p_contentSizeFlag	2017-07-13 02:09:07 -07:00
Nick Terrell	de0414b736	[libzstd] Pull CTables into sub-structure	2017-07-12 19:49:19 -07:00
Yann Collet	8ef666c325	slightly increased buffer pool, to cover normal "full load" scenarios ... 2 buffers per active worker + 1 buffer for input loading + 1 buffer for "next input" when submitting current one + 1 buffer stuck in queue	2017-07-12 14:23:34 -07:00
Yann Collet	052a95f77c	fix : ZSTDMT_compress_advanced() correctly generates checksum ... when params.fParams.checksumFlag==1. This use case used to be impossible when only ZSTD_compress() was available	2017-07-11 17:18:26 -07:00
Yann Collet	2a62f48bf4	release input buffers from inside worker thread ... buffers are released sooner, which makes them available faster for next job. => decreases total nb of buffers necessary	2017-07-11 15:56:40 -07:00
Yann Collet	57236184af	buffer pool : all buffers have same size ... to reduce memory fragmentation. They can be used for in or out, interchangeably.	2017-07-11 15:17:25 -07:00
Yann Collet	34b2b95631	zstdmt : intermediate outBuffer allocated from within worker ... reduces total amount of memory needed, since jobs in queue do not have an outBuffer pre-reserved now	2017-07-11 14:59:10 -07:00
Yann Collet	16261e6951	buffer pool can be invoked from multiple threads	2017-07-11 14:14:07 -07:00
Yann Collet	ef0ff7fe7f	zstdmt: removed margin for improved memory usage	2017-07-11 08:54:29 -07:00
Yann Collet	4616fad18b	improved ZSTDMT_compress() memory usage ... does not need the input buffer for streaming operations also : reduced a few tests time length	2017-07-10 17:16:41 -07:00
Yann Collet	670b1fc547	optimized memory usage for ZSTDMT_compress() ... Previously, each job would reserve a CCtx right before being posted. The CCtx would be "part of the job description", and only released when the job is completed (aka flushed). For ZSTDMT_compress(), which creates all jobs first and only join at the end, that meant one CCtx per job. The nb of jobs used to be == nb of threads, but since latest modification, which reduces the size of jobs in order to spread the load of difficult areas, it also increases the nb of jobs for large sources / small compression level. This resulted in many more CCtx being created. In this new version, CCtx are reserved within the worker thread. It guaranteea there cannot be more CCtx reserved than workers (<= nb threads). To do that, it required to make the CCtx Pool multi-threading-safe : it can now be called from multiple threads in parallel.	2017-07-10 16:30:55 -07:00
Yann Collet	3510efb02d	fix : custom allocator correctly propagated to child contexts	2017-07-10 14:21:40 -07:00
Yann Collet	88da8f1816	fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter() ... also : compile fuzzer with MT enabled	2017-07-10 14:02:33 -07:00
Yann Collet	e32fb0c1fe	added ZSTD_sizeof_CCtx() test	2017-07-10 12:29:57 -07:00
Yann Collet	40156a4967	bumped version nb to v1.3.1	2017-07-08 04:55:09 -07:00
Yann Collet	0f4fc6c20a	fixed several conversion warnings	2017-07-07 17:13:12 -07:00
Yann Collet	9bde061a0b	fixed minor Visual compilation limitation	2017-07-07 16:14:17 -07:00
Yann Collet	593d517ebf	fixed minor cast warning	2017-07-07 16:09:47 -07:00
Yann Collet	ead4dd48f6	new field frameHeader.headerSize	2017-07-07 15:51:24 -07:00
Yann Collet	46396523c0	ZSTD_getFrameHeader : control of windowSize limits is delegated to caller ... Extracting frame header is a separate operation. It's now possible to get frame header, whatever the window size set in it.	2017-07-07 15:32:12 -07:00
Yann Collet	990449b89d	new field : ZSTD_frameHeader.frameType ... Makes frame type (zstd,skippable) detection more straighforward. ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"	2017-07-07 15:21:35 -07:00
Yann Collet	e622330a3b	extended frameHeader.windowSize to unsigned long long	2017-07-07 14:19:01 -07:00
Yann Collet	f04deff4fc	fixed #718, reported by @GregSlazinski, solution suggested by @mcmilk	2017-07-06 01:42:46 -07:00
Yann Collet	d75c0e71c4	minor code refactoring	2017-07-05 18:10:07 -07:00
Yann Collet	49af41820d	clarified status of zstdmt_compress.h API	2017-07-05 17:21:13 -07:00
Yann Collet	27e883371d	fixed wrong assert() condition ... A single job created by ZSTDMT_compress() can be < 256KB if data to compress is < 256 KB (in which case it is delegated to single thread mode)	2017-07-04 19:33:16 -07:00
Yann Collet	2cb9774f5e	more precise estimation of amount to flush at end of stream (single thread mode) ... also : can use DEBUGLEVEL variable in /tests	2017-07-04 12:39:26 -07:00
Yann Collet	6383372dec	fixed : 0-copy in NULL is UB	2017-07-04 10:36:41 -07:00
Yann Collet	5051dd39ca	Merge pull request #743 from facebook/fullbench ... compress_generic() automatic optimization opportunities	2017-07-03 21:26:38 -07:00
Yann Collet	2de2396a36	refactor ZSTDMT_compress()	2017-07-03 16:23:36 -07:00
Yann Collet	2084b041f4	fixed comments	2017-07-03 15:52:19 -07:00
Yann Collet	5a77361595	fixed wrong function name in comment	2017-07-03 15:21:24 -07:00
Nick Terrell	c80fc50a8d	[libzstd] Fix memcpy() on potential NULL source ... * `ZSTD_decompressStream_generic()` `ip` may be `NULL` for one of the calls to `memcpy()` * Assert the source is not `NULL` for calls to `memcpy()` where I believe the source should not be `NULL`.	2017-07-03 12:31:55 -07:00
Yann Collet	2485f88bf8	fixed legacy version init bug	2017-07-01 09:09:34 -07:00
cyan4973	21fdf97e00	Merge branch 'dev' into fullbench	2017-07-01 07:01:08 -07:00
cyan4973	1bafe393e4	fix : ZSTDMT_compressStream_generic() can accept NULL input ... also : converge implementations towards new version of ZSTDMT_compressStream_generic()	2017-07-01 06:59:24 -07:00
Yann Collet	58bd0e70fc	fixed : dictionary compression with new advanced API in Multi-threading mode	2017-06-30 16:01:02 -07:00
Yann Collet	d8b33a598d	Optimized ZSTDMT single-pass mode speed on large sources ... by ensuring job sizes remain "not too large"	2017-06-30 15:44:57 -07:00
Yann Collet	d5c046c609	implemented shortcut for zstd_compress_generic() in MT mode ... added ZSTDMT_compress_advanced() API	2017-06-30 14:51:01 -07:00
Yann Collet	7f40bb1c39	Merge pull request #742 from stellamplau/stack-space ... Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2	2017-06-30 14:50:23 -07:00
Stella Lau	32df49e9f8	Fix typo	2017-06-30 12:56:24 -07:00
Stella Lau	b0513b519c	Add comment to HUF_DECOMPRESS_WORKSPACE_SIZE	2017-06-30 12:53:56 -07:00
Stella Lau	4c71f59c77	Clarify typedef of rankVal_t and rankValCol_t	2017-06-30 09:52:20 -07:00
Stella Lau	28f711ef95	Rename ALIGN and ALIGN_MASK to HUF_ALIGN and HUF_ALIGN_MASK	2017-06-30 09:38:11 -07:00
Stella Lau	70ad6829e7	Delegate HUF_decompress4X_hufOnly to workspace version	2017-06-29 16:22:32 -07:00
Stella Lau	104c4d57c1	Fix bitshift error	2017-06-29 15:40:49 -07:00
Yann Collet	a3d9926c40	compression optimization opportunity ... switch to single-pass mode directly into output buffer when outputSize >= ZSTD_compressBound(inputSize). Speed gains observed with fullbench (~+15% on level 1)	2017-06-29 14:44:49 -07:00
Stella Lau	fedc94de8c	Fix pointer casting warning	2017-06-29 13:04:15 -07:00
Stella Lau	c6a5275a28	Fix alignment warnings with pointer casting	2017-06-29 12:39:34 -07:00
Stella Lau	99e315999c	Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2	2017-06-29 11:49:59 -07:00
Yann Collet	97f2bf66da	minor : fix typo	2017-06-29 11:31:40 -07:00
Yann Collet	acbef3decd	ZSTD_getFrameContentSize() is promoted to "stable" status	2017-06-29 05:19:51 -07:00
Yann Collet	590937df20	Merge pull request #739 from facebook/refPrefix ... ZSTD_refPrefix	2017-06-29 04:36:03 -07:00
Yann Collet	811deaea6f	Merge pull request #736 from terrelln/cover-default-api ... [zdict] Make COVER the default algorithm	2017-06-28 20:25:36 -07:00
Yann Collet	037466245f	refactor ZSTD_check_compressionLevel_monotonicIncrease_memoryBudget() ... use less macro statements the initial version was meant to work with STATIC_ASSERT but since it doesn't work and needs assert() it's possible to rewrite it using normally compiled code which is better for compiler. Downside : the error message is less precise. There is a DEBUGLOG(3,) to compensate.	2017-06-28 20:24:08 -07:00
Yann Collet	2bf428df45	Merge branch 'advancedAPI2' into refPrefix	2017-06-28 16:35:49 -07:00
Yann Collet	1ca76039af	fixed -Wdeclaration-after-statement	2017-06-28 15:40:21 -07:00
Yann Collet	813535105b	added function to control monotonic memory budget increase of ZSTD_defaultCParameters[0] ... It's a runtime test, based on assert(), played once, on first ZSTD_getCParams() usage, when ZSTD_DEBUG is enabled.	2017-06-28 15:34:56 -07:00
Yann Collet	adbe74a8ac	adjusted compression levels to guarantee a monotonically increasing memory budget	2017-06-28 13:22:37 -07:00
Yann Collet	33a6639039	fixed ZSTD_refPrefix with Multithread-enabled CCtx	2017-06-28 11:09:43 -07:00
Yann Collet	2e4274262d	controlled dictMode	2017-06-27 17:09:12 -07:00
Yann Collet	b7372933b8	implemented ZSTD_refPrefix()	2017-06-27 15:49:12 -07:00
Yann Collet	7d3816183f	exposed ZSTD_MAGIC_DICTIONARY in zstd.h ... makes it easier to explain ZSTD_dictMode	2017-06-27 13:50:34 -07:00
Yann Collet	fecc721fd9	added parameter ZSTD_p_refDictContent	2017-06-27 11:46:39 -07:00
Nick Terrell	5b7fd7c422	[zdict] Make COVER the default algorithm	2017-06-26 21:09:22 -07:00
Yann Collet	c7fb884eea	fixed minor conversion warning	2017-06-26 18:02:23 -07:00
Yann Collet	dde10b23fe	refactored ZSTD_estimateDStreamSize() ... now uses windowSize as argument. Also : created ZSTD_estimateDStreamSize_fromFrame()	2017-06-26 17:44:26 -07:00
Yann Collet	09ae03a570	ZSTD_estimateCDictSize_advanced() ... ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict() ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()	2017-06-26 16:47:32 -07:00
Yann Collet	0c9a915a28	ZSTD_estimateCStreamSize_advanced()	2017-06-26 16:02:25 -07:00
Yann Collet	31af8290d1	ZSTD_estimateCCtx_advanced() ... ZSTD_estimateCCtx() is now a "simple" function, taking int compressionLevel as single argument. ZSTD_estimateCCtx_advanced() takes a CParams argument, which is both more complete and more complex to generate.	2017-06-26 15:52:39 -07:00
Yann Collet	ef269c1b68	Merge pull request #725 from facebook/advancedAPI2 ... New Advanced API	2017-06-23 09:50:47 -07:00
Yann Collet	ecb0f46866	add controls over streaming buffers	2017-06-21 17:25:01 -07:00
Yann Collet	dce789281b	fixed : decompression of skippable frames in streaming mode	2017-06-21 15:53:42 -07:00
Yann Collet	204b6b7ef6	fixed streaming buffered allocation with CDict compression	2017-06-21 15:13:00 -07:00
Yann Collet	1e4129b27b	fixed dangling pointer risk, detected by @terrelln	2017-06-21 13:26:10 -07:00
Yann Collet	83095970e6	free cdictLocal faster, suggested by @terrelln	2017-06-21 12:26:40 -07:00
Yann Collet	7bd1a2900e	added ZSTD_dictMode_e to control dictionary loading mode	2017-06-21 11:50:33 -07:00
Yann Collet	9c56b12938	Merge pull request #723 from paulcruz74/dev ... Adding zstd -l	2017-06-21 09:41:55 -07:00
Yann Collet	e51d51bdf7	fixed memcpy() overlap	2017-06-20 17:44:55 -07:00
Yann Collet	466f92eaa6	removed one useless streaming compression stage, detected by @terrelln	2017-06-20 16:25:29 -07:00
Yann Collet	c3bce24ef4	fixed potential dangling pointer, detected by @terrelln	2017-06-20 16:09:11 -07:00
Yann Collet	78b8234554	fixed comments, following suggestion by @terrelln	2017-06-20 14:26:48 -07:00
Yann Collet	b44ab82f7a	ensure new ZSTD_strategy starts at value 1	2017-06-20 14:11:49 -07:00
Yann Collet	c08e649e95	first implementation of bench.c with new API ZSTD_compress_generic() ... Doesn't speed optimize this buffer-to-buffer scenario yet. Still internally defers to streaming implementation. Also : fixed a long standing bug in ZSTDMT streaming API.	2017-06-19 18:25:35 -07:00
Yann Collet	695a0a3449	fixed IA64 compilation error, by @mcmilk	2017-06-19 15:27:30 -07:00
Yann Collet	fe234bf48b	fix attempts : fullbench for VS2008	2017-06-19 15:23:19 -07:00
Nick Terrell	55f9cd4942	[libzstd] Fix UBSAN failure	2017-06-19 15:12:28 -07:00
Yann Collet	bf99150be3	update new api presentation in zstd.h and manual	2017-06-19 12:56:25 -07:00
Yann Collet	d7a3bffba9	new api : setting compression parameters is refused if a dictionary is already loaded	2017-06-19 11:53:01 -07:00
Yuri	92bafda406	INSTALL_DATA instead of INSTALL_LIB for libzstd.a ... INSTALL_LIB can be passed -s flag to strip symbols. Static libraries should not be stripped, only dynamic ones should be stripped.	2017-06-17 00:23:41 -07:00
Yann Collet	381e66cfbd	added ZSTD_clampCParams() ... now ZSTD_adjustCParams() is always successful, it always produces a valid CParams	2017-06-16 17:34:54 -07:00
Yann Collet	aee916e37c	fixed +/-1 error for pledgedSrcSizePlusOne	2017-06-16 17:02:35 -07:00
Yann Collet	d3de3d51a3	fix attempt 2 : Visual sign conversion warning	2017-06-16 16:51:33 -07:00
Yann Collet	944be54774	fixed attempt : minor Visual sign conversion warning	2017-06-16 14:05:01 -07:00
Yann Collet	b26728c9c8	added ZSTD_startNewCompression()	2017-06-16 14:00:46 -07:00
Yann Collet	a0ba849fe6	changed frameContentSize field to pledgedSrcSizePlusOne ... pledgedSrcSize is proper : it's a promise, not yet fulfilled. It will be controlled at the end. PlusOne is meant to have 0 (default) == unknown	2017-06-16 13:29:17 -07:00
Yann Collet	2cf7755da7	fix : pledgedSrcSize correctly reset to unknown in "continue" mode	2017-06-16 12:34:41 -07:00
Yann Collet	9e73f2f320	fix : correctly reset pledgedSrcSize to unknown status ... when starting a new compression with an existing context	2017-06-16 12:24:01 -07:00
Yann Collet	33873f0e74	fixed : new advanced AIP : setting nbThreads to the same value > 1	2017-06-16 12:04:21 -07:00
Yann Collet	559ee82e90	fixed : calling ZSTD_compress_generic() to end-flush a stream in multiple steps	2017-06-16 11:58:21 -07:00
Yann Collet	bd18c885a3	added ZSTD_CCtx_reset	2017-06-16 10:17:50 -07:00
Yann Collet	cc9f9b7f4c	protection : ZSTD_CONTENTSIZE_UNKNOWN automatically disables contentSizeFlag	2017-06-15 18:17:34 -07:00
Yann Collet	05ae4b2190	added protection : MT incompatible with Static allocation	2017-06-15 18:03:34 -07:00
Paul Cruz	a9b77c83e5	cleaning up code for analyzing frames	2017-06-15 14:13:28 -07:00
Yann Collet	f129fd3970	disabled MT code path when ZSTD_MULTITHREAD is not defined	2017-06-11 18:46:09 -07:00
Yann Collet	23aace9778	added control stage to MT mode	2017-06-11 18:32:36 -07:00
Yann Collet	f35e2de61c	linked newAPI to ZSTDMT	2017-06-05 18:32:48 -07:00
cyan4973	c59162e053	minor fix for -Wdocumentation	2017-06-05 00:12:13 -07:00
cyan4973	8bcbf42617	fixed g++ prototype mismatch	2017-06-04 23:52:00 -07:00
Yann Collet	8c910d2097	updated ZSTDMT streaming API ... ZSTDMT streaming API is now similar and has same capabilites as single-thread streaming API. It makes it easier to blend them together.	2017-06-03 01:15:02 -07:00
Yann Collet	58e8d793e1	made debug definitions common within zstd_internal.h	2017-06-02 18:20:48 -07:00
Yann Collet	8ddf4c22d5	fixed missing initialization	2017-06-02 17:16:49 -07:00
Yann Collet	33a7e679e5	significant zlib wrapper code refactoring ... code indentation variable scope and names constify Only coding style changes. The logic should remain the same.	2017-06-02 17:10:49 -07:00
Yann Collet	4effccbf56	zlib_wrapper's uncompress() uses ZSTD_isFrame() for routing ... more generic and safer than using own routing for magic number comparison	2017-06-02 14:27:11 -07:00
Yann Collet	dcb7535352	ensure zlibwrapper uses ZSTD_malloc() and ZSTD_free() ... which is compatible with { NULL, NULL, NULL }	2017-06-02 14:01:21 -07:00
Yann Collet	b877e834b1	minor indent	2017-06-02 13:47:11 -07:00
Yann Collet	6056e4c3eb	added POOL_sizeof() for single-thread	2017-06-02 11:36:47 -07:00
Yann Collet	c35e535002	added support for multithreading parameters	2017-06-01 18:44:06 -07:00
Yann Collet	c4a5a21c5c	created ZSTDMT_sizeof_CCtx() and POOL_sizeof() ... required by ZSTD_sizeofCCtx() while adding a ZSTDMT_CCtx*	2017-06-01 17:56:14 -07:00
Yann Collet	cd2892fd1e	protected impossible switch(){default:} with assert(0) ... can be converted into assume(0) in some future	2017-06-01 09:44:54 -07:00
Yann Collet	06589fe516	Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2	2017-05-31 10:03:20 -07:00
Yann Collet	18ab5affa5	fixed visual warning	2017-05-31 09:59:22 -07:00
Yann Collet	9a691e0f55	fixed visual warnings	2017-05-31 01:17:44 -07:00
Yann Collet	01b1549f83	finally converted ZSTD_compressStream_generic() to use {in,ou}Buffer ... replacing the older read/write variables from ZBUFF_* era. Mostly to help code readability. Fixed relevant callers.	2017-05-30 18:10:26 -07:00
Yann Collet	c4f46b94ce	ZSTD_createCCtx_advanced() now uses ZSTD_calloc() ... initially uses calloc() instead of memset(). Performance improvement is unlikely measurable, since ZSTD_CCtx is now very small, with all tables transferred into workSpace.	2017-05-30 17:45:37 -07:00
Yann Collet	deee6e523f	expose ZSTD_compress_generic_simpleArgs() ... which is a binding towards ZSTD_compress_generic() using only integral types for arguments.	2017-05-30 17:42:00 -07:00
Yann Collet	ae728a43b8	removed defaultCustomMem ... now ZSTD_customCMem is promoted as new default. Advantages : ZSTD_customCMem = { NULL, NULL, NULL}, so it's natural default after a memset. ZSTD_customCMem is public constant (defaultCustomMem was private only). Also : makes it possible to introduce ZSTD_calloc(), which can now default to stdlib's calloc() when it detects system default. Fixed zlibwrapper which depended on defaultCustomMem.	2017-05-30 17:11:39 -07:00
Yann Collet	5bcef1ada2	removed mtctx->cstream ... use the first cctx in pool when ZSTDMT is used in single-thread mode now that cctx and cstream are the same object.	2017-05-30 16:37:19 -07:00
Yann Collet	beb62b15a8	Merge branch 'dev' into advancedAPI2 ... Fixed conflic in zstd_decompress.c	2017-05-30 16:18:57 -07:00
Yann Collet	44e45e8423	added ZSTDMT_createCCtx_advanced() ... make it possible to use custom allocators	2017-05-30 16:12:06 -07:00
Yann Collet	f45ca527a1	Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2	2017-05-30 10:02:03 -07:00
Yann Collet	b6dec4c3ae	fixed minor cast warning	2017-05-27 17:09:06 -07:00
Yann Collet	e071159101	mtctx->jobs allocate its own memory space ... to make ZSTDMT_CCtx_s size predictable so that it can be included in CCtx	2017-05-27 00:21:33 -07:00
Yann Collet	b8136f019a	static dctx is incompatible with legacy support ... documented, and runtime tested	2017-05-27 00:03:08 -07:00
Yann Collet	7028cbd7fd	fixed a few code comments : ZSTD_getFrameParams => ZSTD_getFrameHeader	2017-05-25 18:29:08 -07:00
Yann Collet	cdf7e82222	Added ZSTD_initStaticCDict()	2017-05-25 18:05:49 -07:00
Dmitry V. Levin	1ea655c765	Fix typo in libzstd.a-mt make rules ... The macro name is ZSTD_MULTITHREAD, not ZSTD_MULTHREAD. Fixes: ca6fae7808 ("Add MT enabled targets for libzstd")	2017-05-25 23:43:05 +00:00
Yann Collet	57827f906f	added ZSTD_initStaticDDict()	2017-05-25 15:44:06 -07:00
Yann Collet	25989e361c	updated ZSTD_estimate?DictSize() to pass parameter byReference ... resulting ?Dict object is smaller when created byReference. Seems better than a documentation note.	2017-05-25 15:07:37 -07:00
Yann Collet	0fdc71c3dc	added ZSTD_initStaticDCtx()	2017-05-24 17:41:41 -07:00
Yann Collet	ba183005d3	merged DStream's inBuff and outBuff into a single buffer ... Saves one malloc(). Also : makes it easier to implement static allocation	2017-05-24 15:42:24 -07:00
Nick Terrell	55fc1f91fd	[zstd] Fix up formatting edge cases for clang-format	2017-05-24 13:50:10 -07:00
Yann Collet	2e4db3e531	fixed performance regression with ZSTD_decompress() on small files ... memset() was a quick fix to initialization problems, but initialize too much space (tables, buffers) which show up in decompression speed of ZSTD_decompress() since it needs to recreate DCtx at each invocation. Fixed by only initialization relevant pointers and size fields.	2017-05-24 13:15:19 -07:00
Yann Collet	11ea2f7fda	Merged ZSTD_DCtx and ZSTD_DStream objects ... They are now the same object. It's recommended to keep both types in source code as previous versions of library (<v1.3.0) still need this differentiation.	2017-05-23 16:19:43 -07:00
Yann Collet	b81f19ffce	move MEM_readMINMATCH() into zstd_opt.h ... which is its only user. Use case too narrow to belong to mem.h. renamed to ZSTD_readMINMATCH()	2017-05-23 15:41:55 -07:00
Yann Collet	c7fe262dc9	added ZSTD_initStaticCCtx() ... makes it possible to statically or externally allocate CCtx. static CCtx will only use provided memory area, it will never resize nor malloc.	2017-05-23 13:20:41 -07:00
Yann Collet	5ac72b417c	Buffered are now allocated inside workSpace	2017-05-23 11:18:24 -07:00
Yann Collet	1880337c30	Simplifier compression call graph ... Everything converge towards ZSTD_compressBegin_internal which delegated to ZSTD_copyCCtx_internal if cdict!=NULL. This simplifies routing which was previously depending on cdict.	2017-05-22 18:21:51 -07:00
Yann Collet	b0739bcf8f	simplified reset by removing full-reset policy ... this was meant to be applied prior to dictionary loading. But effectively, it seems redundant with later loading stage, so it can be skipped safely.	2017-05-22 17:45:15 -07:00
Yann Collet	1ad7c82eb5	Implemented separation between requested and applied parameters ... first version to pass cli tests with -DZSTD_NEWAPI	2017-05-22 17:06:04 -07:00
Yann Collet	24de7b0346	Implemented ZSTD_CCtx_refCDict()	2017-05-22 13:05:45 -07:00
Yann Collet	ee970398b2	Merge branch 'dev' into advancedAPI2	2017-05-22 12:33:56 -07:00
Yann Collet	8b21ec42a9	ZSTD_compress_generic() can handle dictionary compression	2017-05-19 19:46:15 -07:00
Nick Terrell	a1280406b0	[libzstd] Allow users to define custom visibility	2017-05-19 18:01:59 -07:00
Yann Collet	334a288d0d	ZSTD_CCtx_setParameter() only works during initialization stage ... and generate a stage_wrong error otherwise.	2017-05-19 11:04:41 -07:00
Yann Collet	48855fa0d2	fixed declaration-after-statement warning	2017-05-19 10:56:11 -07:00
Yann Collet	fa3671eac7	changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX ... Also : change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize() created ZSTD_BLOCKSIZELOG_MAX	2017-05-19 10:51:30 -07:00
Yann Collet	009d604e00	ZSTD_compress_generic() supports multiple successive frames ... also : clarified streaming API implementation	2017-05-19 10:17:59 -07:00
Yann Collet	6d4fef36de	Added ZSTD_compress_generic() ... Used in fileio.c (zstd cli). Need to set macro ZSTD_NEWAPI to trigger it.	2017-05-17 18:36:15 -07:00
Yann Collet	23c256e44b	removed useless variable from CCtx ... CStream's pledgedSrcSize is no longer necessary srcSize control is realized within bufferless interface.	2017-05-16 18:10:11 -07:00
Yann Collet	9f95e445ab	minor comment clarifications	2017-05-16 17:26:43 -07:00
Yann Collet	0bdb575c31	Merge branch 'dev' into advancedAPI2	2017-05-16 16:32:29 -07:00
Yann Collet	7101434ec9	pedantic : added one error check ... on a function which (today) never fails. But who knows, maybe tomorrow ...	2017-05-16 16:28:24 -07:00
Yann Collet	bfff8999c5	added prototype ZSTD_versionString()	2017-05-16 16:12:23 -07:00
Yann Collet	4eff8136aa	added prototype ZSTD_decompressBegin_usingDDict (#700)	2017-05-16 16:05:27 -07:00
Yann Collet	2d4d31c18a	removed gcc compilation flag -Wbad-function-cast ... It makes it more difficult to directly cast the result of a function, requiring to store the result in an intermediate variable. It does not necessarily help readability, and this restriction can be difficult to overcome in some constructions, like some macros. also : fixed minor Visual conversion warnings in datagencli.c	2017-05-16 11:34:38 -07:00
Yann Collet	133f0aee54	fixed redundant declarations in legacy v0.5 and v0.7 decoders ... triggered by new flag -Wredundant-decls	2017-05-15 17:44:04 -07:00
Yann Collet	83d0c764dc	added several compilation flags	2017-05-15 17:15:46 -07:00
Yann Collet	a5ffe3d370	pushed enum values for strategy by one (ZSTD_fast==1) ... this makes it possible to use `0` to mean: "do not change strategy"	2017-05-12 16:29:19 -07:00
Yann Collet	add66f816d	changed macro LOADCPARAMS by static function ZSTD_cLevelToCParams() ... for improved compiler checks. Also : ensure most parameters can receive value "0" to mean "do not change".	2017-05-12 16:01:15 -07:00
Yann Collet	b0edb7fb0e	added ZSTD_CCtx_setParameter()	2017-05-12 15:31:53 -07:00
Yann Collet	ef738c1b23	better error code when compressing using NULL CDict ... which is not allowed (but detected, and generates an error).	2017-05-12 13:55:25 -07:00
Yann Collet	db8e21d5a0	made ZSTD_compress_generic() definition accessible ... note that the implementation is not done yet.	2017-05-12 13:46:49 -07:00
Yann Collet	33eb7ac6b6	updated Advanced API proposal ... only declarations in zstd.h	2017-05-12 12:36:11 -07:00
Yann Collet	bd1964a988	Merge pull request #696 from joscollin/wip-lib-legacy-fallthrough-warn ... lib/legacy: warning: this statement may fall through	2017-05-11 10:45:01 -07:00
Yann Collet	4c1cfc0bb6	Merge pull request #695 from joscollin/wip-lib-compress-fallthrough-warn ... lib/compress: warning: this statement may fall through	2017-05-11 10:44:27 -07:00
Jos Collin	280510f2d5	lib/legacy: warning: this statement may fall through ... The following warning appears during build at sevaral places. ../lib/legacy/zstd_v04.c:819:40: warning: this statement may fall through [-Wimplicit-fallthrough=] case 7: bitD->bitContainer += (size_t)(((const BYTE*)(bitD->start))[6]) << (sizeof(size_t)*8 - 16); ../lib/legacy/zstd_v05.c:821:40: warning: this statement may fall through [-Wimplicit-fallthrough=] case 7: bitD->bitContainer += (size_t)(((const BYTE*)(bitD->start))[6]) << (sizeof(size_t)*8 - 16); ../lib/legacy/zstd_v06.c:913:40: warning: this statement may fall through [-Wimplicit-fallthrough=] case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16); ../lib/legacy/zstd_v07.c:583:40: warning: this statement may fall through [-Wimplicit-fallthrough=] case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16); Signed-off-by: Jos Collin <jcollin@redhat.com>	2017-05-11 14:27:40 +05:30
Jos Collin	7cd7a7564b	lib/compress: warning: this statement may fall through ... The following warning appears during build. ../lib/compress/huf_compress.c: In function ‘HUF_compress1X_usingCTable’: ../lib/compress/huf_compress.c:444:8: warning: this statement may fall through [-Wimplicit-fallthrough=] if (sizeof((stream)->bitContainer)*8 < HUF_TABLELOG_MAX*4+7) HUF_FLUSHBITS(stream) ^ ../lib/compress/huf_compress.c:465:18: note: in expansion of macro ‘HUF_FLUSHBITS_2’ HUF_FLUSHBITS_2(&bitC); ^~~~~~~~~~~~~~~ ../lib/compress/huf_compress.c:466:9: note: here case 2 : HUF_encodeSymbol(&bitC, ip[n+ 1], CTable); ../lib/compress/zstd_compress.c: In function ‘ZSTD_compressStream_generic’: ../lib/compress/zstd_compress.c:3366:34: warning: this statement may fall through [-Wimplicit-fallthrough=] zcs->streamStage = zcss_flush; /* pass-through to flush stage */ ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~ ../lib/compress/zstd_compress.c:3369:9: note: here case zcss_flush: Signed-off-by: Jos Collin <jcollin@redhat.com>	2017-05-11 13:17:26 +05:30
Jos Collin	05286fdd5a	lib/common: warning: this statement may fall through ... The following warning appears during the build. Fixed the review comments too. zstd/lib/common/bitstream.h: In function ‘BIT_initDStream’: zstd/lib/common/bitstream.h:277:33: warning: this statement may fall through [-Wimplicit-fallthrough=] case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16); Signed-off-by: Jos Collin <jcollin@redhat.com>	2017-05-11 09:10:02 +05:30
Nick Terrell	374f868354	Update whitespace	2017-05-10 17:48:42 -07:00
Nick Terrell	5f2c7213c7	Merge remote-tracking branch 'upstream/dev' into btopt ... * upstream/dev: (305 commits) added test for ZSTD_estimateCStreamSize() changed variable name, for clarity fixed ZSTD_estimateCStreamSize() shortened ZSTD_createCStream_Advanced() fixed symbols test added ZSTD_estimateDStreamSize() changed name frameParams into frameHeader regroup memory usage function declarations separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize() bumped version number added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() Updated ZSTD_freeCCtx() updated ZSTD_estimateCCtxSize() Updated ZSTD_sizeof_CCtx() merged CCtx and CStream as a single same object cli : -d and -t do not stop after a failed decompression added dev branch CircleCI badge added dev branch Appveyor badge keep dev branch status only creates a binary archive without the `programs` directory ...	2017-05-10 16:49:58 -07:00
Yann Collet	ba41b26405	Merge pull request #689 from facebook/cctxMerge ... Cctx merge	2017-05-10 14:53:54 -07:00
Yann Collet	cef02d9317	changed variable name, for clarity ... fhiPtr -> zfhPtr https://github.com/facebook/zstd/pull/689#discussion_r115638676	2017-05-10 11:14:08 -07:00
Yann Collet	669346fe8b	fixed ZSTD_estimateCStreamSize() ... https://github.com/facebook/zstd/pull/689#discussion_r115637721	2017-05-10 11:08:00 -07:00
Yann Collet	6fb2f24132	shortened ZSTD_createCStream_Advanced() ... https://github.com/facebook/zstd/pull/689#discussion_r115637613	2017-05-10 11:06:06 -07:00
Yann Collet	f16f4497ca	added ZSTD_estimateDStreamSize()	2017-05-09 16:18:17 -07:00
Yann Collet	542c9dfcf8	changed name frameParams into frameHeader ... ZSTD_frameParams => ZSTD_frameHeader ZSTD_getFrameParams() -> ZSTD_getFrameHeader() The new naming is more distinctive from ZSTD_frameParameters, which is used during compression. ZSTD_frameHeader is clearer in its intention to described frame header content. It also implies we are decoding a ZSTD frame, hence we are at decoding stage.	2017-05-09 15:46:07 -07:00
Yann Collet	5a36c069e7	regroup memory usage function declarations ... in a single paragraph in zstd.h, for clarity	2017-05-09 15:11:30 -07:00
Yann Collet	fa8dadb294	separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize() ... for clarity	2017-05-08 18:24:16 -07:00
Yann Collet	51652522a2	bumped version number	2017-05-08 17:52:46 -07:00
Yann Collet	a1d6704d7f	added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() ... it complements ZSTD_estimateCCtxSize() for the special case of ZSTD_initCStream_usingDict()	2017-05-08 17:51:49 -07:00
Yann Collet	7855366598	Updated ZSTD_freeCCtx() ... which can also contain streaming buffers now. Redirected ZSTD_freeCStream() towards it.	2017-05-08 17:15:00 -07:00
Yann Collet	fc5145955a	updated ZSTD_estimateCCtxSize() ... added a parameter streaming, to estimate memory allocation size when the CCtx is used for streaming (CStream). Note : this function is not able to estimate memory cost of a potential internal CDict which can only happen when starting with ZSTD_initCStream_usingDict()	2017-05-08 17:07:59 -07:00
Yann Collet	791d744279	Updated ZSTD_sizeof_CCtx() ... can now contain buffers if object used as CStream. ZSTD_sizeof_CStream() is now just a thin wrapper of ZSTD_sizeof_CCtx().	2017-05-08 16:17:30 -07:00
Yann Collet	0be6fd3429	merged CCtx and CStream as a single same object ... To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()	2017-05-08 16:08:01 -07:00
Yann Collet	d47709b6ea	Merge pull request #654 from iburinoc/splittable ... [RFC] Splittable Format and API	2017-05-08 13:41:56 -07:00
Yann Collet	a00e9599f1	removed -g from DEBUGFLAGS ... It inflates binary sizes, which is negative for the Windows build. It also makes it impossible to check if 2 different source codes get nonetheless compiled to the same binary, since checksum will be different, due to integrated source code.	2017-05-04 17:24:29 -07:00
Yann Collet	606c04c228	Merge branch 'dev' of github.com:facebook/zstd into dev	2017-05-02 12:13:52 -07:00
Yann Collet	072484a3bf	Merge pull request #683 from terrelln/odev ... [CLI] Make cover the default dictionary builder	2017-05-02 12:13:23 -07:00
Nick Terrell	f376d47c11	[CLI] Switch dictionary builder on CLI to cover	2017-05-02 11:18:27 -07:00
Nick Terrell	020b960e13	[cover] Make optimization faster	2017-05-02 11:02:48 -07:00
Nick Terrell	f2d9ef1dc0	[cover] Optimize case where d <= 8	2017-05-02 11:02:43 -07:00
Nick Terrell	865918dd04	Fix typo in zdict.h	2017-05-02 11:02:37 -07:00
Yann Collet	b184589c4c	minor code refactoring for clarity	2017-05-01 11:35:47 -07:00
Yann Collet	33c38b0925	fixed const in prototype, that Visual doesn't accept	2017-05-01 11:12:30 -07:00
Yann Collet	f39a6731ec	sync bitstream.h from fse library	2017-05-01 09:56:03 -07:00
Yann Collet	202082f285	sync bitstream from FSE project ... add assert into unsafe *_fast() variants	2017-04-28 17:00:31 -07:00
Yann Collet	89f50deec7	minor code refactoring ... clearer tables	2017-04-28 16:52:36 -07:00
Yann Collet	68a7d3d49a	added HUF_PUBLIC_API macro to huf.h ... to make it possible to control symbol visibility. Also : better separation and comments between "public" and "static" sections	2017-04-28 12:46:48 -07:00
Yann Collet	a51cab6e68	Merge pull request #678 from facebook/apiChange ... Breaking API Change around CDict	2017-04-28 10:02:45 -07:00
Yann Collet	29297c6751	Changed default level 18 (large input) ... Previous -18 : 4.7 MB/s, R:3.833 New -18 : 5.1 MB/s. R:3.825 It's a better fit within -17 (6.8 MB/s) and -19 (4.0 MB/s) The new level 18 also uses significantly less memory. And, it makes a good transition between level 17 (mml5) and level 19 (mml3). Up to now, there was no level with mml4. (note : minmatch setting can have a large impact on some (specific) datasets)	2017-04-27 17:44:01 -07:00
Yann Collet	a92cbb7004	Added a secondary test, checking dictID presence after setting noDictIdFLag=1	2017-04-27 15:08:56 -07:00
Yann Collet	d3694e6c70	removed C4204	2017-04-27 14:29:35 -07:00
Yann Collet	1c3ab0c77f	fixed init error on Visual 2008	2017-04-27 12:57:11 -07:00
Yann Collet	8b669535f8	bumped version number to v1.2.0	2017-04-27 12:50:20 -07:00
Yann Collet	77bf59ef50	added ZSTD_initCStream_usingCDict_advanced()	2017-04-27 11:43:04 -07:00
Yann Collet	f4bd857d81	created ZSTD_compress_usingCDict_advanced()	2017-04-27 11:31:55 -07:00
Yann Collet	69a54d138a	fixed compilation warning : declaration-after-statement	2017-04-27 01:11:26 -07:00
Yann Collet	31533bacce	Changed ZSTD_createCDict_advanced() ... It now only uses compressionParameters as argument. It produces many changes throughout user code, though hopefully they tend to be simple : just provide the cParams part from existing ZSTD_parameters. Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters. This change will force them to revisit this strategy and fix it, since frame parameters are effectively silently ignored in current version.	2017-04-27 00:29:04 -07:00
Yann Collet	768df129d2	changed ZSTD_compressBegin_usingCDict() ... No longer takes `pledgedSrcSize` as argument this is in line with similar functions ZSTD_compress_usingCDict() and ZSTD_initCStream_usingCDict().	2017-04-26 15:42:10 -07:00
Yann Collet	e42afbc6fa	Comply with suggested comments by @terrelln ... created FSE_CTABLE_SIZE() and FSE_DTABLE_SIZE()	2017-04-26 11:39:35 -07:00
Sean Purcell	7d37ca1d5b	Merge remote-tracking branch 'origin/dev' into splittable	2017-04-21 14:18:39 -07:00
Yann Collet	7271203bdb	transferred entropy scratch space from CCtx into workSpace ... Saved 6 KB	2017-04-20 23:21:19 -07:00
Yann Collet	a408645f50	made some room for entropy scratch space	2017-04-20 23:09:39 -07:00
Yann Collet	71aaa32c3c	transferred FSE tables from CCtx into workspace ... Saved 5 KB from CCtx	2017-04-20 23:03:38 -07:00
Yann Collet	71ddeb67b1	made room in workspace for FSE tables ... still need to be transferred from CCtx into workspace	2017-04-20 22:54:54 -07:00
Yann Collet	a34a39c183	changed size evaluation of entropy tables ... so that memcpy() does no longer depends on fse pointer being a static table	2017-04-20 18:26:25 -07:00
Yann Collet	7bb60b17d8	init entropy table pointers only once ... per workSpace resize	2017-04-20 17:38:56 -07:00
Yann Collet	e6fa70a0a1	reorganized ZSTD_resetCCtx_internal() ... clearer separation between variables and buffers clearer buffers category kept static buffers at the beginning, favoring cache locality (it will be easier to add FSE tables there later) This break a few assumptions that hashTable was always at the beginning. This is fixed. And remaining assumptions (namely that tables stand next to each other in memory) are now tested with assert.	2017-04-20 17:28:31 -07:00
Yann Collet	c17e020c9a	disable assert when compiling paramgrill ... paramgrill is a benchmark calibration function. Speed accuracy is critical, it cannot be altered by assert.	2017-04-20 12:50:02 -07:00
Yann Collet	16f9c572fc	Merge branch 'dev' into compressionFlow	2017-04-20 11:16:40 -07:00
Yann Collet	e348dad305	minor long line reformatting	2017-04-20 11:14:13 -07:00
Yann Collet	e847730452	slightly refined README comments on lib-mt	2017-04-18 23:15:28 -07:00
Yann Collet	2c5514c759	fixed ZSTDMT_initCStream_advanced() ... Must use the new ZSTD_compressBegin_usingCDict_advanced() to enforce correct frame parameters	2017-04-18 22:52:41 -07:00
Sean Purcell	98cf7fcb2a	Update README	2017-04-18 17:03:37 -07:00
Sean Purcell	0f7bd772e6	Update seekable API to simplify IO	2017-04-18 16:48:30 -07:00
Yann Collet	a4cab80183	added ZSTD_copyCCtx_internal() ... which respects provided fParams.	2017-04-18 14:54:54 -07:00
Sean Purcell	ca6fae7808	Add MT enabled targets for libzstd	2017-04-18 14:13:01 -07:00
Yann Collet	30fb499208	Changed ZSTD_resetCCtx_advanced() into ZSTD_resetCCtx_internal() ... for naming consistency : _advanced() can be invoked while _internal() are strictly static	2017-04-18 14:08:50 -07:00
Yann Collet	715b9aa113	created ZSTD_compressBegin_usingCDict_advanced()	2017-04-18 13:55:53 -07:00
Yann Collet	af4f45b682	Improved code comments for block functions	2017-04-18 03:17:44 -07:00
Yann Collet	4f818182b8	clarified frame parameters for ZSTD_compress*_usingCDict() ... created ZSTD_compressBegin_usingCDict_internal(), which gives direct control to frame Parameters. ZSTD_resetCStream_internal() now points into it.	2017-04-17 18:29:06 -07:00
Yann Collet	c47c68f6ca	proper evaluation of Huffman CTable size	2017-04-17 16:14:21 -07:00
Sean Purcell	5ee1135f30	s/chunk/frame/	2017-04-12 11:15:50 -07:00
Yann Collet	88009a8ba2	removed srcSize control from CStream ... since it's already done from lower bufferless API level	2017-04-12 00:51:24 -07:00
Yann Collet	20d5e03893	content size is controlled at bufferless level ... so it's active for all entry points Also : added relevant test (wrong content size) in fuzzer	2017-04-11 18:34:02 -07:00
Sean Purcell	d048fefef7	Move seekable format content to /contrib	2017-04-11 14:38:56 -07:00
Sean Purcell	45f3bc4801	Add format specification	2017-04-11 13:53:09 -07:00
Sean Purcell	a3b7c22604	Make seekable streams work w/ small buffers, misc fixes	2017-04-11 13:53:09 -07:00
Sean Purcell	c3ba15e48f	Seekable compression demo	2017-04-11 13:53:09 -07:00
Yann Collet	4ee6b15dac	force contentSizeFlag=0 when using ZSTD_initCStream_usingCDict() ... because by definition srcSize is not known when using this prototype. added relevant test Note : this use was already working, because at a later stage (both ZSTD_compressBegin_usingCDict() and ZSTD_copyCCtx()) pledgedSrcSize=0 is translated into "unknown", no matter the frame parameter. This is not correct, but of little importance, as the medium term plan is to no longer set fParams within CDict	2017-04-11 11:59:44 -07:00
Yann Collet	ab9162ebb4	simplified call graph ... by calling ZSTD_compressBegin_internal() instead of ZSTD_compressBegin_advanced()	2017-04-11 10:46:20 -07:00
Yann Collet	e88034fe26	simplified ZSTD_initCStream*() flow ... all variants converge towards ZSTD_initCStream_stage2()	2017-04-10 22:24:02 -07:00
Yann Collet	4b987ad8ce	Introduce ZSTD_initCStream_internal() ... This is now the regroup point for ZSTD_initCStream*() functions ZSTD_initCStream_advanced() now properly checks for parameters validity. Also : added <assert.h> usage inside zstd_compress.c Needs ZSTD_DEBUG=1 macro to be triggered. Will be triggered by default from `tests` directory	2017-04-10 17:50:44 -07:00
Yann Collet	0181fef545	ensure cctx internal buffer is correctly sized in case of memory error	2017-04-06 01:25:26 -07:00
Yann Collet	36c2a03757	updated comments for ZSTD_resetCStream()	2017-04-05 22:06:21 -07:00
Yann Collet	003a244324	DStream : ensure correct size of internal buffers in case of error	2017-04-05 15:28:56 -07:00
Yann Collet	02d37aa1c1	ensure correct size of internal buffers in case of error	2017-04-05 14:53:51 -07:00
Nick Terrell	405d2a1027	Explicitly convert scratchBuffer to unsigned*	2017-04-04 16:35:31 -07:00
Nick Terrell	16a739cab0	Switch call of FSE_count() to FSE_count_wksp()	2017-04-04 16:17:21 -07:00
Yann Collet	7cf78f1be7	Protects ZSTD_compressBegin_usingCDict() vs NULL cdict dereference ... Will issue an error (GENERIC) is cdict==NULL	2017-04-04 12:38:14 -07:00
Nick Terrell	26b046a7c4	Remove unnecessary dictID store	2017-04-03 21:46:28 -07:00
Nick Terrell	39a6cc5172	Make ZSTD_compress_usingCDict() respect contentSizeFlag	2017-04-03 21:09:55 -07:00
Nick Terrell	62ecad3819	Fix ZSTD_initCStream_usingCDict() to use dictionary	2017-04-03 21:05:59 -07:00
Yann Collet	30c7698970	optimize ZSTDMT_compress() memory usage ... does no longer allocate temporary buffers when there is enough room in dstBuffer to decompress directly there. (previous method would skip that for 1st chunk only). Also : fix ZSTD_compressBound() for small srcSize	2017-03-31 18:27:03 -07:00
Yann Collet	3f75d52527	Changed ZSTD_compressBound() ... required so that if Total = A+B compressBound(Total) <= compressBound(A) + compressBound(B) under condition of a minimum size for A and B Will help for ZSTDMT_compress() memory allocation	2017-03-31 17:11:38 -07:00
Yann Collet	7b70a1969e	Merge branch 'dev' into zstdmt	2017-03-31 16:22:33 -07:00
Yann Collet	53203e7c38	Merge pull request #640 from facebook/memAccess ... Changed memory strategy to __packed for gcc	2017-03-31 15:49:12 -07:00
Yann Collet	eea7858e2b	fixed minor warnings in debug code	2017-03-30 16:47:19 -07:00
Yann Collet	34cc487d05	overlap at full windowSize for max compression level ... as it provides max compression ratio	2017-03-30 16:23:22 -07:00
Yann Collet	458e955c23	improved ZSTDMT_compress() ... Use a bit more threads by default. Uses overlap segments to boost compression ratio (like the streaming variant)	2017-03-30 15:51:58 -07:00
Yann Collet	6476c51b86	Merge pull request #637 from facebook/zstdmt ... Zstdmt	2017-03-30 14:18:37 -07:00
Yann Collet	274f59919d	Changed memory strategy to __packed for gcc ... Method 1 __packed is always as good or better than memcpy(). But it's not portable, as it depends on compiler extension. For gcc, __pakced directive works fine. Furthermore, gcc has serious performance issues with memcpy() on ARM 32 bits. See #620	2017-03-30 12:52:14 -07:00
Nick Terrell	5152fb2cb2	Convert all tabs to spaces	2017-03-29 18:51:58 -07:00
Yann Collet	ca5a8bbe36	re-added patch ...	2017-03-29 17:15:27 -07:00
Yann Collet	2e2e78de47	removed unnecessary restriction on minmatchLength ... it's now transparently translated to nearest value when unsupported (7->6) (3->4)	2017-03-29 16:02:47 -07:00
Yann Collet	26769d88bc	Merge branch 'dev' of github.com:facebook/zstd into dev	2017-03-29 15:21:30 -07:00
Yann Collet	933ce4a1dd	fix : minmatch 7 conversion ... minmatch 7 now converted to minmatch 6 for strategies which do not support 7 Used to folded into "default", which applied minmatch 4	2017-03-29 14:35:38 -07:00
Sean Purcell	4708394bdd	Remove extra 'F' from skippable magic mask	2017-03-29 11:46:57 -07:00
Yann Collet	4cf0093571	restored bonus rule	2017-03-26 14:51:00 -07:00
Yann Collet	69017bf253	Merge branch 'dev' into LegacyDictBuilder	2017-03-26 14:39:13 -07:00
Yann Collet	582760818f	minor refactor ... add const changed if for easier to add new conditions	2017-03-26 03:04:56 -07:00
Yann Collet	858f72eeb8	fixed dictBuilder issue ... dictionary loading would fail during entropy analysis	2017-03-26 02:50:00 -07:00
Yann Collet	ecee9f2ef8	fixed conversion warnings	2017-03-26 00:59:14 -07:00
Yann Collet	0246d5c531	Merge pull request #630 from facebook/advancedCliCommands ... changed advanced commands --maxdict= and --dictID=	2017-03-26 00:13:35 -07:00
Yann Collet	4c41d37fcc	changed test for new syntax ... --dictID= and --maxdict=	2017-03-24 18:36:56 -07:00
Yann Collet	d41f707e88	minor improvement : remove duplicates with 1 char prefix difference	2017-03-24 17:56:45 -07:00
Yann Collet	b364caf455	Merge pull request #628 from facebook/dictBuilder_limits ... Ensure all limits derived from same constants	2017-03-24 17:54:42 -07:00
Yann Collet	2238870eb6	Merge pull request #625 from facebook/loadCDict ... limited CDict acceptation criteria to be the same as DDict	2017-03-24 16:06:20 -07:00
Yann Collet	96aa3019b2	changed advanced commands --maxdict= and --dictID= ... now works with the `=` variant, which is the recommended one. Old variant `--dictID #` still works, for compatibility with existing scripts. Long term objective is to remove the old variant..	2017-03-24 16:04:29 -07:00
Yann Collet	9da3b215ec	Ensure all limits derived from same constants ... Now uses ZDICT_DICTSIZE_MIN and ZDICT_CONTENTSIZE_MIN from zdict.h. Also : reduced values to 256 and 128 respectively	2017-03-24 15:02:09 -07:00
Yann Collet	ebe9963cf6	Merge pull request #626 from facebook/stricterDictBuilder ... dictBuilder fails to create dictionary on certain input	2017-03-24 14:27:28 -07:00
Yann Collet	16a0b10781	fixed ZSTD_loadZstdDictionary() ... forgot to add the dictionary content (tests were not failing, just compressing less). Also : added size protections when adding dict content since hc/bt table filling would fail if size < 8	2017-03-24 12:46:46 -07:00
Yann Collet	23776ce290	fixed ERROR_GENERIC on dstSize_tooSmall ... required by users which depends on this error code to size dest buffer	2017-03-23 17:59:50 -07:00
Yann Collet	f332ece468	dictBuilder fails to create dictionary on certain input ... Properly expressed with an error code (see zstd_errors.h) and a cli return code != 0	2017-03-23 16:24:02 -07:00
Yann Collet	bea78e8fc2	limited CDict acceptation criteria to be the same as DDict	2017-03-23 15:46:06 -07:00
Sean Purcell	042ba122ae	Change g_displayLevel to int and fix DISPLAYUPDATE flush	2017-03-23 11:21:59 -07:00
Nick Terrell	eaf69b07f0	Zero pointers after freeing	2017-03-21 13:20:59 -07:00
Yann Collet	f3dfcdccd1	bump version number	2017-03-21 12:18:28 -07:00
Przemyslaw Skibinski	8086d623ca	updated build of Windows packages	2017-03-18 11:19:09 +01:00
Yann Collet	7e35b352c6	Merge pull request #602 from iburinoc/doc ... Add functions missing from manual, and fix parameter alignment	2017-03-14 14:08:41 -07:00
Sean Purcell	dec2b96536	Add functions missing from manual, and fix parameter alignment	2017-03-14 11:24:09 -07:00
Sean Purcell	9830aeeea6	Fix legacy support=0 case and accidental double include of version headers	2017-03-13 17:19:37 -07:00
Sean Purcell	120df494e9	Update builds to not support legacy v01-v03	2017-03-13 14:44:08 -07:00
Sean Purcell	334cb34edb	ZSTD_LEGACY_SUPPORT defines lowest supported version	2017-03-13 14:32:30 -07:00
Sean Purcell	784082f49c	Change gotoDict type to uPtrDiff	2017-03-10 10:34:45 -08:00
Sean Purcell	8fe5c6862c	Fix undefined behaviour in decompressor	2017-03-10 10:17:42 -08:00
Nick Terrell	f35ef5c8e9	Whitespace only: tabs to spaces	2017-03-09 12:51:33 -08:00
Nick Terrell	eeb31eed39	s/ZSTD_btopt2/ZSTD_btultra/g	2017-03-09 11:44:25 -08:00
Nick Terrell	e65aab8e0f	Remove 'mem.h' dependency from ZSTD_WINDOWLOG_MAX	2017-03-08 15:40:13 -08:00
Yann Collet	a41a4ed39a	Merge pull request #594 from terrelln/bugs ... Small fixes	2017-03-08 14:56:07 -08:00
Nick Terrell	81512e9ebe	Avoid '#define inline /* ... */' ... Take definition of `FORCE_INLINE` from `zstd_internal.h`.	2017-03-08 14:00:21 -08:00
Nick Terrell	e06c303475	Fix ZSTD_sizeof_CStream()	2017-03-08 13:45:10 -08:00
Sean Purcell	881abe44f1	Reduce point at which we reduce offsets to protect against UB	2017-03-07 16:58:08 -08:00
Sean Purcell	3437bf2feb	Add build targets to the Makefile, and update CircleCI tests	2017-03-06 15:05:02 -08:00
Yann Collet	8b1d004031	added -Wformat-security flag, as recommended by @pixelb	2017-03-05 21:17:32 -08:00
Yann Collet	1f2c95c5f3	minor code refactor in HUF module	2017-03-05 21:07:20 -08:00
Yann Collet	5d801278dc	Merge pull request #586 from terrelln/repeat-heuristic ... Always check Huffman tables for ZSTD_lazy+	2017-03-03 19:38:56 -08:00
Nick Terrell	54c4babd8f	Always check Huffman tables for ZSTD_lazy+ ... The compressor always reuses the existing Huffman table if the literals size is at most 1 KiB. If the compression strategy is `ZSTD_lazy` or stronger always check to see if reusing the previous table or creating a new table is better. This doesn't yet weigh in decompression speed. I don't want to add any heuristics there until I have real data to work with to ensure that the heuristic works for at least one use case, preferably more.	2017-03-03 16:49:38 -08:00
Yann Collet	1af570bd05	Merge pull request #585 from terrelln/cover-leak ... Fix COVER_optimizeTrainFromBuffer() resource leaks	2017-03-02 20:46:35 -08:00
Yann Collet	f44b55c18d	Merge pull request #584 from terrelln/huff-repeat ... Allow compressor to repeat Huffman tables	2017-03-02 17:20:11 -08:00
Yann Collet	fe5d27062e	disable prefetch-decode for 32-bits target ... This decoder variant is detrimental to x86 architecture likely due to register pressure. Note that the variant is disabled for all 32-bits targets. It's unclear if it would help for different architectures, such as ARM, MIPS or PowerPC.	2017-03-02 17:09:21 -08:00
Nick Terrell	d051cd5b43	Use workspace for count and CTable	2017-03-02 16:38:07 -08:00
Nick Terrell	976e325b2e	Fix COVER_optimizeTrainFromBuffer() resource leaks ... Thanks to @nemequ for reporting the resource leaks.	2017-03-02 15:54:39 -08:00
Sean Purcell	553f67e0c1	Remove 'generic' inline strategy ... Seems to avoid performance loss for compression. Same strategy tested on decompression side, did not appear to improve speed.	2017-03-02 15:18:13 -08:00
Sean Purcell	3d95925a59	Merge remote-tracking branch 'origin/dev' into m32	2017-03-02 15:17:56 -08:00
Nick Terrell	a419777eb1	Allow compressor to repeat Huffman tables ... * Compressor saves most recently used Huffman table and reuses it if it produces better results. * I attempted to preserve CPU usage profile. I intentionally left all of the existing heuristics in place. There is only a speed difference on the second block and later. When compressing large enough blocks (say >= 4 KiB) there is no significant difference in compression speed. Dictionary compression of one block is the same speed for blocks with literals <= 1 KiB, and after that the difference is not very significant. * In the synthetic data, with blocks 10 KB or smaller, most blocks can't use repeated tables because the previous block did not contain a symbol that the current block contains. Once blocks are about 12 KB or more, most previous blocks have valid Huffman tables for the current block, and the compression ratio and decompression speed jumped. * In silesia blocks as small as 4KB can frequently reuse the previous Huffman table (85%), but it isn't as profitable, and the previous Huffman table only gets used about 3% of the time. * Microbenchmarks show that `HUF_validateCTable()` takes ~55 ns and `HUF_estimateCompressedSize()` takes ~35 ns. They are decently well optimized, the first versions took 90 ns and 120 ns respectively. `HUF_validateCTable()` could be twice as fast, if we cast the `HUF_CElt*` to a `U32*` and compare to 0. However, `U32` has an alignment of 4 instead of 2, so I think that might be undefined behavior. * I've ran `zstreamtest` compiled normally, with UASAN and with MSAN for 4 hours each. The worst case for the speed difference is a bunch of small blocks in the same frame. I modified `bench.c` to compress the input in a single frame but with blocks of the given block size, set by `-B`. Benchmarks on level 1: | Program | Block size | Corpus | Ratio | Compression MB/s | Decompression MB/s | |-----------|------------|-----------|-------|------------------|--------------------| | zstd.base | 256 | synthetic | 2.364 | 110.0 | 297.0 | | zstd | 256 | synthetic | 2.367 | 108.9 | 297.0 | | zstd.base | 256 | silesia | 2.204 | 93.8 | 415.7 | | zstd | 256 | silesia | 2.204 | 93.4 | 415.7 | | zstd.base | 512 | synthetic | 2.594 | 144.2 | 420.0 | | zstd | 512 | synthetic | 2.599 | 141.5 | 425.7 | | zstd.base | 512 | silesia | 2.358 | 118.4 | 432.6 | | zstd | 512 | silesia | 2.358 | 119.8 | 432.6 | | zstd.base | 1024 | synthetic | 2.790 | 192.3 | 594.1 | | zstd | 1024 | synthetic | 2.794 | 192.3 | 600.0 | | zstd.base | 1024 | silesia | 2.524 | 148.2 | 464.2 | | zstd | 1024 | silesia | 2.525 | 148.2 | 467.6 | | zstd.base | 4096 | synthetic | 3.023 | 300.0 | 1000.0 | | zstd | 4096 | synthetic | 3.024 | 300.0 | 1010.1 | | zstd.base | 4096 | silesia | 2.779 | 223.1 | 623.5 | | zstd | 4096 | silesia | 2.779 | 223.1 | 636.0 | | zstd.base | 16384 | synthetic | 3.131 | 350.0 | 1150.1 | | zstd | 16384 | synthetic | 3.152 | 350.0 | 1630.3 | | zstd.base | 16384 | silesia | 2.871 | 296.5 | 883.3 | | zstd | 16384 | silesia | 2.872 | 294.4 | 898.3 |	2017-03-02 13:27:52 -08:00
Yann Collet	fdb0fd34b3	Merge pull request #583 from terrelln/set-dictid ... Set dictID to 0 for content only dictionaries	2017-03-02 13:15:31 -08:00
Nick Terrell	3475b9b431	Set dictID to 0 for content only dictionaries	2017-03-02 12:33:02 -08:00
Sean Purcell	d44703d145	Offsets >= 32MB in 32-bits mode	2017-03-01 16:27:56 -08:00
Yann Collet	76f0494089	xxhash can be included twice in any order ... Previously, followed by : would fail to include the static definitions, because the second include was simply skipped by guard macro. Now it works as intended : the missing static part is included during the second include.	2017-03-01 13:29:29 -08:00
Yann Collet	4bcc69b761	solves warnings when compiling with global XXH_STATIC_LINKING_ONLY ... XXH_STATIC_LINKING_ONLY protection macro is intended to be triggered just before the include. The main idea is to keep this setting local : user module shall explicitly understand and accept the static linking restriction which becomes transparent when triggering the macro at project level. Global definition also triggers redefinition warnings for user modules which do locally define the macro. This new version compiles lib and cli without warning when the macro is set globally. That's not a scenario to be recommended, since it trades a local effect for a global one, but it was easy enough to provide from zstd side.	2017-03-01 11:33:25 -08:00
Yann Collet	31432cc57d	Merge pull request #579 from iburinoc/multiframe ... Check to ensure ddict isn't null before dereference	2017-03-01 11:02:04 -08:00
Sean Purcell	a81d4fee58	Check to ensure ddict isn't null before dereference	2017-02-28 15:28:29 -08:00
Yann Collet	22d79762ef	fixed multi frames	2017-02-28 02:12:42 -08:00
Yann Collet	a33ae64204	fixed decoding skippable frames	2017-02-28 01:15:28 -08:00
Yann Collet	d1760113ec	Improved speed of ZSTD_decompressStream() ... When ZSTD_decompressStream() detects that there is enough space in dst to complete decompression in a single pass, delegates to ZSTD_decompress(), for an extra ~5% speed boost	2017-02-28 00:14:28 -08:00
Yann Collet	a81c2e7e44	Merge pull request #573 from facebook/ddict ... Improved DDict memory usage	2017-02-27 20:54:42 -08:00
Yann Collet	dccd6b6f65	cli : fix : --rm is silent when input is stdin ... previously, app would produce an error message, and stop.	2017-02-27 15:57:50 -08:00
Yann Collet	0b9b894b2d	reduced ZSTD_DDict memory usage ... saved 128 KB	2017-02-27 00:27:30 -08:00
Yann Collet	bd7fa21deb	added ZSTD_refDDict() ... Now DDict does no longer depends on DCtx duplication	2017-02-26 14:43:07 -08:00
Yann Collet	d73eebc00f	loadEntropy works on new ZSTD_entropy_t type	2017-02-26 10:16:42 -08:00
Yann Collet	8629f0e41f	created entropy structure type	2017-02-25 18:33:31 -08:00
Yann Collet	8dff956dbf	Added DDict unit test in fuzzer ... also : slightly modified loadEntropy : know src must points at start of dictionary	2017-02-25 10:11:15 -08:00
Yann Collet	14312d833e	zstdmt : fix : loading prefix from previous segments ... There used to be a (very small) chance that loading prefix from previous segment would be confused with a real zstd dictionary. For that to happen, the prefix needs to start with the same value as dictionary magic. That's 1 chance in 4 billions if all values have equal probability. But in fact, since some values are more common (0x00000000 for example) others are less common, and dictionary magic was selected to be one of them, so probabilities are likely even lower. Anyway, this risk is no down to zero by adding a new CCtx parameter : ZSTD_p_forceRawDict Current parameter policy : the parameter "stick" to its CCtx, so any dictionary loading after ZSTD_p_forceRawDict is set will be loaded in "raw" ("content only") mode, even if CCtx is re-used multiple times with multiple different dictionary. It's up to the user to reset this value differently if it needs so.	2017-02-23 23:42:12 -08:00
Yann Collet	831b4890ce	minor tests/Makefile refactoring ... and update of zstd_manual,html	2017-02-23 23:09:10 -08:00
Yann Collet	cce8d8ba2b	Merge pull request #560 from iburinoc/findcompressedsize ... Change name to to findFrameCompressedSize and add skippable support	2017-02-23 13:39:23 -08:00
Sean Purcell	83038d236a	Fix bug in FSE distribution normalization	2017-02-22 13:52:48 -08:00
Sean Purcell	64417cd2ff	Describe ambiguity around skippable frames	2017-02-22 13:29:01 -08:00
Sean Purcell	9757cc811b	Update comment	2017-02-22 12:28:21 -08:00
Sean Purcell	9050e1925e	Change name to to findFrameCompressedSize and add skippable support	2017-02-22 12:12:34 -08:00
Przemyslaw Skibinski	d8114e5802	zstd_compress.c: fix memory leaks	2017-02-21 18:59:56 +01:00
Anders Oleson	517577bf53	spelling fixes in comments ... i.e. occurred labeled Huffman	2017-02-20 12:08:59 -08:00
Sean Purcell	6b010dec80	execSequence copies up to 2*WILDCOPY_OVERLENGTH extra	2017-02-16 12:05:40 -08:00
Sean Purcell	887eaa9e21	Fix wildcopy overwriting data still in window	2017-02-15 16:43:45 -08:00
Yann Collet	2252d29a5a	Merge branch 'dev' of github.com:facebook/zstd into dev	2017-02-15 12:00:50 -08:00
Yann Collet	4596037042	updated fse version ... feature minor refactoring (removing FSE_abs()) also : fix a few minor issues recently introduced in examples	2017-02-15 12:00:03 -08:00
Yann Collet	44f82d781f	Merge pull request #545 from terrelln/force-window ... [zstdmt] Fix MSAN failure with ZSTD_p_forceWindow	2017-02-15 10:20:15 -08:00
Yann Collet	f0b9a8dddb	Merge pull request #547 from inikep/dev11 ... Avoid fseek()'s 2GiB barrier with MacOS and *BSD	2017-02-14 12:29:00 -08:00
Yann Collet	9696bfc2ad	Merge pull request #544 from ds77/avoid-empty ... Portable way to avoid empty unit warning in threading.c	2017-02-14 00:54:55 -08:00
Przemyslaw Skibinski	b876b96ce1	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2017-02-14 09:26:03 +01:00
Nick Terrell	ecf90ca24b	[zstdmt] Fix MSAN failure with ZSTD_p_forceWindow ... Reproduction steps: ``` make zstreamtest CC=clang CFLAGS="-O3 -g -fsanitize=memory -fsanitize-memory-track-origins" ./zstreamtest -vv -t4178 -i4178 -s4531 ``` How to get to the error in gdb (may be a more efficient way): * 2 breaks at zstd_compress.c:2418 -- in ZSTD_compressContinue_internal() * 2 breaks at zstd_compress.c:2276 -- in ZSTD_compressBlock_internal() * 1 break at zstd_compress.c:1547 Why the error occurred: When `zc->forceWindow == 1`, after calling `ZSTD_loadDictionaryContent()` we have `zc->loadedDictEnd == zc->nextToUpdate == 0`. But, we've really loaded up to `iend` into the dictionary. Then in `ZSTD_compressBlock_internal()` we see that `current > zc->nextToUpdate + 384`, so we load the last 192 bytes a second time. In this case the bytes we are loading are a block of all 0s, starting in the previous block. So when we are loading the last 192 bytes, we find a `match` in the future, 183 bytes beyond `ip`. Since the block is all 0s, the match extends to the end of the block. But in `ZSTD_count()` we only check that `pIn < pInLoopLimit`, but since `pMatch > pIn`, `pMatch` eventually points past the end of the buffer, causing the MSAN failure. The fix: The line changed sets sets `zc->nextToUpdate` to the end of the dictionary. This is the behavior that existed before `ZSTD_p_forceWindow` was introduced. This fixes the exposing test case. Since the code doesn't fail without `zc->forceWindow`, it makes sense that this works. I've run the command `./zstreamtest -T2mn` 64 times without failures. CI should also verify nothing obvious broke.	2017-02-13 19:11:22 -08:00
Yann Collet	58af614ef2	push version and NEWS to v1.1.4	2017-02-13 18:32:44 -08:00
ds77	08e6a88a97	avoid empty translation unit warning without #pragma	2017-02-14 00:46:47 +01:00
Przemyslaw Skibinski	09c8e5390d	__builtin_bswap requires gcc 4.3+	2017-02-13 12:45:53 +01:00
Sean Purcell	d7bfcac18a	Expose frameSrcSize to experimental API	2017-02-10 11:55:44 -08:00
Sean Purcell	5069b6c2c3	Merge branch 'dev' into multiframe	2017-02-10 10:08:55 -08:00
Yann Collet	bbba42acd1	Merge pull request #537 from terrelln/small-bugs ... Fix small bugs	2017-02-10 04:35:43 -08:00
Yann Collet	a28c34cb7a	Merge pull request #538 from iburinoc/errorstring ... Fix ZSTD_getErrorString and add tests	2017-02-10 03:59:56 -08:00
Sean Purcell	269b2cd3d8	Documentation updates	2017-02-09 13:25:30 -08:00
Sean Purcell	2db7249265	Make pledgedSrcSize meaning clear for other functions ... - Added tests - Moved new size functions to static link only	2017-02-09 11:49:58 -08:00
Nick Terrell	545987996a	Fix deprecation warnings for clang with C++14	2017-02-08 17:38:17 -08:00
Sean Purcell	e0b3265e87	Fix ZSTD_getErrorString and add tests	2017-02-08 17:28:49 -08:00
Sean Purcell	0f5c95af44	Disambiguate pledgedSrcSize == 0 ... - Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size - Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize	2017-02-08 15:12:46 -08:00
Sean Purcell	ba2ad9f25c	ZSTD_decompress now handles multiple frames	2017-02-08 14:50:10 -08:00
Sean Purcell	4e709712e1	Decompressed size functions now handle multiframes and distinguish cases ... - Add ZSTD_findDecompressedSize - Traverses multiple frames to find total output size - Add ZSTD_getFrameContentSize - Gets the decompressed size of a single frame by reading header - Deprecate ZSTD_getDecompressedSize	2017-02-08 14:50:10 -08:00
Przemyslaw Skibinski	cdf5a7bd9f	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2017-02-08 13:49:35 +01:00
Nick Terrell	71c5263c00	Attribute cover dictionary code	2017-02-07 11:35:07 -08:00
Przemyslaw Skibinski	7060aee8c2	platform.h added to build_package.bat	2017-02-06 19:43:13 +01:00
Yann Collet	b54e235bf3	fixed Mac OS-X specific directory in $(RM) list ... these directories are now removed with -r command	2017-02-05 10:22:58 -08:00
Yann Collet	c2a4632789	release builds use less debug symbols and warnings ... release build are triggered through either `make`, or their specific target `make zstd-release` and `make lib-release`.	2017-02-02 20:54:41 -08:00
Yann Collet	48bed91606	Merge pull request #527 from facebook/zstdmt ... zstdmt refinements	2017-01-31 16:36:46 -08:00
Yann Collet	b2e1b3d670	fixed overlapLog==0 => no overlap	2017-01-30 14:54:46 -08:00
Yann Collet	3672d06d06	zstdmt : section size is set to be a minimum of overlapSize ... the minimum size condition size is applied transparently (no warning, no error) like previous minimum section size condition (1 KB) which still applies.	2017-01-30 13:35:45 -08:00
Yann Collet	88df1aed61	changed advanced parameter overlapLog ... Follows a positive logic (increasing value => increasing overlap) which is easier to use	2017-01-30 11:00:00 -08:00
Yann Collet	b5fd15ccb2	fixed : legacy decoders v04 and v05	2017-01-30 10:45:58 -08:00
Yann Collet	cc3d1bc262	Merge pull request #525 from terrelln/covermt ... Multithreaded COVER dictionary training	2017-01-30 10:15:33 -08:00
Nick Terrell	43474313f8	Fix documentation about memory usage	2017-01-27 18:43:05 -08:00
Nick Terrell	b42dd27ef5	Add include guards and extern C	2017-01-27 16:00:19 -08:00
Yann Collet	f6d4a786fc	reduced zstdmt latency when using small custom section sizes with high compression levels ... Previous version was requiring a fairly large initial amount of input data before starting to create compression jobs. This new version starts the process much sooner.	2017-01-27 15:55:30 -08:00
Nick Terrell	c43c27127f	Merge branch 'dev' into buck ... * dev: updated NEWS fixed MSAN warnings in legacy decoders Fix cmake build updated NEWS Edits as per comments, and change wildcard 'X' to '?' Fix Visual Studios project Fix pool.c threading.h import Fix zstdmt_compress.h include Fixed commented issues Updated format specification to be easier to understand improved #232 fix Fixed https://github.com/facebook/zstd/issues/232 .travis.yml: different tests for "master" branch .travis.yml: optimized order of short tests .travis.yml: test jobs 12-15 JOB_NUMBER -eq 9 improved ZSTD_compressBlock_opt_extDict_generic	2017-01-27 12:05:48 -08:00
Nick Terrell	2fe9126591	Add multithread support to COVER	2017-01-27 11:56:02 -08:00
Yann Collet	609c123a01	Merge pull request #522 from terrelln/benchmt ... Fix some includes	2017-01-27 11:40:25 -08:00
Yann Collet	cafdd31a38	fixed MSAN warnings in legacy decoders ... In some extraordinary circumstances, *Length field can be generated from reading a partially uninitialized memory segment. Data is correctly identified as corrupted later on, but the read taints some later pointer arithmetic operation.	2017-01-27 10:44:03 -08:00
Nick Terrell	9c018cc140	Add BUCK files for Nuclide support	2017-01-27 10:43:12 -08:00
Przemyslaw Skibinski	29157320fb	improved ZSTD_compressBlock_opt_extDict_generic	2017-01-27 10:43:02 -08:00
Nick Terrell	e628eaf87a	Fix pool.c threading.h import	2017-01-26 15:29:10 -08:00
Yann Collet	717c65d690	Merge pull request #519 from inikep/dev11 ... Dev11	2017-01-26 14:23:44 -08:00
Yann Collet	ef33d00532	fixed : ZSTD_setCCtxParameter() properly exposed in DLL	2017-01-26 12:24:21 -08:00
Yann Collet	4a62f79ec9	fixed clang documentation warning	2017-01-26 09:16:56 -08:00
Yann Collet	8dafb1acf5	CLI : automatically set overlap size to max (windowSize) for max compression level	2017-01-25 17:01:13 -08:00
Yann Collet	06e7697f96	added test of new parameter ZSTD_p_forceWindow	2017-01-25 16:39:03 -08:00
Yann Collet	bb0027405a	fixed zstdmt corruption issue when enabling overlapped sections ... see Asana board for detailed explanation on why and how to fix it	2017-01-25 16:25:38 -08:00
Yann Collet	943cff9c37	fixed zstdmt cli freeze issue with large nb of threads ... fileio.c was continually pushing more content without giving a chance to flush compressed one. It would block the job queue when input data was accumulated too fast (requiring to define many threads). Fixed : fileio flushes whatever it can after each input attempt.	2017-01-25 12:35:19 -08:00
Yann Collet	dc8dae596a	overlapped section, for improved compression ... Sections 2+ read a bit of data from previous section in order to improve compression ratio. This also costs some CPU, to reference read data. Read data is currently fixed to window>>3 size	2017-01-24 22:32:12 -08:00
Yann Collet	f14a669054	refactor job creation ... code shared accross ZSTDMT_{compress,flush,end}Stream(), for easier maintenance	2017-01-24 17:41:49 -08:00
Yann Collet	512cbe8c10	zstdmt cli and API allow selection of section sizes ... By default, section sizes are 4x window size. This new setting allow manual selection of section sizes. The larger they are, the (slightly) better the compression ratio, but also the higher the memory allocation cost, and eventually the lesser the nb of possible threads, since each section is compressed by a single thread. It also introduces a prototype to set generic parameters, ZSTDMT_setMTCtxParameter() The idea is that it's possible to add enums to extend the list of parameters that can be set this way. This is more long-term oriented than a fixed-size struct. Consider it as a test.	2017-01-24 17:08:53 -08:00
Yann Collet	3488a4a473	ZSTDMT now supports frame checksum	2017-01-24 11:48:40 -08:00
Przemyslaw Skibinski	96f152f708	improved ZSTD_compressBlock_opt_extDict_generic	2017-01-24 13:18:50 +01:00
Yann Collet	94364bf87a	refactor ZSTDMT streaming flush code ... now shared by both ZSTDMT_compressStream() and ZSTDMT_flushStream()	2017-01-23 11:50:44 -08:00
Yann Collet	1cbf251e43	ZSTDMT streaming : fall back to (regular) single thread mode ... when nbThreads==1	2017-01-23 01:43:58 -08:00
Yann Collet	84581ff8d7	ZSTDMT_compressCCtx : fallback to single-thread mode when nbChunks==1	2017-01-23 01:20:27 -08:00
Yann Collet	1a2547f654	ZSTDMT_compressStream() becomes blocking when required to ensure forward progresses ... In some (rare) cases, job list could be blocked by a first job still being processed, while all following ones are completed, waiting to be flushed. In such case, the current job-table implementation is unable to accept new job. As a consequence, a call to ZSTDMT_compressStream() can be useless (nothing read, nothing flushed), with the risk to trigger a busy-wait on the caller side (needlessly loop over ZSTDMT_compressStream() ). In such a case, ZSTDMT_compressStream() will block until the first job is completed and ready to flush. It ensures some forward progress by guaranteeing it will flush at least a part of the completed job. Energy-wasting busy-wait is avoided.	2017-01-22 23:49:52 -08:00
Yann Collet	c593348722	ZSTDMT_initCStream_usingDict() can outlive dict ... Like ZSTD_initCStream_usingDict(), ZSTDMT_initCStream_usingDict() now keep a copy of dict internally. This way, dict can be released : it does not longer have to outlive all future compression sessions.	2017-01-22 16:44:15 -08:00
Yann Collet	9d6f7637ec	protected (mutex) read to jobCompleted, as suggested by @terrelln	2017-01-21 22:14:08 -08:00
Yann Collet	0cf74fa957	optimized pool allocation by 1 slot	2017-01-21 22:06:49 -08:00
Yann Collet	6ed29a8f44	minor : tab to spaces	2017-01-21 21:56:36 -08:00
Yann Collet	317604e0ad	fixed : compilation of zstreamtest in dll mode	2017-01-20 17:18:41 -08:00
Yann Collet	d7e3cb58c5	Resolved merge conflict dev+zstdmt	2017-01-20 16:44:50 -08:00
cyan4973	2e3b659ae1	fixed minor warnings (Visual, conversion, doxygen)	2017-01-20 14:43:09 -08:00
cyan4973	5fba09fa41	updated util's time for Windows compatibility ... Correctly measures time on Posix systems when running with Multi-threading Todo : check Windows measurement under multi-threading	2017-01-20 12:57:31 -08:00
Yann Collet	b459aad5b4	renamed savedRep into repToConfirm	2017-01-19 17:33:37 -08:00
Yann Collet	500014af49	zstd cli can now compress using multi-threading ... added : command -T# added : ZSTD_resetCStream() (zstdmt_compress) added : FIO_setNbThreads() (fileio)	2017-01-19 17:04:28 -08:00
Yann Collet	19d670ba9d	Added ZSTDMT_initCStream_advanced() variant ... Correctly compress with custom params and dictionary Added relevant fuzzer test in zstreamtest Also : new macro ZSTDMT_SECTION_LOGSIZE_MIN, which sets a minimum size for a full job (note : a flush() command can still generate a partial job anytime)	2017-01-19 15:32:07 -08:00
Yann Collet	0f984d94c4	changed MT enabling macro to ZSTD_MULTITHREAD	2017-01-19 14:05:07 -08:00
Yann Collet	736788f8e8	added streaming fuzzer tests for MT API ... Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize which is possible when many flushes are issued while there is not enough dst buffer to flush completed ones.	2017-01-19 12:15:29 -08:00
Yann Collet	32dfae6f98	fixed Multi-threaded compression ... MT compression generates a single frame. Multi-threading operates by breaking the frames into independent sections. But from a decoder perspective, there is no difference : it's just a suite of blocks. Problem is, decoder preserves repCodes from previous block to start decoding next block. This is also valid between sections, since they are no different than changing block. Previous version would incorrectly initialize repcodes to their default value at the beginning of each section. When using them, there was a mismatch between encoder (default values) and decoder (values from previous block). This change ensures that repcodes won't be used at the beginning of a new section. It works by setting them to 0. This only works with regular (single segment) variants : extDict variants will fail ! Fortunately, sections beyond the 1st one belong to this category. To be checked : btopt strategy. This change was only validated from fast to btlazy2 strategies.	2017-01-19 10:32:55 -08:00
Yann Collet	37226c1e9f	Simplified compressChunk job ... minor refactoring : compression done in a single call on first chunk Avoid a mutable hSize variable and eventual recombination to cSize at the end	2017-01-19 10:18:17 -08:00
Yann Collet	dab5ea93f2	Merge pull request #515 from iburinoc/emptydict ... Don't create dict in streaming apis if dictSize == 0	2017-01-19 09:02:42 -08:00
Yann Collet	6073b3e6b8	ZSTDMT_endStream : nullify input buffer after flush ... There will be no more input after ZSTDMT_endStream invocation : only flush/end is allowed (to fully collect compressed result).	2017-01-18 15:32:38 -08:00
Yann Collet	3a01c46b26	ZSTDMT_initCStream() supports restart from invalid state ... ZSTDMT_initCStream() will correcly scrub for resources when it detects that previous compression was not properly finished.	2017-01-18 15:18:17 -08:00
Yann Collet	4885f591b3	trap compression errors, collect back resources from workers	2017-01-18 14:11:37 -08:00
Sean Purcell	0b5370ae38	Prefix notes with /**<	2017-01-18 13:45:02 -08:00
Yann Collet	563ef8acf4	CCtxPool starts empty, as suggested by @terrelln ... Also : make zstdmt now a target from root	2017-01-18 12:12:10 -08:00
Yann Collet	a6db7a7b9b	fixed cmaketest ... (buffer_t){NULL,0} is not considered a constant. {NULL,0} is.	2017-01-18 11:57:34 -08:00
Yann Collet	0d6b8f65a9	ZSTDMT_free() scrubs potentially unfinished jobs to release their resources ... In some complex scenarios (free() without finishing compression), it is possible that some resources are still into jobs and not collected back into pools. In which case, previous version of free() would miss them. This would be equivalent to a leak. New version ensures that it even foes after such resource. It requires job consumers to properly mark resources as released, by replacing entries by NULL after releasing back to the pool. Obviously, it's not recommended to free() zstdmt context mid-term, still that's now a supported scenario. The same methodology is also used to ensure proper resource collection after an error is detected. Still to do : - detect compression errors (not just allocation ones) - properly manage resource when init() is called without finishing previous compression.	2017-01-17 17:46:33 -08:00
Yann Collet	d0a1d45582	ZSTDMT_{flush,end}Stream() now block on next job completion when nothing to flush ... The main issue was to avoid a caller to continually loop on {flush,end}Stream() when there was nothing ready to be flushed but still some compression work ongoing in a worker thread. The continuous loop would have resulted in wasted energy. The new version makes call to {flush,end}Stream blocking when there is nothing ready to be flushed. Of course, if all worker threads have exhausted job, it will return zero (all flush completed). Note : There are still some remaining issues to report error codes and properly collect back resources into pools when an error is triggered.	2017-01-17 16:15:18 -08:00
Yann Collet	a73c412932	completed ZSTDMT streaming compression ... Provides the baseline compression API : size_t ZSTDMT_initCStream(ZSTDMT_CCtx* zcs, int compressionLevel); size_t ZSTDMT_compressStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input); size_t ZSTDMT_flushStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output); size_t ZSTDMT_endStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output); Not tested yet	2017-01-17 15:31:16 -08:00
Sean Purcell	57d423c5df	Don't create dict in streaming apis if dictSize == 0	2017-01-17 14:31:35 -08:00
Przemyslaw Skibinski	8a0bc30a2d	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2017-01-17 13:02:29 +01:00
Przemyslaw Skibinski	d72f4b6b7a	added "Makefile is validated"	2017-01-17 12:40:06 +01:00
Gregory Szorc	7d6f478d15	Set dictionary ID in ZSTD_initCStream_usingCDict() ... When porting python-zstandard to use ZSTD_initCStream_usingCDict() so compression dictionaries could be reused, an automated test failed due to compressed content changing. I tracked this down to ZSTD_initCStream_usingCDict() not setting the dictID field of the ZSTD_CCtx attached to the ZSTD_CStream instance. I'm not 100% convinced this is the correct or full solution, as I'm still seeing one automated test failing with this change.	2017-01-14 17:44:54 -08:00
Yann Collet	5b726dbe4d	fix gcc-arm warning "suggest braces around empty body"	2017-01-12 17:46:46 +01:00
Yann Collet	ad9f6bd123	zstdmt : fix : resources properly collected even when early fail ... In previous version, main function would return early when detecting a job error. Late threads resources were therefore not collected back into pools. New version just register the error, but continue the collecting process. All buffers and context should be released back to pool before leaving main function.	2017-01-12 03:06:35 +01:00
Sean Purcell	834ab50fa3	Fixed decompress_usingDict not propagating corrupted dictionary error	2017-01-11 17:31:34 -08:00
Yann Collet	b05c4828ea	zstdmt : correctly check for cctx and buffer allocation ... Result from getBuffer and getCCtx could be NULL when allocation fails. Now correctly checks : job creation stop and last job reports an allocation error. releaseBuffer and releaseCCtx are now also compatible with NULL input. Identified a new potential issue : when early job fails, later jobs are not collected for resource retrieval.	2017-01-12 02:01:28 +01:00
Yann Collet	107bcbbbc2	zstdmt : changed internal naming from frame to chunk ... Since the result of mt compression is a single frame, changed naming, which implied the concatenation of multiple frames. minor : ensures that content size is written in header	2017-01-12 01:25:46 +01:00
Yann Collet	5eb749e734	ZSTDMT_compress() creates a single frame ... The new strategy involves cutting frame at block level. The result is a single frame, preserving ZSTD_getDecompressedSize() As a consequence, bench can now make a full round-trip, since the result is compatible with ZSTD_decompress(). This strategy will not make it possible to decode the frame with multiple threads since the exact cut between independent blocks is not known. MT decoding needs further discussions.	2017-01-11 18:21:25 +01:00
Yann Collet	04cbc36499	minor refactor (release CCtx 1st) and comment clarification	2017-01-11 16:08:08 +01:00
Yann Collet	085179bb78	fixed ZSTDMT_createCCtx() : checked inner objects are properly created	2017-01-11 15:58:05 +01:00
Yann Collet	8ce1cc2bec	improved ZSTD_createCCtxPool() cancellation ... use ZSTD_freeCCtxPool() to release the partially created pool. avoids to duplicate logic. Also : identified a new difficult corner case : when freeing the Pool, all CCtx should be previously released back to the pool. Otherwise, it means some CCtx are still in use. There is currently no clear policy on what to do in such a case. Note : it's supposed to never happen. Since pool creation/usage is static, it has no external user, which limits risks.	2017-01-11 15:44:26 +01:00
Yann Collet	47557ba2b2	fixed ZSTDMT_createCCtxPool() when inner CCtx creation fails	2017-01-11 15:35:56 +01:00
Nick Terrell	8d984699db	Document memory requirements for COVER algorithm	2017-01-09 18:20:10 -08:00
Nick Terrell	555e281637	Handle large input size in 32-bit mode correctly	2017-01-09 18:20:06 -08:00
Nick Terrell	3a1fefcf00	Simplify COVER parameters	2017-01-02 17:51:38 -08:00
Nick Terrell	96b39f65fa	Add COVER dictionary builder	2017-01-02 13:22:51 -08:00
Yann Collet	6334b04d61	compile object files, for faster recompilation	2017-01-02 03:22:18 +01:00
Yann Collet	f1cb55192c	fixed linux warnings	2017-01-02 01:11:55 +01:00
Yann Collet	0ec6a95ba1	minor fixes	2017-01-02 00:49:42 +01:00
Yann Collet	2ec635a162	use pthread_cond to send signals between threads	2017-01-01 17:31:33 +01:00
Nick Terrell	bb13387d7d	Fix pool for threading.h	2016-12-31 19:10:47 -05:00
Nick Terrell	4204e03e77	Add threading.h condition variables	2016-12-31 19:10:29 -05:00
Yann Collet	3b9d434356	extended ZSTDMT code support for non-MT systems and WIN32 (preliminary)	2016-12-31 16:32:19 +01:00
Yann Collet	c8efc1c874	simplified Buffer Pool	2016-12-31 14:45:33 +01:00
Yann Collet	3b29dbd9e8	new zstdmt version using generic treadpool	2016-12-31 06:04:25 +01:00
Yann Collet	c6a6417458	bench correctly measures time for multi-threaded compression (posix only)	2016-12-31 03:31:26 +01:00
Yann Collet	f765a375a5	Merge pull request #504 from terrelln/thread-pool ... [zstdmt] Add thread pool	2016-12-30 15:31:49 +01:00
Nick Terrell	e777a5be6b	Add a thread pool for ZSTDMT and COVER	2016-12-29 23:39:44 -08:00
Yann Collet	e70912c72b	Changed : input divided into roughly equal parts. ... Debug : can measure time waiting for mutexes to unlock.	2016-12-29 01:24:01 +01:00
Yann Collet	6c0ed9483a	compression threads use ZSTD_compressCCtx()	2016-12-28 17:08:28 +01:00
Yann Collet	8d7432914f	Merge pull request #503 from inikep/dev11 ... Dev11	2016-12-28 16:50:39 +01:00
Yann Collet	ce9e1452fd	protect buffer pool with a mutex	2016-12-28 15:31:19 +01:00
Przemyslaw Skibinski	75f3a3a335	changed default PREFIX and MANDIR	2016-12-28 12:32:41 +01:00
Yann Collet	3d93f2fce7	first zstdmt sketch	2016-12-27 07:19:36 +01:00
Yann Collet	39c105c605	Merge branch 'dev' of github.com:facebook/zstd into dev	2016-12-23 22:25:31 +01:00
Yann Collet	aca113f4f5	fixed ZSTD_sizeof_?Dict()	2016-12-23 22:25:03 +01:00
Yann Collet	c07d2e3a31	Merge pull request #499 from inikep/dev11 ... improved *BSD and Solaris compatibility	2016-12-23 21:32:03 +01:00
Przemyslaw Skibinski	63b0014b96	BSD: improved "make install"	2016-12-23 10:05:49 +01:00
Nick Terrell	1b5d4a7d53	ZDICT_finalizeDictionary() flipped comparison	2016-12-22 18:14:57 -08:00
Nick Terrell	bcbe77e994	ZDICT_finalizeDictionary() flipped comparison ... `ZDICT_finalizeDictionary()` had a flipped comparison. I also allowed `dictBufferCapacity == dictContentSize`. It might be the case that the user wants to fill the dictionary completely up, and then let zstd take exactly the space it needs for the entropy tables.	2016-12-22 18:01:14 -08:00
Nick Terrell	78a0072d5a	Fix failing test due to deprecation warning	2016-12-22 17:36:16 -08:00
Yann Collet	d76d1a9ef0	added ZDICT_finalizeDictionary()	2016-12-22 20:18:43 +01:00
Przemyslaw Skibinski	b999170311	Solaris: working "make -C lib install"	2016-12-22 20:14:37 +01:00
Yann Collet	ba75e9d8c3	fix : zlib wrapper compile in gnu90 mode	2016-12-21 19:57:18 +01:00
Yann Collet	0819abe3c1	added ZSTD_createDDict_byReference() body	2016-12-21 19:25:15 +01:00
Yann Collet	4e5eea61a8	added ZSTD_createDDict_byReference()	2016-12-21 16:44:35 +01:00
Yann Collet	8333106b8a	Merge branch 'dev' of github.com:facebook/zstd into dev	2016-12-21 16:44:24 +01:00
Yann Collet	0d7e84899f	Merge pull request #489 from inikep/v112 ... improved detection of POSIX	2016-12-21 16:42:46 +01:00
Yann Collet	1f57c2ed32	added : ZSTD_createCDict_byReference()	2016-12-21 16:20:11 +01:00
Przemyslaw Skibinski	2f6ccee6af	platform.h: removed Compiler Options	2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski	5736db219e	fix basic types redefinition	2016-12-21 09:26:00 +01:00
Nick Terrell	8157a4c3cc	Fix dictionary loading bug causing an MSAN failure ... Offset rep codes must be in the range `[1, dictSize)`. Fix dictionary loading to reject `0` as a offset rep code.	2016-12-20 10:47:52 -08:00
Przemyslaw Skibinski	f8046b8e72	Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112 ... # Conflicts: # appveyor.yml	2016-12-19 08:20:26 +01:00
Yann Collet	d564faa3c6	fix : ZSTD_initCStream_srcSize() correctly set srcSize in frame header	2016-12-18 21:39:15 +01:00
Yann Collet	1496c3dc47	Fix : size estimation when some samples are very large	2016-12-18 11:58:23 +01:00
Yann Collet	d46ecb58a5	added dll compilation tests	2016-12-17 16:28:12 +01:00
Nick Terrell	8de46ab51a	Export all API functions	2016-12-16 13:27:30 -08:00
Przemyslaw Skibinski	0a1caeef6d	VS: fixed 32-bit DLL compilation	2016-12-15 12:12:46 +01:00
Przemyslaw Skibinski	4e10bd339d	appveyor.yml: added tests of fullbench-dll fullbench-lib	2016-12-15 12:09:23 +01:00
Przemyslaw Skibinski	60f10aab6c	introduced ZSTDLIB_VISIBILITY	2016-12-15 11:32:31 +01:00
Yann Collet	2b36b238d3	changed variable name to estimatedSrcSize, to emphasize it does not need to be exact	2016-12-13 17:59:55 +01:00
Yann Collet	e795c8a5f6	Added ZSTD_initCStream_srcSize(). ... Added relevant test cases in zstreamtest	2016-12-13 17:00:14 +01:00
Yann Collet	5397a66b19	minor BMI version check	2016-12-13 15:21:06 +01:00
Yann Collet	35168679bd	Merge pull request #478 from terrelln/wildcopy-ub ... Fix execSequence wildcopy undefined behavior	2016-12-13 11:33:00 +01:00
Nick Terrell	064a143520	Fix execSequence wildcopy undefined behavior ... execSequence relied on pointer overflow to handle cases where `sequence.matchLength < 8`. Instead of passing an `size_t` to wildcopy, pass a `ptrdiff_t`.	2016-12-12 19:01:23 -08:00
Nick Terrell	e474aa55b4	Fix decompression buffer overrun ... Allows an adversary to write up to 3 bytes beyond the end of the buffer. Occurs if the match overlaps the `extDict` and `currentPrefix`, and the match length in the `currentPrefix` is less than `MINMATCH`, and `op-(16-MINMATCH) >= oMatchEnd > op-16`.	2016-12-12 18:05:30 -08:00
Yann Collet	c3a5c4bef8	introduced cycleLog	2016-12-12 00:47:30 +01:00
Yann Collet	c261f71f6a	minor variation of rescale fix	2016-12-12 00:25:07 +01:00
Nick Terrell	3826207a70	Simplify segfault fix ... Take advantage of the fact that `chainLog <= windowLog`.	2016-12-10 18:46:55 -08:00
Nick Terrell	0012332ce0	Fix compression segfault ... When the overflow protection kicks in, it makes sure that ip - ctx->base isn't too large. However, it didn't ensure that saved offsets are still valid. This change ensures that any valid offsets (<= windowLog) are still representable after the update. The bug would shop up on line 1056, when `offset_1 > current + 1`, which causes an underflow. This in turn, would cause a segfault on line 1063. The input must necessarily be longer than 1 GB for this issue to occur. Even then, it only occurs if one of the last 3 matches is larger than the chain size and block size.	2016-12-09 17:15:33 -08:00
Yann Collet	383b8088a3	minor lib build refactoring	2016-12-08 18:42:27 -08:00
Yann Collet	6e754fe76a	fixed lib soname. ... example : simple_compression : size overflow check	2016-12-08 18:26:56 -08:00
Przemyslaw Skibinski	7687913178	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2016-12-08 10:42:42 +01:00
Yann Collet	426a9d4b71	changed : dll : only approved ZSTD symbols are now exposed. All other symbols remain internal.	2016-12-07 16:39:34 -08:00
Przemyslaw Skibinski	4da53219a0	zstd Manual updated to 1.1.2	2016-12-07 11:18:40 +01:00
Yann Collet	379908be3d	fixed zstd.h for manual	2016-12-06 10:36:15 -08:00
Yann Collet	9dfd0af164	ZBUFF_ as a wrapper to ZSTD streaming API.	2016-12-06 17:16:41 +01:00
Yann Collet	825dffbc43	moved zbuff source files into lib/deprecated	2016-12-05 19:28:19 -08:00
Yann Collet	8f8e2b0b4a	fixed initialization warning	2016-12-05 18:00:50 -08:00
Yann Collet	e7a41a5955	added : dictID retrieval functions. ... added : unit tests for dictID retrieval functions	2016-12-05 16:21:06 -08:00
Yann Collet	9ffbeea875	API : changed : streaming decompression : implicit reset on starting new frames	2016-12-02 18:37:38 -08:00
Yann Collet	2238312c2f	fix dict loading	2016-12-02 11:36:11 -08:00
Przemyslaw Skibinski	821bf1febc	fixed Doxygen trailing comment	2016-12-02 16:13:41 +01:00
Yann Collet	b89af20353	reduced table sizes for HUF_readDTableX4	2016-12-01 18:24:59 -08:00
Yann Collet	a0d742b1e4	introduced HUF_buildCTable_wksp(), to reduce stack memory usage	2016-12-01 17:47:30 -08:00
Yann Collet	643d9a234b	replaced usage of FSE_buildCTable by FSE_buildCTable_wksp, using less stack space in the process	2016-12-01 16:24:04 -08:00
Yann Collet	e928f7e16d	introduced ext_wksp variants of count to reduce stack memory usage	2016-12-01 16:13:35 -08:00
Yann Collet	979cab412b	fixed some minor visual silent cast warnings. ... introduced FSE_count_parallel_wksp().	2016-11-30 18:10:38 -08:00
Yann Collet	5e00b848a8	FSE_compress_wksp() uses less stack space	2016-11-30 16:46:13 -08:00
Yann Collet	d79a9a00d9	Introduced FSE_compress_wksp() and FSE_buildCTable_wksp() to reduce stack memory usage	2016-11-30 15:52:20 -08:00
Yann Collet	766431909f	introduced FSE_decompress_wksp(), to use less stack space	2016-11-30 12:36:45 -08:00
Yann Collet	95eb43be09	updated pkg config file	2016-11-30 11:06:58 -08:00
Yann Collet	42247705a3	Merge pull request #461 from obache/neatsrc/pkgconfig ... libzstd.pc.in: Change to use variables for libdir and includedir	2016-11-30 20:03:42 +01:00
Yann Collet	ff504de391	minor decompression speed improvement	2016-11-29 17:42:46 -08:00
Yann Collet	25f46dcc0f	minor const	2016-11-29 16:59:27 -08:00
Yann Collet	a56ac2815c	restored normal decoder speed	2016-11-29 15:30:23 -08:00
Yann Collet	37870d7a66	fixed minor visual warning	2016-11-29 14:31:57 -08:00
Yann Collet	013a2b58ae	Merge pull request #464 from terrelln/guards ... Fix ZSTD_STATIC_LINKING_ONLY with double include	2016-11-29 23:06:10 +01:00
Yann Collet	167c494748	Merge branch 'dev' of github.com:facebook/zstd into dev	2016-11-29 14:05:15 -08:00
Yann Collet	4f5350f610	long matches support overflow	2016-11-29 13:12:24 -08:00
Nick Terrell	05c00f2ff7	Fix ZSTD_STATIC_LINKING_ONLY with double include	2016-11-29 11:54:39 -08:00
OBATA Akio	c53eea7c1a	libzstd.pc.in: Change to use variables for libdir and includedir	2016-11-29 16:47:53 +09:00
Yann Collet	52e136ed3d	long decoder compatible with round and separate buffers	2016-11-28 19:59:11 -08:00
Yann Collet	ce3527ca0c	combined normal and long decoder	2016-11-28 18:38:52 -08:00
Yann Collet	8993bee997	restored normal mode	2016-11-28 16:11:30 -08:00
Yann Collet	764e70a4f3	added decodeSequencesLong	2016-11-28 15:50:16 -08:00
Yann Collet	73f88a66f1	added prefetch	2016-11-23 15:43:30 -08:00
Yann Collet	50524bf0da	delayed decompression	2016-11-23 15:11:07 -08:00
Przemyslaw Skibinski	fc4193bda5	fixed g++ warnings	2016-11-23 18:17:18 +01:00
Przemyslaw Skibinski	9ca65af810	zstd_opt.h: improved price function	2016-11-23 17:22:54 +01:00
Przemyslaw Skibinski	ad3e94512c	fixed warnings from static analyzer in zstd_opt.h	2016-11-21 20:22:12 +01:00
Przemyslaw Skibinski	eec700a3b7	exclude zbuff_compress.c and zbuff_decompress.c	2016-11-21 14:34:03 +01:00
Przemyslaw Skibinski	62d19a6f39	lib\README.md: added Using MinGW+MSYS to create DLL	2016-11-21 14:22:08 +01:00
Przemyslaw Skibinski	b85f767743	files to generate ZSTD Windows binary package	2016-11-21 14:10:55 +01:00
Przemyslaw Skibinski	8bb86e330b	create DLL with Windows	2016-11-21 12:51:01 +01:00
Przemyslaw Skibinski	93a09eedf1	added libzstd.def	2016-11-21 12:33:27 +01:00
Przemyslaw Skibinski	5a17223691	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2016-11-18 11:47:01 +01:00
Przemyslaw Skibinski	3d18088b38	updated windres	2016-11-17 18:04:41 +01:00
Yann Collet	0d761dbe95	Merge pull request #453 from inikep/dev11 ... fullbench-dll	2016-11-16 15:45:30 -08:00
Yann Collet	52afb3993e	zbuff API now generates deprecation warnings	2016-11-16 08:50:54 -08:00
Przemyslaw Skibinski	179555c1d1	working fullbench-dll	2016-11-15 18:05:46 +01:00
Nick Terrell	4359d21ad7	Merge two memset() calls into one	2016-11-14 17:52:51 -08:00
Nick Terrell	24701de877	Fix uninitialized memory read	2016-11-14 13:57:05 -08:00
Yann Collet	8e4901eccd	removed zbuff.h from include installation	2016-11-08 15:45:39 -08:00
Yann Collet	fd3be6bc97	bump version number to 1.1.2	2016-11-07 14:35:41 -08:00
Nick Terrell	dc904ad17b	Fix bug in zstd v0.{5, 6} dictionary decompression ... Introduced by bb68062c59.	2016-11-04 16:18:59 -07:00
Przemyslaw Skibinski	38b590ad69	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 ... # Conflicts: # lib/Makefile	2016-11-04 10:10:54 +01:00
Yann Collet	407a11f63e	fixed Visual compatibility	2016-11-03 15:52:01 -07:00
Yann Collet	11812260d1	Merge pull request #439 from terrelln/dev ... ZSTD_compress_usingDict() specify dict == NULL	2016-11-03 14:15:36 -07:00
Nick Terrell	c8a9ac312b	Fix dynamic libzstd symlinks	2016-11-03 12:32:48 -07:00
Przemyslaw Skibinski	3a415594b1	fixed MinGW compilation	2016-11-03 12:59:20 +01:00
Yann Collet	7347869fb6	fixed make install	2016-11-02 22:28:37 -07:00
Nick Terrell	d82efd8a70	ZSTD_compress_usingDict() when dict gets loaded ... Specify that when `dict == NULL || dictSize < 8` no dictionary gets loaded. Also add some periods.	2016-11-02 18:07:16 -07:00
Yann Collet	179b19776f	fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h ... Added : ZSTD_isFrame() in experimental section	2016-11-02 17:30:49 -07:00
Yann Collet	f3f13211ae	Fix #419 : no warning when setting custom LDFLAGS	2016-11-02 17:02:45 -07:00
Yann Collet	0a5a5fb7fd	Fix #418 : printing selected segments in zdict debug mode can segfault with certain pathological patterns	2016-11-02 13:57:55 -07:00
Yann Collet	31e660e7aa	more accurate default maximum window size	2016-10-29 03:56:45 -07:00
Yann Collet	2115724c22	Merge pull request #430 from terrelln/exec-sequences ... ZSTD_execSequence() accepts match in last 7 bytes	2016-10-28 10:45:05 -07:00
Nick Terrell	10bfd0c0d5	Fix ZSTD_execSequence() performance regression ... Commit ae1cb3b3d0 caused the regression. It is an instruction alignment issue, because if it is `U64 i` instead of `U32 i`, the regression returns. This patch fixes the regression in gcc, but only gets some of the clang performance back. Benchmarks: Run on `silesia.tar`. I only show levels 1-5 because the performance regression was uniform across all levels. I did one run on levels 1-19 and it looked good. | Build | Level | Before | While | After | |-------|-------|-------:|------:|------:| | gcc | 1 | 931.4 | 904.4 | 932.8 | | gcc | 2 | 849.1 | 822.6 | 851.2 | | gcc | 3 | 815.6 | 790.6 | 818.9 | | gcc | 4 | 794.1 | 770.7 | 798.0 | | gcc | 5 | 785.7 | 760.7 | 788.8 | | clang | 1 | 705.5 | 683.2 | 693.8 | | clang | 2 | 670.0 | 649.2 | 660.7 | | clang | 3 | 659.6 | 639.8 | 651.4 | | clang | 4 | 652.5 | 634.7 | 645.9 | | clang | 5 | 646.9 | 625.5 | 637.7 |	2016-10-27 16:19:57 -07:00
Yann Collet	ee5b725823	ZSTD_initCStream() optimization : do not allocate a CDict when no dictionary used	2016-10-27 14:20:55 -07:00
Nick Terrell	eb7873a048	ZSTD_execSequence() accepts match in last 7 bytes ... The zstd reference compressor will not emit a match in the last 7 bytes of a block. The decompressor will also not accept a match in the last 7 bytes. This patch makes the decompressor accept a match in the last 7 bytes.	2016-10-25 21:24:15 -07:00
Yann Collet	335ad5d4d4	added ZSTD_initDStream_usingDDict() . ... slightly optimized ZSTD_initDStream() when no dictionary . fixed ZSTD_sizeof_CStream() .	2016-10-25 17:47:02 -07:00
Yann Collet	9516234e67	first sketch for ZSTD_initCStream_usingCDict()	2016-10-25 16:19:52 -07:00
Yann Collet	62d9a7ddfd	Merge pull request #429 from inikep/btopt2 ... Btopt2	2016-10-25 14:48:43 -07:00
Przemyslaw Skibinski	5c5f01f3da	added ZSTD_btopt2 strategy	2016-10-25 12:25:07 +02:00
Yann Collet	7b5948cca7	Merge pull request #426 from terrelln/fixes ... Fix various {A, M}SAN bugs	2016-10-24 23:42:26 -07:00
Yann Collet	37d130031d	updated comments on context re-use	2016-10-24 17:22:12 -07:00
Nick Terrell	b2c39a22b0	Fix compiler narrowing warning	2016-10-24 14:50:13 -07:00
Nick Terrell	f698ad6deb	Merge remote-tracking branch 'upstream/dev' into fixes ... * upstream/dev: added doc\zstd_manual.html added contrib\gen_html zstd_compression_format.md moved to doc/ Fix small bug in ZSTD_execSequence() improved ZSTD_compressBlock_opt_extDict_generic protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob zstd_opt.h: small improvement in compression ratio improved dicitonary segment merge use implicit rules to compile zstd_decompress.c detect early impossible decompression scenario in legacy decoder v0.5 no repeat mode in legacy v0.5 fixed invalid invocation of dictionary in legacy decoder v0.5 fix edge case fix command line interpretation fixed minor corner case zstd.h: added the Introduction section fixed clang 3.5 warnings zstd.h: updated comments	2016-10-24 13:10:13 -07:00
Yann Collet	4239a207dd	Merge pull request #425 from inikep/dev11 ... Doc	2016-10-24 11:11:40 -07:00
Nick Terrell	f9c9af3c2e	Reject dictionaries with incomplete entropy tables ... If a dictionary specifies that a symbol has probability zero in its `matchLength`, `literalLength`, or `offset` FSE table, but the symbol appears when compressing input, the compressor fails. Ensure that dictionaries support all `matchLength`, and `literalLength` codes. They must also support all of the `offset` codes required to represent every possible offset that can appear in the first block.	2016-10-24 10:42:44 -07:00
Przemyslaw Skibinski	984b66cd72	added contrib\gen_html	2016-10-24 15:59:51 +02:00
Przemyslaw Skibinski	3ee94a7600	zstd_compression_format.md moved to doc/	2016-10-24 15:58:07 +02:00
Yann Collet	97611611a3	Merge pull request #423 from terrelln/exec-seq-patch ... Fix small bug in ZSTD_execSequence()	2016-10-21 17:02:06 -07:00
Nick Terrell	ae1cb3b3d0	Fix small bug in ZSTD_execSequence() ... `memmove(op, match, sequence.matchLength)` is not the desired behavior. Overlap is allowed, and handled as if we did `*op++ = *match++`, which is not how `memmove()` handles overlap. Only triggered if both of the following conditions are met: * The match spans extDict & currentPrefixSegment * `oLitEnd <= oend_w < oLitEnd + length1 < oMatchEnd <= oend`. These two conditions imply that the block is less than 15 bytes long. This bug isn't triggered by the streaming API, because it allocates enough space for the window size + the block size, so there cannot be a match that is within 8 bytes of the end and overlaps with itself. It cannot be triggered by the block decompression API because all of the decompressed data is in the currentPrefixSegment. Introduced by commit 7158584399	2016-10-21 12:13:44 -07:00
Przemyslaw Skibinski	4732074a71	improved ZSTD_compressBlock_opt_extDict_generic	2016-10-21 11:19:00 +02:00
Yann Collet	da3bd8b6de	protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob	2016-10-20 20:11:00 -07:00
Przemyslaw Skibinski	d365ae3497	zstd_opt.h: small improvement in compression ratio	2016-10-20 11:49:02 +02:00
Przemyslaw Skibinski	575ab00db7	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2016-10-20 11:01:52 +02:00
Nick Terrell	d760529a05	Fix stack buffer overrun when weightTotal == 0 ... If `weightTotal == 0`, then `BIT_highbit32(weightTotal)` is undefined behavior in the case that it calls `__builtin_clz()`. If `tableLog == HUF_TABLELOG_ABSOLUTEMAX` then we will access one byte beyond the end of the buffer.	2016-10-19 11:39:11 -07:00
Nick Terrell	bb68062c59	Unitialized memory read in ZSTD_decodeSeqHeaders() ... Caused by two things: 1. Not checking that `ip` is in range except for the first byte. 2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.	2016-10-18 16:41:33 -07:00
Yann Collet	52c1bf93fe	improved dicitonary segment merge	2016-10-18 16:34:58 -07:00
Nick Terrell	7b06ad7a05	Backport fix from commit 125d817 ... This fixes a read of unitialized memory. Full commit hash: 125d81774f.	2016-10-18 14:52:34 -07:00
Nick Terrell	f45b157d95	Backport fix from commit 9e8b09a ... Fixes uninitialized memory reads. Full commit hash: 9e8b09a7bd	2016-10-18 14:22:49 -07:00
Yann Collet	f7906d5955	detect early impossible decompression scenario in legacy decoder v0.5	2016-10-18 13:48:32 -07:00
Yann Collet	9313c8d953	no repeat mode in legacy v0.5	2016-10-18 13:36:15 -07:00
Yann Collet	83d7bdee4b	fixed invalid invocation of dictionary in legacy decoder v0.5	2016-10-18 12:25:43 -07:00
Yann Collet	197a55ee7b	fix edge case	2016-10-18 11:27:52 -07:00
Nick Terrell	fd98087047	Fix stack buffer overflow in HUF_readCTable() ... If `w ==0` on line 153, then `CTable[n].nbBits == tableLog + 1`. Then `nbPerRank[CTable[n].nbBits]` and `valPerRank[CTable[n].nbBits]` are stack buffer overflows.	2016-10-17 18:16:59 -07:00
Yann Collet	06573e17be	fixed minor corner case	2016-10-17 17:28:28 -07:00
Nick Terrell	bfd943ace5	Fix buffer overrun in ZSTD_loadDictEntropyStats() ... The table log set by `FSE_readNCount()` was not checked in `ZSTD_loadDictEntropyStats()`. This caused `FSE_buildCTable()` to stack/heap overflow in a few places. The benchmarks look good, there is no obvious compression performance regression: > ./zstds/zstd.opt.0 -i10 -b1 -e10 ~/bench/silesia.tar 1#silesia.tar : 211988480 -> 73656930 (2.878), 271.6 MB/s , 716.8 MB/s 2#silesia.tar : 211988480 -> 70162842 (3.021), 204.8 MB/s , 671.1 MB/s 3#silesia.tar : 211988480 -> 66997986 (3.164), 156.8 MB/s , 658.6 MB/s 4#silesia.tar : 211988480 -> 66002591 (3.212), 136.4 MB/s , 665.3 MB/s 5#silesia.tar : 211988480 -> 65008480 (3.261), 98.9 MB/s , 647.0 MB/s 6#silesia.tar : 211988480 -> 62979643 (3.366), 65.2 MB/s , 670.4 MB/s 7#silesia.tar : 211988480 -> 61974560 (3.421), 44.9 MB/s , 688.2 MB/s 8#silesia.tar : 211988480 -> 61028308 (3.474), 32.4 MB/s , 711.9 MB/s 9#silesia.tar : 211988480 -> 60416751 (3.509), 21.1 MB/s , 718.1 MB/s 10#silesia.tar : 211988480 -> 60174239 (3.523), 22.2 MB/s , 721.8 MB/s > ./compress_zstds/zstd.opt.1 -i10 -b1 -e10 ~/bench/silesia.tar 1#silesia.tar : 211988480 -> 73656930 (2.878), 273.8 MB/s , 722.0 MB/s 2#silesia.tar : 211988480 -> 70162842 (3.021), 203.2 MB/s , 666.6 MB/s 3#silesia.tar : 211988480 -> 66997986 (3.164), 157.4 MB/s , 666.5 MB/s 4#silesia.tar : 211988480 -> 66002591 (3.212), 132.1 MB/s , 661.9 MB/s 5#silesia.tar : 211988480 -> 65008480 (3.261), 96.8 MB/s , 641.6 MB/s 6#silesia.tar : 211988480 -> 62979643 (3.366), 63.1 MB/s , 677.0 MB/s 7#silesia.tar : 211988480 -> 61974560 (3.421), 44.3 MB/s , 678.2 MB/s 8#silesia.tar : 211988480 -> 61028308 (3.474), 33.1 MB/s , 708.9 MB/s 9#silesia.tar : 211988480 -> 60416751 (3.509), 21.5 MB/s , 710.1 MB/s 10#silesia.tar : 211988480 -> 60174239 (3.523), 21.9 MB/s , 723.9 MB/s	2016-10-17 16:55:52 -07:00
Nick Terrell	4db751668f	Fix buffer overrun in ZSTD_loadEntropy() ... The table log set by `FSE_readNCount()` was not checked in `ZSTD_loadEntropy()`. This caused `FSE_buildDTable(dctx->MLTable, ...)` to overwrite the beginning of `dctx->hufTable`. The benchmarks look good, there is no obvious performance regression: > ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar 1#silesia.tar : 211988480 -> 73656930 (2.878), 268.2 MB/s , 701.0 MB/s 2#silesia.tar : 211988480 -> 70162842 (3.021), 199.5 MB/s , 666.9 MB/s 3#silesia.tar : 211988480 -> 66997986 (3.164), 154.9 MB/s , 655.6 MB/s 4#silesia.tar : 211988480 -> 66002591 (3.212), 128.9 MB/s , 648.4 MB/s 5#silesia.tar : 211988480 -> 65008480 (3.261), 98.4 MB/s , 633.4 MB/s > ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar 1#silesia.tar : 211988480 -> 73656930 (2.878), 266.1 MB/s , 703.7 MB/s 2#silesia.tar : 211988480 -> 70162842 (3.021), 199.0 MB/s , 666.6 MB/s 3#silesia.tar : 211988480 -> 66997986 (3.164), 156.2 MB/s , 656.2 MB/s 4#silesia.tar : 211988480 -> 66002591 (3.212), 133.2 MB/s , 647.4 MB/s 5#silesia.tar : 211988480 -> 65008480 (3.261), 96.3 MB/s , 633.3 MB/s	2016-10-17 15:51:15 -07:00
Nick Terrell	ccfcc643da	Check if dict is empty before reading first byte	2016-10-17 11:46:03 -07:00
Yann Collet	2b361cf2f1	minor opt	2016-10-14 16:09:07 -07:00
Yann Collet	7933434fdf	Merge branch 'dev' of github.com:facebook/zstd into dev	2016-10-14 13:32:35 -07:00
Yann Collet	d4cda27b63	new command -M#, to limit memory usage during decompression (#403)	2016-10-14 13:32:20 -07:00
Nick Terrell	3b9cdf9220	Fix ubsan failures (pass NULL to memcpy)	2016-10-12 20:54:42 -07:00
Yann Collet	5d919e7ac3	added ZSTD_error_frameParameter_windowTooLarge (#403)	2016-10-12 17:29:24 -07:00
Yann Collet	e19111c42f	make creates libzstd binaries (#415)	2016-10-12 11:09:36 -07:00
Yann Collet	8b70d012f0	fix cmake	2016-10-12 10:23:53 -07:00
Yann Collet	38fb0dc4cf	Merge pull request #416 from terrelln/exec-sequence ... Fix ZSTD_execSequence() edge case	2016-10-12 10:17:53 -07:00
Nick Terrell	7158584399	Fix ZSTD_execSequence() edge case	2016-10-12 10:05:26 -07:00
Yann Collet	f52cd03e73	bumped version number	2016-10-11 17:29:27 -07:00
Yann Collet	ef2357d0d3	created error_private.c, so that a single list of error strings get included	2016-10-11 17:24:50 -07:00
Yann Collet	14efab827b	added zstd_errors.h to include installation	2016-10-11 16:51:29 -07:00
Yann Collet	a17fd7312a	changed error_public.h into zstd_errors.h	2016-10-11 16:41:09 -07:00
Yann Collet	18b51b99c0	sync fse	2016-10-11 08:21:09 -07:00
inikep	2d2613399a	zstd.h: added the Introduction section	2016-10-06 16:28:21 +02:00
inikep	ba1db376ac	fixed clang 3.5 warnings	2016-10-06 14:22:48 +02:00
inikep	82057aa7ec	zstd.h: updated comments	2016-10-06 13:23:52 +02:00
Yann Collet	df6797447f	update dictionary builder warning comments	2016-09-27 15:14:32 +02:00
Yann Collet	47094ea66b	added comment on filePos	2016-09-26 18:03:33 +02:00
Yann Collet	cf409a7e2a	fixed : init*_advanced() followed by reset() with different pledgedSrcSiz	2016-09-26 16:41:05 +02:00
Yann Collet	2f2639438a	zstreamtest can fuzztest pledgedSrcSize	2016-09-26 14:06:08 +02:00
Christophe Chevalier	dc245e91cb	Changed to use ZSTDLIBv06_API and ZSTDLIBv07_API for DLL exports to fix warning ... - changed name to prevent collision with ZSTDLIB_API used by non-legacy dll exports	2016-09-23 17:09:36 +02:00
Yann Collet	21412bb3f6	Merge branch 'dev' of github.com:Cyan4973/zstd into dev	2016-09-22 15:57:56 +02:00
Yann Collet	51f4d566c2	small decompression speed boost for very small data	2016-09-22 15:57:28 +02:00
Yann Collet	97b378a6f8	Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize.	2016-09-21 17:20:19 +02:00
Yann Collet	993060e0f2	cli : better adaptation to small files	2016-09-21 16:46:08 +02:00
Yann Collet	1eb2fdc74f	bumped version number	2016-09-18 12:21:47 +02:00
Yann Collet	a6bdf55759	fixed memory leak	2016-09-15 17:02:06 +02:00
Yann Collet	644a8da88a	fixed minor conversion warning	2016-09-15 16:16:21 +02:00
Yann Collet	4cb212938c	introduced ZSTD_resetCStream()	2016-09-15 14:54:07 +02:00
Yann Collet	fa0c09760c	variable renaming	2016-09-15 14:11:01 +02:00
Yann Collet	d7c6589df8	support ZSTD_sizeof_*() on NULL ... added ZSTD_sizeof_CDict()	2016-09-15 02:57:27 +02:00
Yann Collet	e91c4b4cef	introduced ZSTD_resetDStream() . ... added : ZSTD_sizeof_DDict()	2016-09-14 16:55:44 +02:00
Yann Collet	d092d77cfc	minor variable renaming	2016-09-14 16:14:57 +02:00
Yann Collet	64deef3bee	Fixed srcSize=1	2016-09-14 00:16:07 +02:00
Yann Collet	26ec254066	new strategy for faster DDict decompression	2016-09-13 16:52:16 +02:00
Yann Collet	ac175d46d4	updated comments	2016-09-13 00:51:47 +02:00
Yann Collet	a3481d6de0	make uninstall	2016-09-12 05:04:26 +02:00
Yann Collet	b3060f7a9e	changed streaming decoder behavior : now, when all compressed frame is consumed, it means decompression is completed, with regenerated data fully flushed.	2016-09-09 16:44:16 +02:00
Yann Collet	01c199226a	updated decompression streaming example	2016-09-08 19:29:04 +02:00
Yann Collet	5c6d244973	Merge branch 'dev' of github.com:facebook/zstd into dev	2016-09-07 14:54:54 +02:00
Yann Collet	ac8bace6b1	support large skippable frames	2016-09-07 14:54:23 +02:00
Yann Collet	0e07bf3f60	added comments on searchLength min / max (#337)	2016-09-07 06:33:02 +02:00
Yann Collet	95d07d7447	introduced CHECK_E	2016-09-06 16:38:51 +02:00
Yann Collet	3e21ec5b01	introduced CHECK_F	2016-09-06 15:36:19 +02:00
Yann Collet	5c956d593c	FORCE_INLINE common definition	2016-09-06 15:05:19 +02:00
Yann Collet	edbcd9f5b2	fixed zbufftest	2016-09-06 14:30:57 +02:00
Yann Collet	b624922b14	fixed checksum	2016-09-06 11:16:57 +02:00
Yann Collet	a7737f6a60	improved compression on small files when using same parameters	2016-09-06 09:44:59 +02:00
Yann Collet	7ae67bb18a	small compression speed gains with using_CDict	2016-09-06 06:28:05 +02:00
Yann Collet	1d4208c029	clarified streaming decompression inlined doc	2016-09-06 05:16:40 +02:00
Yann Collet	7c83dfd5c2	ZSTD_frameHeaderSize_prefix (#340), as result of ZSTD_initStream	2016-09-05 19:47:43 +02:00
Yann Collet	fa72f6bdce	clarified inline doc for streaming	2016-09-05 17:39:56 +02:00
Yann Collet	c73a8109bb	Merge pull request #344 from inikep/dev10 ... unified error codes for legacy decoders	2016-09-05 07:46:33 -07:00
inikep	45db83f98d	ZSTD_decodeLiteralsBlock renamed to ZSTDv01_decodeLiteralsBlock	2016-09-05 14:46:24 +02:00
inikep	476964f6a1	ZSTD_decodeSeqHeaders renamed to ZSTDv01_decodeSeqHeaders	2016-09-05 13:34:57 +02:00
inikep	c13faa1b0f	legacy decoders: restored #include <intrin.h> for VC++	2016-09-05 13:25:07 +02:00
inikep	8161e7321a	unified error codes for legacy decoders	2016-09-05 12:29:51 +02:00
Thomas Klausner	b85cdabd50	Enable install targets for NetBSD.	2016-09-04 14:37:57 +02:00
Yann Collet	33a0465a51	fixed a few links	2016-09-02 22:11:49 -07:00
Yann Collet	d56dbc02d3	removed g_displayLevel	2016-09-02 17:28:41 -07:00
Yann Collet	855766d73d	clarified dictionary in format description	2016-09-02 17:04:49 -07:00
Yann Collet	d725427a3c	g_time => local displayTime	2016-09-02 15:32:39 -07:00
Yann Collet	1563bfeabc	fixing FORCE_INLINE for older compilers (#330)	2016-09-02 11:44:21 -07:00
Yann Collet	7304eb7c09	bumped version number	2016-09-01 15:49:26 -07:00
Yann Collet	901e85fe26	version bump	2016-08-31 07:51:25 -07:00
Yann Collet	1c59c20903	removed redundant files	2016-08-31 07:15:44 -07:00
Yann Collet	599c69d917	minor Makefile updates	2016-08-30 13:33:20 -07:00
David Lam	e10f7f3dcb	merge	2016-08-30 12:03:36 -07:00
Yann Collet	4ded9e591c	added boilerplate	2016-08-30 11:06:28 -07:00
Yann Collet	3b15f1f10f	minor refactor	2016-08-30 09:58:50 -07:00
Yann Collet	240795bef7	Merge branch 'dev' of github.com:Cyan4973/zstd into dev	2016-08-30 06:51:55 -07:00
Yann Collet	14200a20f0	Fixed issue #304, reported by @borzunov	2016-08-30 06:51:00 -07:00
David Lam	da9d3b7057	Cleanup some errors in typedef comments and remove duplicated HOWTO from zbuff_decompress.c	2016-08-29 17:31:51 -07:00
Yann Collet	09c3c8e885	Merge pull request #307 from inikep/dev08 ... updated README.md	2016-08-29 16:32:33 -07:00
inikep	6416b0d705	updated README.md	2016-08-29 13:04:26 +02:00
Yann Collet	23b6e05d8e	ZSTD_malloc() and ZSTD_free(), to simplify customMem	2016-08-28 21:05:43 -07:00
Yann Collet	5f53b0335e	fixed continuation context	2016-08-28 10:00:49 -07:00
Yann Collet	767d8f66fa	legacy contexts can be re-used	2016-08-28 08:19:47 -07:00
Yann Collet	4bf317dd00	first version supporting legacy streams (transparent decoding)	2016-08-28 07:43:34 -07:00
Yann Collet	e19a9ef05d	update compression level table	2016-08-26 20:02:49 +02:00
Yann Collet	9a021c1aae	fixed some minor clang warnings	2016-08-26 09:05:06 +02:00
Yann Collet	cb5a320705	made -Wdocumentation a clang only flag	2016-08-26 08:06:36 +02:00
Yann Collet	87c18b2ebd	fixed multiple minor warnings for XCode	2016-08-26 01:43:47 +02:00
Yann Collet	0d59a6f73a	removed debug strings	2016-08-25 22:42:46 +02:00
Yann Collet	5a02b69215	reinforced fix for huge files	2016-08-25 22:24:59 +02:00
Yann Collet	96bdd87de4	fixed : compression bug on very large files	2016-08-25 14:34:42 +02:00
inikep	a3a47ec4d0	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into Other	2016-08-24 21:25:49 +02:00
Yann Collet	a2cdffe556	fixed wrong parameter issue	2016-08-24 19:42:15 +02:00
inikep	e416e30019	remove unnecessary comments	2016-08-24 17:32:09 +02:00
inikep	4e90f6c1e0	removed ZSTD_LOG_ENCODE and ZSTD_LOG_BLOCK	2016-08-24 17:24:11 +02:00
inikep	83388e109f	removed ZSTD_LOG_PARSER	2016-08-24 17:22:20 +02:00
inikep	8a36f8527c	removed stats in debug mode	2016-08-24 17:19:12 +02:00
inikep	57ef4b1a0d	zstd_v07.c: removed unused macros	2016-08-24 17:16:56 +02:00
Yann Collet	24b68a5b23	update cLevel table for 256KB	2016-08-24 14:22:26 +02:00
Yann Collet	c54692faeb	improved level 3	2016-08-24 01:45:46 +02:00
Yann Collet	17e482efdd	added ZSTD_setDStreamParameter()	2016-08-23 16:58:10 +02:00
Yann Collet	3071c3e303	STREAM_WINDOW_MAX : protect streaming from unreasonable memory requirements	2016-08-23 01:34:34 +02:00
Yann Collet	70e3b31306	fixed playtests on os-x	2016-08-23 01:18:06 +02:00
Yann Collet	cb3276329a	added sizeof CStream and DStream	2016-08-23 00:31:59 +02:00
Yann Collet	d1733f7417	fixed crc bug in rare timing conditions within bench.c	2016-08-21 01:04:46 +02:00
Yann Collet	8baf78a291	minor coding style	2016-08-20 13:04:20 +02:00
Yann Collet	1bee2d5e08	slight decompression speed improvement	2016-08-20 02:59:04 +02:00
Yann Collet	0cfe2ec2fd	sync fse version	2016-08-20 00:26:26 +02:00
Yann Collet	af1960396b	sync huff0	2016-08-19 19:38:19 +02:00
Yann Collet	7be46bf8f2	promoted streaming API to stable ... (except _advanced() variants)	2016-08-19 18:39:36 +02:00
Yann Collet	fdba57d513	update version number	2016-08-19 18:32:30 +02:00
Yann Collet	da3fbcb302	Added ZDICT_getDictID()	2016-08-19 14:23:58 +02:00
Yann Collet	a5dbf9f629	Merge pull request #297 from borzunov/dev ... Export functions related to dictionary compression from DLL	2016-08-18 15:05:01 +02:00
Yann Collet	49d105cfcf	better warning and error messages in case of dictionary training failure (#292)	2016-08-18 15:02:11 +02:00
Alexander Borzunov	0f6f17a14f	Rename ZSTDLIB_API to ZDICTLIB_API in zdict.h	2016-08-18 16:47:06 +05:00
Alexander Borzunov	1f48382b1a	Export functions related to dictionary compression from DLL	2016-08-18 16:12:49 +05:00
Yann Collet	e80d15304a	Merge pull request #296 from inikep/Other ... Other	2016-08-18 11:48:48 +02:00
inikep	a7bb322a93	removed never referenced functions	2016-08-18 10:30:21 +02:00
Yann Collet	18442c1482	minor refactoring	2016-08-18 01:40:32 +02:00
Yann Collet	c411902230	fixed g++ conversion warning	2016-08-17 01:50:54 +02:00
Yann Collet	53e17fbd5e	updated streaming API	2016-08-17 01:39:22 +02:00
Yann Collet	655393cc72	updated doc for streaming API	2016-08-16 15:11:28 +02:00
Yann Collet	104e5b072d	added : streaming decompression API	2016-08-16 15:11:28 +02:00
Yann Collet	5a0c8e2439	new streaming API (compression)	2016-08-16 15:11:27 +02:00
Yann Collet	ba92046031	Merge branch 'dev' of github.com:Cyan4973/zstd into dev	2016-08-11 22:10:04 +02:00
Yann Collet	e9b414d825	fixed msan warning (#281)	2016-08-11 22:09:09 +02:00
inikep	5f49eba512	added usage of rep[0]-1 for the optimal parser	2016-08-10 15:01:53 +02:00
inikep	98e08cbe34	fixed: tree not updated after finding very long rep matches	2016-08-10 15:00:30 +02:00
inikep	038d1497c9	fixed compilation with Visual Studio 2005	2016-08-10 14:30:10 +02:00
inikep	48849f86f0	fixed compilation with Intel Compiler with Windows	2016-08-10 14:26:35 +02:00
Yann Collet	1ea5622a32	updated xxhash	2016-08-10 09:40:08 +02:00
Yann Collet	666398e7ed	added : xxhash namespace enforced from xxhash.h. ... added : xxhash namespace test. removed : -DXXH_NAMESPACE	2016-08-10 08:16:51 +02:00
Yann Collet	8ded0b84aa	update xxhash to v0.6.2	2016-08-10 07:40:40 +02:00
Yann Collet	280f9a8754	minor comment	2016-08-08 00:44:00 +02:00
Yann Collet	e0b4a2d40f	fixed dictionary generation, reported by Bartosz Taudul	2016-08-03 03:36:03 +02:00
Yann Collet	ae40b18d55	bumped library number	2016-08-03 01:59:21 +02:00
Yann Collet	0763905f44	ZSTD_compress_usingCDict() correctly provides original size by default in frame header ... Fixed dictionary examples	2016-08-03 01:57:57 +02:00
Yann Collet	bf2bc112bb	bench : controlled display update when loading lot of files	2016-08-02 23:48:13 +02:00
Yann Collet	346efccc35	fixed doc typo	2016-08-02 14:26:00 +02:00
Yann Collet	f116e87f59	fixed analyzer warning	2016-08-01 19:15:18 +02:00
Yann Collet	9ba929f1d4	Merge branch 'dev' of github.com:Cyan4973/zstd into dev	2016-08-01 02:26:59 +02:00
Yann Collet	3ca750372d	updated doc (#269)	2016-08-01 02:26:20 +02:00
Yann Collet	c55eb18c11	Merge pull request #267 from inikep/dev08 ... fixed ZSTD_compressBlock_opt_extDict_generic	2016-07-31 22:00:16 +02:00
inikep	056df510aa	fixed ZSTD_compressBlock_opt_extDict_generic	2016-07-31 20:08:53 +02:00
Yann Collet	917fe188f1	Implemented repOffset "minus 1" on ll==0	2016-07-31 04:01:57 +02:00
Yann Collet	2a2ba3691c	Merge pull request #266 from jrmarino/master ... Enable build on FreeBSD ports (includes DragonFly BSD) [dev branch]	2016-07-31 02:21:26 +02:00
jrmarino	0d07ec0c0c	Enable build on FreeBSD ports (includes DragonFly BSD) ... Zstd has been introduced to FreeBSD ports (http://www.freshports.org/archivers/zstd/) which DragonFly BSD also uses. FreeBSD and DragonFly use the install targets (albeit modified in some cases) so they must be added to the associated Makefile filters.	2016-07-30 19:11:15 -05:00
Yann Collet	8cebfd1d26	fix attempt on test-zstd-speed	2016-07-31 01:59:23 +02:00
Yann Collet	66f69e58d2	restore decompression speed on fizzle	2016-07-30 15:32:47 +02:00
Yann Collet	3b2bd1d11c	zstd_opt uses same tables as zstd_compress	2016-07-30 13:21:41 +02:00
Yann Collet	f714f59c16	fixed visual warning	2016-07-30 12:05:28 +02:00
Yann Collet	761f8dbbd2	back to normal table cell copy	2016-07-30 11:43:53 +02:00
Yann Collet	3c6b808870	minor decompression speed gains	2016-07-30 03:20:47 +02:00
Yann Collet	70a9ff4af3	fixed too large selectivity level, reported by Ilona Papava	2016-07-30 01:09:14 +02:00
Yann Collet	c0ce4f1211	slightly improved compression speed	2016-07-30 00:55:13 +02:00
Yann Collet	ed57d8530a	new seqStore	2016-07-29 21:22:17 +02:00
Yann Collet	6b615d32cd	Updated API comments, following suggestions by Bryan O'Sullivan	2016-07-29 19:40:37 +02:00
Yann Collet	c00d30fbe4	Merge pull request #264 from inikep/dev08 ... Dev08	2016-07-29 17:42:30 +02:00
inikep	6b68ba2079	zstd_opt.h: fixed checking of rep codes (2)	2016-07-29 16:45:39 +02:00
inikep	59b86fc141	zstd_opt.h: fixed checking of rep codes	2016-07-29 11:00:33 +02:00
Yann Collet	6a82f0f8bf	minor comments	2016-07-29 00:55:45 +02:00
Yann Collet	ffa7d0ac1e	clarified comment	2016-07-28 21:01:17 +02:00
Yann Collet	4c5bbf64f9	fixed : frame concatenation without checksum	2016-07-28 20:30:25 +02:00
Yann Collet	60ba31c570	zbuff uses ZSTD_compressEnd()	2016-07-28 19:55:09 +02:00
Yann Collet	16e73033ad	introduced stage zbf_end	2016-07-28 16:32:34 +02:00
Yann Collet	62470b4bab	Changed ZSTD_compressEnd()	2016-07-28 15:29:08 +02:00
Yann Collet	e7bf9156d1	Clarified API comments, from suggestions by ‎Bryan O'Sullivan‎	2016-07-28 05:00:57 +02:00
Yann Collet	d469a98c01	Clarified API comments, from suggestions by ‎Bryan O'Sullivan‎	2016-07-28 04:55:03 +02:00
Yann Collet	19c1002e46	applied ZSTD_compressContinueThenEnd()	2016-07-28 01:25:46 +02:00
Yann Collet	5b56739b63	created ZSTD_compressContinueThenEnd()	2016-07-28 01:17:22 +02:00
Yann Collet	c991cc1828	new frame end, 32-bits checksums	2016-07-28 00:55:43 +02:00
Yann Collet	d4180cad9c	minor code refactoring	2016-07-27 21:21:36 +02:00
Yann Collet	731ef16fc1	minor code style refactoring	2016-07-27 21:05:12 +02:00
Yann Collet	4b9ca0a6b5	minor example variation	2016-07-27 19:53:19 +02:00
Yann Collet	4110534886	ZSTD_maxCLevel() is promoted to "stable" API (#254, by @FrancescAlted)	2016-07-27 15:09:11 +02:00
Yann Collet	55a8bea0b5	fixed dictionary generation	2016-07-27 14:48:47 +02:00
Yann Collet	c154d9d6a2	better support for large dictionaries (> 128 KB)	2016-07-27 14:37:00 +02:00
Yann Collet	07626dfa51	improved dictbuilder notifications on selectivity	2016-07-27 13:28:46 +02:00
Yann Collet	f796f7ab45	removed fastscan mode	2016-07-27 12:53:54 +02:00
Yann Collet	dd25a27702	added tutorial warning messages for dictBuilder	2016-07-27 12:43:09 +02:00
inikep	003c7a8568	optimal parser: removed ZSTD_REP_INIT	2016-07-27 11:07:13 +02:00
Yann Collet	04cdd8660d	Merge pull request #262 from ebiggers/misc_updates ... Miscellaneous updates	2016-07-27 01:25:45 +02:00
Eric Biggers	0a55e7a0bb	ZSTD_decompressFrame(): use remainingSize instead of iend - ip ... Same behavior, but no need to have redundant variables.	2016-07-26 13:22:27 -07:00
Eric Biggers	aa6c70bf60	ZSTD_decompressFrame(): pass up error code from ZSTD_decodeFrameHeader()	2016-07-26 13:22:27 -07:00
Eric Biggers	e4d0265ea9	Replace remaining references to "direct mode" with "single segment mode"	2016-07-26 13:22:27 -07:00
Yann Collet	d50f9db3ea	Improved speed on clang and gcc -O2, thanks to @ebiggers ! (#263)	2016-07-26 21:30:35 +02:00
Yann Collet	7adc2328a3	fixed --test on zero-length files, reported by @amnilsson	2016-07-26 15:49:24 +02:00
inikep	4178f5c289	fixed gcc warning: always_inline function might not be inlinable	2016-07-25 21:17:45 +02:00
inikep	fca90f8f60	legacy decoder for v0.7 format	2016-07-25 17:49:08 +02:00
Yann Collet	cbc5e9dc19	fixes oob read	2016-07-24 18:02:04 +02:00
Yann Collet	38b75ddeb2	removed special case all-1 huffman distribution	2016-07-24 15:35:59 +02:00
Yann Collet	7ed5e33b89	minor comment changes	2016-07-24 14:26:11 +02:00
Yann Collet	10b9c13d07	fixed doc on cLevel default, reported by Oliver Lange	2016-07-24 01:21:53 +02:00
Yann Collet	f8e7b5363f	unified encoding types	2016-07-23 16:31:49 +02:00
Yann Collet	571a59034a	changed enccoding type order : raw, rle, compressed, repeat-stats	2016-07-23 15:52:05 +02:00
Yann Collet	c2e1a68d81	changed streamNb order to 1-4-4-4	2016-07-22 17:30:52 +02:00
Yann Collet	772d912c2f	more complete support for literals repeat mode	2016-07-22 15:04:25 +02:00
Yann Collet	9f2d82d4a4	fixed : big-endian decoding	2016-07-22 14:37:10 +02:00
Yann Collet	32faf6c8e7	fixed conversion warnings	2016-07-22 14:37:09 +02:00
Yann Collet	5e45a5fbb3	force loop-align to 32 for zstd_decompress	2016-07-22 14:37:09 +02:00
Yann Collet	5288ac0cb7	changed filed order	2016-07-22 14:37:09 +02:00
Yann Collet	198e6aac44	Literals header fields use little endian convention	2016-07-22 14:37:09 +02:00
Yann Collet	6fa05a2371	cBlockSize uses little-endian convention	2016-07-22 14:37:09 +02:00
Yann Collet	7bf72bbf5e	update header to v0.8	2016-07-22 14:37:09 +02:00
Yann Collet	5894ea8d01	updated cLevels	2016-07-22 14:36:46 +02:00
Yann Collet	d5c5a77990	minor comments clarifications	2016-07-20 13:35:14 +02:00
Yann Collet	572b817be3	Merge pull request #253 from gymdis/heapmode_off_legacy_fix ... Fix compile issue with ZSTD_LEGACY_SUPPORT=1 and ZSTD_HEAPMODE=0	2016-07-19 13:52:03 +02:00
Christopher Bergqvist	780a9fa857	Fix compile issue with ZSTD_LEGACY_SUPPORT=1 and ZSTD_HEAPMODE=0	2016-07-19 13:25:38 +02:00
Yann Collet	cf05b9d477	ZSTD_getBlockSizeMax()	2016-07-18 16:52:10 +02:00
Yann Collet	16aa38b0e0	minor doc clarifications	2016-07-18 03:52:47 +02:00
Yann Collet	85f3919960	moved zstd.h to /lib	2016-07-17 20:42:21 +02:00
Yann Collet	9375590462	update version to v0.7.5	2016-07-17 16:44:18 +02:00
Yann Collet	e557fd5e92	minor compression level corrections	2016-07-17 16:21:37 +02:00
Yann Collet	d54b2d23b4	minor static assert for 32/64 bits system. Suggested by @ebiggers	2016-07-17 15:53:18 +02:00
Yann Collet	972e5806ee	fixed : premature frame end on zero-sized raw block - reported by @ebiggers	2016-07-17 15:39:24 +02:00
luben karavelov	10f999f856	Add legacy support for the low-level streaming API	2016-07-17 01:03:26 +02:00
Yann Collet	6cacd34d44	minor formatting changes	2016-07-15 17:58:13 +02:00
Yann Collet	f6ff53cd4e	implemented dictID reserved ranges	2016-07-15 17:03:38 +02:00
Yann Collet	98c8884999	added target zstd in root Makefile	2016-07-15 16:12:38 +02:00
Yann Collet	961b6a0e34	ZSTD_compressBlock() limits block size depending on windowLog parameter	2016-07-15 11:58:49 +02:00
Yann Collet	227cc39e15	improved efficiency for large messages with small dictionaries	2016-07-15 11:27:09 +02:00
Yann Collet	ea2ecdc315	fixed issue with small dictionary	2016-07-14 23:27:31 +02:00
Yann Collet	e9ed5cdc94	fixed minor coverity warning	2016-07-14 21:02:57 +02:00
Yann Collet	b23e1ce319	removed debugging traces	2016-07-14 17:46:38 +02:00
Yann Collet	17508f1a16	fixed a few minor coverity warnings	2016-07-14 17:18:20 +02:00
Yann Collet	8847238cac	simplified ZSTD_estimateCCtxSize()	2016-07-14 17:05:38 +02:00
Yann Collet	69c2cdb45c	fixed conversion warning	2016-07-14 16:52:45 +02:00
Yann Collet	5e80dd3261	fixed minor coverity warnings	2016-07-13 19:21:57 +02:00
Yann Collet	3c174f4da9	fixed minor coverity warning	2016-07-13 17:25:53 +02:00
Yann Collet	2b1a3638e6	changed macro name to ZSTDCLI_CLEVEL_DEFAULT	2016-07-13 15:16:00 +02:00
Yann Collet	3c242e79d3	updated compression levels table	2016-07-13 14:56:24 +02:00
Yann Collet	fbc69f8649	changed for #245	2016-07-13 13:52:58 +02:00
Yann Collet	eed2081e55	fixed conversion warning	2016-07-12 15:11:40 +02:00
Yann Collet	a43a854cdb	updated paramgrill	2016-07-12 13:42:10 +02:00
Yann Collet	73d74a05b9	fixed dfast strategy	2016-07-12 13:03:48 +02:00
Yann Collet	45dc35628c	first version of doubleFast	2016-07-12 09:47:31 +02:00
Yann Collet	d158c35e9f	added ZSTD_estimateDCtxSize()	2016-07-11 13:46:25 +02:00
Yann Collet	8e0ee681b8	added ZSTD_sizeofDCtx()	2016-07-11 13:09:52 +02:00
Yann Collet	3ae543ce75	added ZSTD_estimateCCtxSize()	2016-07-11 03:12:17 +02:00
Yann Collet	25c506601c	promote ZSTD_getDecompressedSize() to stable API	2016-07-10 01:46:18 +02:00
Yann Collet	3b6ae77e15	comment clarification	2016-07-08 23:42:22 +02:00
Yann Collet	722e14bb65	fixed compilation error in decompression module	2016-07-08 19:22:16 +02:00
Yann Collet	bd10607063	updated spec	2016-07-08 19:16:57 +02:00
Yann Collet	c5fb5b7fcd	support offset > 128 MB	2016-07-08 13:13:37 +02:00
Yann Collet	ed3845d3fa	introduced ZSTD_WINDOWLOG_MAX_32 (#239), suggested by @GregSlazinski	2016-07-08 12:57:10 +02:00
Yann Collet	26f681451f	updated doc	2016-07-08 11:45:08 +02:00
Yann Collet	19c27d27f1	simplified legacy functions, no longer need magic number	2016-07-07 14:40:13 +02:00
Yann Collet	e72efeb0a1	removed "error_public.h" dependency from "zstd.h"	2016-07-07 14:17:40 +02:00
Yann Collet	974f52fc5d	Added "dictionary decompression" example	2016-07-07 14:08:00 +02:00
Yann Collet	e09d38e921	removed mem.h dependency from zbuff.h (experimental section)	2016-07-07 13:17:37 +02:00
Yann Collet	f323bf7d32	added : ZSTD_getDecompressedSize()	2016-07-07 13:14:21 +02:00
Yann Collet	52c04fe58f	removed mem.h dependency from zstd.h (experimental section)	2016-07-07 11:53:18 +02:00
Yann Collet	f246cf5423	ZSTD_decompress_usingDDict() compatible with Legacy mode	2016-07-06 20:32:27 +02:00
Yann Collet	29652e2618	sample set limitation closer to 2 GB	2016-07-06 16:25:46 +02:00
Yann Collet	99b045b70a	dictBuilder protection vs huge sample sets (>2 GB)	2016-07-06 16:12:38 +02:00
Yann Collet	445d49d898	fixed conversion warning	2016-07-06 13:27:22 +02:00
Yann Collet	a295b3170f	fixed conversion warning	2016-07-06 13:13:12 +02:00
Yann Collet	517e1ba623	fixed dictBuilder issue with HC levels. Reported by Bartosz Taudul.	2016-07-06 12:35:09 +02:00
Yann Collet	fe07eaa972	simplified ZSTD_decodeSequence()	2016-07-06 02:25:44 +02:00
Yann Collet	9ca73364e6	updated spec	2016-07-05 10:53:38 +02:00
Yann Collet	f9cac7a734	Added GNU separator --, to specifies that all following arguments are necessary file names (and not commands). Suggested by @chipturner (#230)	2016-07-04 18:18:24 +02:00
Yann Collet	23f05ccc6b	updated specifications	2016-07-04 16:13:11 +02:00
Yann Collet	d916c908e0	updated doc	2016-07-04 00:42:58 +02:00
Yann Collet	698cb63305	Updated specifications	2016-07-03 18:49:35 +02:00
Yann Collet	d57dffbe76	ZSTD_storeSeq takes an U32 as offset type	2016-07-03 01:48:26 +02:00
Yann Collet	302ff036f6	simplified repcodes for lazy_extDict	2016-07-03 01:28:16 +02:00
Yann Collet	9634f67107	fix lazy parser	2016-07-03 01:23:58 +02:00
Yann Collet	92d75667e4	fix for fast mode	2016-07-03 01:10:53 +02:00
Yann Collet	5e734ad09b	revert fix	2016-07-02 23:55:34 +02:00
Yann Collet	0d5bf8f06f	fixed risk of segfault on very large files (multiple GB)	2016-07-02 21:39:47 +02:00
Yann Collet	2fa9904844	update specification and comments	2016-07-01 20:55:28 +02:00
Yann Collet	c093208ab8	fix : potential leak (#229)	2016-06-30 14:07:30 +02:00
Yann Collet	6c6e1751f6	use ZSTD_getParams() to simplify code	2016-06-27 15:28:45 +02:00
Yann Collet	3d2cd7f816	Introduced ZSTD_getParams() ... bench now uses ZSTD_createCDict_advanced()	2016-06-27 15:12:26 +02:00
Yann Collet	529d9c7dee	updated version to v0.7.2	2016-06-27 10:03:10 +02:00
Yann Collet	d4f4e58ee1	fixed ZSTD_decompressBlock() using multiple blocks	2016-06-27 01:31:35 +02:00
Yann Collet	63b5e7a2ea	Improved comments	2016-06-26 17:42:15 +02:00
Yann Collet	3755eb8fea	fixed strict-aliasing warning on gcc6	2016-06-22 13:15:53 +02:00
Yann Collet	23042929da	Fixed : dictBuilder fails if first sample is too small	2016-06-22 11:05:34 +02:00
Yann Collet	391a128794	fix : segfault in command line during automatic overwrite protection mode	2016-06-21 17:06:25 +02:00
Yann Collet	bda68c253b	refactored ZBUFF_compressEnd() for better maintainability	2016-06-21 15:18:11 +02:00
Yann Collet	aa29226b7c	fix : ZBUFF_compressEnd() gives right amount remaining to flush, including future epilogue	2016-06-21 14:04:57 +02:00
Yann Collet	f15c1cb00c	Fixed : ZBUFF_compressEnd() called multiple times with too small dst buffer (#206)	2016-06-21 13:11:48 +02:00
Yann Collet	a49e066b26	clarified comments on ZSTD_compressContinue()	2016-06-21 11:54:03 +02:00
Yann Collet	d4f38d0dcd	updated library to v0.7.1	2016-06-21 10:15:43 +02:00
Yann Collet	22d76322ce	minor refactor	2016-06-21 08:01:51 +02:00
Yann Collet	a436a529bc	minor : fast_extDict does no longer skip first byte	2016-06-20 23:34:04 +02:00
Yann Collet	4623d11571	new correction, less extreme replacement value	2016-06-20 19:15:37 +02:00
Yann Collet	5477cc25f7	fixed corruption error related to inter-blocks rep-offset	2016-06-20 18:31:25 +02:00
Yann Collet	e4811ba761	Modified : ZSTD_createDDict() accepts dictionary < 8 bytes in pure content mode (reported by @chipturner)	2016-06-19 23:06:54 +02:00
Yann Collet	06d9a73b48	minor refactor, using WILDCOPY_OVERLENGTH macro instead of hard-coded 8	2016-06-19 14:27:21 +02:00
Yann Collet	19cab46f2f	Joined seqStore initialization at dispatch point	2016-06-17 12:54:52 +02:00
Yann Collet	510cff3570	minor comment change	2016-06-16 16:39:55 +02:00
Yann Collet	4948f270b3	make room for reserved "information bit" in frame header	2016-06-16 15:38:51 +02:00
Yann Collet	23ba41533a	Fixed zstd_opt encoding error with repeat-offsets	2016-06-16 13:20:46 +02:00
Yann Collet	80d033fb43	fixed ptr arithmetic warning	2016-06-16 01:41:50 +02:00
Yann Collet	ad39b7a718	zdict stores standard rep-offset. ... It can use custom ones, but the proper formula and impact on statistics is not done yet.	2016-06-16 01:14:41 +02:00
Yann Collet	736d419289	strengthened dict loading on decompresson side	2016-06-16 01:05:04 +02:00
Yann Collet	8e36a9c169	decoder restores repOffsets from dictionary	2016-06-16 01:05:04 +02:00
Yann Collet	52a0622beb	RepsCodes are saved into Dict ... (uncomplete : need decompression to regenerate them)	2016-06-16 01:05:04 +02:00
Yann Collet	efd0b4993a	fixed fuzzer error (inter-block repeated offsets)	2016-06-16 00:53:56 +02:00
Yann Collet	9b998e4d08	Fixed decompression of literals in dictionary mode	2016-06-15 23:11:20 +02:00
Yann Collet	d059092897	fixed conversion warnings	2016-06-14 15:34:24 +02:00
Yann Collet	45c03c564f	fixed corruption with inter-blocks repeated offsets	2016-06-14 13:46:11 +02:00
Yann Collet	4266c0a2fd	adding inter-blocks rep-offsets	2016-06-14 01:49:25 +02:00
Yann Collet	43dfe01919	Check repIndex for validity	2016-06-13 21:43:06 +02:00
Yann Collet	18c8f79f3e	fixed gcc warning on uninitialized structure variable	2016-06-12 22:51:52 +02:00
Yann Collet	cd98f93cff	Fixed decompression issue with invalid data	2016-06-11 23:26:22 +02:00
Yann Collet	37fece22e8	enable repeat-entropic-stats mode	2016-06-11 02:52:42 +02:00
Yann Collet	d60a5bf900	Literal decompression builds Huffman tables within shared space (for later re-use)	2016-06-11 02:35:31 +02:00
Yann Collet	237ad4beb3	Added single-stream decompression variant using external DTable	2016-06-11 01:46:03 +02:00
Yann Collet	289bbd52e5	Updated huff0	2016-06-11 01:31:54 +02:00
Yann Collet	1869f7966e	Merge pull request #205 from inikep/dev ... legacy decoder for v0.6	2016-06-10 17:13:07 +02:00
Yann Collet	0974f681a4	completed .gitignore	2016-06-10 14:44:16 +02:00
Yann Collet	9dd12742f3	litBlockType_t is an enum	2016-06-10 00:12:26 +02:00
inikep	4923222412	fixed warnings from Travis	2016-06-09 20:03:30 +02:00
inikep	4000945a1d	project updated for legacy decoder zstd_v06.c	2016-06-09 18:12:06 +02:00
inikep	bf853d5510	added legacy decoder for v0.6 format	2016-06-09 17:59:18 +02:00
Yann Collet	662a541431	updated huff0 - now generates a common HUF_DTable type for all decoding tables	2016-06-08 11:11:02 +02:00
Yann Collet	302fb53a76	Removed ZSTD_*_usingPrepared?Ctx() declaration from public space	2016-06-07 12:16:49 +02:00
Yann Collet	81e13ef7cf	first implementation of the new dictionary API (untested)	2016-06-07 00:51:51 +02:00
Yann Collet	9d504ae85b	Added decoding of RLE blocks	2016-06-06 19:52:35 +02:00
Yann Collet	2cc72f1fd3	fixed initialization issue in bench	2016-06-06 17:50:07 +02:00
Yann Collet	e3d529403d	fixed initialization mismatch in ZSTD_copyCCtx()	2016-06-06 11:07:33 +02:00
Yann Collet	142acbdea7	fixed minor visual conversion warning	2016-06-06 00:46:56 +02:00
Yann Collet	673f0d7cdc	new frame format, allowing custom window size	2016-06-06 00:26:38 +02:00
Yann Collet	89703d20fb	reduced dependencies	2016-06-05 01:50:33 +02:00
Yann Collet	51778b7cca	updated README following merging of *_static.h	2016-06-05 01:38:10 +02:00
Yann Collet	a91ca620cf	removed HUF_readStats() from public space	2016-06-05 01:33:55 +02:00
Yann Collet	d0e2cd15cb	Merged fse_static into fse.h . Now requires FSE_STATIC_LINKING_ONLY macro.	2016-06-05 00:58:01 +02:00
Yann Collet	130fe11394	merged huf_static.h into huf.h . Requires HUF_STATIC_LINKING_ONLY macro.	2016-06-05 00:42:28 +02:00
Yann Collet	dc048d18d3	minor comment (detailing an #include motivation)	2016-06-05 00:32:23 +02:00
Yann Collet	49bb0041af	removed ZSTD_highbit() from zstd_internal.h, as it is only used by zstd_compress.c	2016-06-04 20:17:38 +02:00
Yann Collet	d3b7f8d21f	Merged zstd_static.h into zstd.h . Now requires ZSTD_STATIC_LINKING_ONLY macro	2016-06-04 19:47:02 +02:00
Yann Collet	ac110a1f21	Removed ZBUFF internal util function from public area	2016-06-04 19:16:49 +02:00
Yann Collet	5347aee8f7	merged zbuff_static.h into zbuff.h . Now requires ZBUFF_STATIC_LINKING_ONLY macro	2016-06-04 19:12:48 +02:00
Yann Collet	e69b8ccceb	merged zdict_static.h into zdict.h. Now requires ZDICT_STATIC_LINKING_ONLY macro.	2016-06-04 18:56:23 +02:00
Yann Collet	198d127b35	minor comment change (unfinished description of new header format)	2016-06-04 18:40:55 +02:00
Yann Collet	f4f5affdf7	restore ZBUFF full-block-size, for better performance on small input	2016-06-03 23:09:28 +02:00
Yann Collet	ab7b6f1ece	Merge pull request #198 from inikep/dev070 ... Dev070	2016-06-03 21:37:49 +02:00
inikep	3640396b1a	fixed: deallocation of structures in case of error in ZBUFF_createCCtx and ZBUFF_createDCtx	2016-06-03 16:36:50 +02:00
Yann Collet	fe48775868	minor decoder code refactoring	2016-06-03 15:41:51 +02:00
inikep	2a74609b90	zlibWrapper: ZWRAP_createCCtx and ZWRAP_freeCCtx use custom memory allocation functions	2016-06-03 14:53:51 +02:00
inikep	3763c77f6b	defaultCustomNULL replaced with defaultCustomMem	2016-06-03 13:28:20 +02:00
inikep	36fac00149	removed calloc calls from lib/	2016-06-03 13:23:04 +02:00
inikep	db2f540414	added defaultCustomNULL	2016-06-03 12:56:56 +02:00
inikep	b74a468fad	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev070' into dev070	2016-06-02 22:09:09 +02:00
Yann Collet	923938edde	Added -Wdeclaration-after-statement compilation flag	2016-06-02 17:56:00 +02:00
inikep	ff9114aee3	zlibWrapper: added support for custom memory allocation functions	2016-06-02 16:52:36 +02:00
inikep	c4807f4d2f	default custom allocation functions moved to zstd_internal.h	2016-06-02 15:11:39 +02:00
inikep	2866951558	opaque parameter for custom memory allocation functions	2016-06-02 13:04:18 +02:00
inikep	9242816b56	fparamsPtr->windowLog==0 means that a frame is skippable	2016-06-01 18:47:04 +02:00
Yann Collet	70d1301d6e	Changed ZSTD_adjustCParams() prototype ... `ZSTD_adjustCParams()` is now automatically invoked at the end of `ZSTD_getCParams()`	2016-06-01 18:45:34 +02:00
Yann Collet	83c3f4427c	upgraded zbufftest to also test advanced frame parameters no/checksum no/dictID	2016-06-01 17:44:53 +02:00
inikep	13f42d9085	VS2010 project: reverted zstdlib.rc	2016-06-01 14:44:31 +02:00
inikep	5c2771710d	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev070' into dev070 ... # Conflicts: # .gitignore # lib/decompress/zstd_decompress.c # programs/zbufftest.c	2016-06-01 09:16:11 +02:00
Yann Collet	202844ebd0	fixed zbufftest :	2016-06-01 00:44:36 +02:00
Yann Collet	8e3a36a6db	decompression validates frame content checksum	2016-06-01 00:18:28 +02:00
inikep	a6b942018d	Merge remote-tracking branch 'refs/remotes/origin/dev' into dev070 ... # Conflicts: # .travis.yml # Makefile # lib/common/zstd_static.h # programs/Makefile # projects/VS2008/zstd/zstd.vcproj # projects/VS2008/zstdlib/zstdlib.vcproj # projects/cmake/lib/CMakeLists.txt # projects/cmake/programs/CMakeLists.txt	2016-06-01 00:07:09 +02:00
Yann Collet	f2a3b6e7b4	added : frame content checksum	2016-05-31 22:23:45 +02:00
inikep	43aa9fe8b3	fixed skippable frame	2016-05-31 19:36:51 +02:00
inikep	f772bf54a5	support for skippable frames	2016-05-31 12:43:46 +02:00
Giuseppe Ottaviano	370b751e24	Expose function to add entropy tables to pre-built dictionary. ... In some cases a custom dictionary building algorithm tailored for a specific input can be more effective than the one produced by `ZDICT_trainFromBuffer`, but with the current API it's not possible encode the entropy tables into the custom-built dictionary. This commit extracts the logic to add entropy tables to a dictionary from `ZDICT_trainFromBuffer` and exposes it as a function `ZDICT_addEntropyTablesFromBuffer`.	2016-05-30 19:50:09 -07:00
Yann Collet	290aaa7521	Added : ability to manually select the dictionary ID of a newly created dictionary	2016-05-30 21:18:52 +02:00
Yann Collet	30009521d7	fuzzer tests dictBuilder. ... Added : ability to not store dictID during compression; decompression doesn't check dictID then	2016-05-30 16:17:33 +02:00
Yann Collet	c0a9bf3c2e	minor code refactoring	2016-05-30 04:48:32 +02:00
Yann Collet	c46fb924df	added dictionary ID (incomplete)	2016-05-29 05:01:04 +02:00
Yann Collet	f51e0660f4	Simplified list of *.c files	2016-05-29 01:39:19 +02:00
Yann Collet	0c5e8b17ad	moved xxhash to lib/common	2016-05-29 01:06:30 +02:00
inikep	957823f56f	zstdcli: -r (operate recursively on directories) works with dictBuilder and compression	2016-05-25 15:30:55 +02:00
inikep	e75909ef93	ZBUFF_createCCtx_advanced moved to zbuff_static.h	2016-05-25 11:31:16 +02:00
inikep	02c244bf78	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev	2016-05-24 17:15:04 +02:00
inikep	fb5df613cf	zstd_stats.h included only in debug mode	2016-05-24 15:36:37 +02:00
inikep	71dd67cf57	fullbench: tests for ZBUFF_createCCtx_advanced and ZBUFF_createDCtx_advanced	2016-05-24 10:57:14 +02:00
inikep	13ba880b49	added ZBUFF_createCCtx_advanced and ZBUFF_createDCtx_advanced	2016-05-23 17:04:23 +02:00
inikep	107e243195	added ZSTD_createDCtx_advanced	2016-05-23 16:24:52 +02:00
inikep	50e82c015d	added ZSTD_createCCtx_advanced	2016-05-23 15:49:09 +02:00
inikep	35b891c5b0	zstd_stats.h updated to v0.6	2016-05-20 19:42:20 +02:00
Yann Collet	f22a0d653d	huff0 dynamic reduction	2016-05-20 14:36:36 +02:00
Yann Collet	45f84ab7cc	all programs use ZSTD_VERSION_STRING from zstd.h to display their version	2016-05-20 12:34:40 +02:00
Yann Collet	de4d4da52d	changed version to 0.6.2 ... removed make-controlled version (now used from zstd.h) removed std=c99 by default added : c90/gno90/c99/gnu99/c11 tests	2016-05-20 12:20:12 +02:00
Yann Collet	7591a7f224	minor refactoring	2016-05-20 11:44:43 +02:00
Yann Collet	236d94fa9a	reverted default compression level to 1	2016-05-18 12:06:33 +02:00
Yann Collet	78aa05228a	fixed Visual warning in debug mode	2016-05-17 14:30:19 +02:00
Yann Collet	c8e9c4046f	Merge branch 'dev' of github.com:Cyan4973/zstd into dev	2016-05-17 12:02:35 +02:00
Yann Collet	ea4daa308d	added ZSTDv05_getFrameParams() to "zstd_v05.h"	2016-05-17 12:01:55 +02:00
klemens	268775dae8	spelling-fixes, as of lintian.debian.org	2016-05-16 22:23:24 +02:00
Yann Collet	d9b3cca4d3	restore openVMS support (detected by @thatsafunnyname)	2016-05-13 12:43:36 +02:00
inikep	63ecd747de	added common/entropy_common.c	2016-05-13 11:27:56 +02:00
Yann Collet	ccd6e86d45	improved decoding speed by inlining x_isError() functions	2016-05-12 15:55:26 +02:00
Yann Collet	16871680e3	minor refactor : more accurate variable scope	2016-05-12 14:27:00 +02:00
Yann Collet	b19d5d76ec	restored inline for better decompression speed	2016-05-12 14:06:31 +02:00
Yann Collet	1ceb5a9295	restored DStream init ... due to strange speed regressions and warnins on gcc 5.3	2016-05-12 13:50:13 +02:00
Yann Collet	ce27e85209	fixed clang conversion warning	2016-05-11 18:55:27 +02:00
Yann Collet	1032fbe714	update FSE library	2016-05-11 18:30:24 +02:00
Yann Collet	249ae0ca99	better big-endian / little-endian r/w support	2016-05-11 16:38:18 +02:00
Yann Collet	3329d98df9	fixed minor asan warning in legacy decoder	2016-05-11 11:07:54 +02:00
Yann Collet	c75e4c25e8	Added : ZSTD_getErrorString(), to get error string from error enum (#168)	2016-05-10 17:47:11 +02:00
Yann Collet	b3a9938d0c	Merge pull request #182 from inikep/dev ... support for Visual Studio 2012, 2013, 2015	2016-05-10 17:18:12 +02:00
Yann Collet	bd39d54576	zbuff supports byte-by-byte decompression scenarios	2016-05-10 14:14:19 +02:00
inikep	ed9a08538c	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev ... # Conflicts: # lib/common/util.h # programs/paramgrill.c # visual/2013/fullbench/fullbench.vcxproj.filters # visual/2013/fuzzer/fuzzer.vcxproj.filters	2016-05-10 13:20:01 +02:00
Yann Collet	918b934a26	minor comment update	2016-05-10 07:00:07 +02:00
inikep	349fcf7ea2	fixed Linux compilation	2016-05-09 18:04:07 +02:00
inikep	aaaf923a17	common/util.h moved to programs/util.h ... decompress/fse_decompress.c moved to common/	2016-05-09 16:19:25 +02:00
inikep	d44ec48b1c	detection of snprintf for Visual <= 2012	2016-05-09 14:37:41 +02:00
Yann Collet	59b6ba7677	Merge pull request #180 from thatsafunnyname/patch-1 ... No stdint.h on OpenVMS.	2016-05-09 11:56:42 +02:00
Yann Collet	4ff73a2eff	added : midipix support (#181)	2016-05-09 11:44:00 +02:00
Yann Collet	d275236e6a	Fixed Windows API, as detected by @KrzysFR (#171)	2016-05-09 08:19:27 +02:00
Yann Collet	f6ca09b5ff	Reduced console display on loading lots of files with zstd --train. Reported by @KrzysFR, see #177	2016-05-09 04:44:45 +02:00
Yann Collet	e1a69a88c0	Expose Advanced Streaming API for v0.5.x legacy	2016-05-08 09:13:40 +02:00
Yann Collet	a5dacdf551	updated fse (minor refactor)	2016-05-06 16:55:27 +02:00
Yann Collet	18dedece91	Support legacy ZSTD_decompress_usingDict() (starting v0.5+)	2016-05-06 16:43:23 +02:00
Yann Collet	8283a2f0aa	Fixed decoding corruption error with quad-symbols huffman on legacy decoders	2016-05-06 01:51:31 +02:00
inikep	13c8424ea0	code cleaning	2016-05-05 13:58:56 +02:00
inikep	0bd0faec32	fixed compatibility issues	2016-05-05 13:10:57 +02:00
Yann Collet	6d1d25299a	fixed Huff0 quad-symbols decoder (#173)	2016-05-05 12:41:36 +02:00
inikep	9c22e57bfb	Compiler Options moved to util.h	2016-05-05 11:53:42 +02:00
Yann Collet	ddb8ebd5b3	Stricter gnu90 tests	2016-05-05 04:59:53 +02:00
Yann Collet	3f8ed50bf1	Added gnu90 test ... work-around : disabled automatic selection of huff0 quad-decoder (see #173)	2016-05-05 03:01:13 +02:00
inikep	3163403855	UTIL_prepareFileList	2016-05-05 00:25:38 +02:00
thatsafunnyname	b0e5427d1a	No stdint.h on OpenVMS. ... No stdint.h on OpenVMS, it is inttypes.h instead, some projects use HAVE_STDINT_H.	2016-05-04 15:43:42 +01:00
Yann Collet	6f9c056662	Fixed BMI behavior on x86 and x32 targets, reported by @tycho (#178)	2016-05-01 10:26:30 +02:00
inikep	55d047aa92	getTotalFileSize moved to common/util.h	2016-04-28 16:50:13 +02:00
inikep	d5ff2c3d9a	ordering of #include	2016-04-28 14:40:45 +02:00
inikep	83c76b4594	bench.c: time functions moved to common/util.h	2016-04-28 13:16:01 +02:00
inikep	69fcd7c0ae	getFileSize moved to common/util.h	2016-04-28 12:23:33 +02:00
inikep	1007a1fe3d	get rid of some Visual C++ warnings	2016-04-25 15:23:09 +02:00
inikep	6cb083fe3d	fixed 64-bit compilation with Visual C++	2016-04-25 14:42:15 +02:00
inikep	ea4ee3eee0	added compatibility with Visual C++ 2012	2016-04-25 13:09:06 +02:00
inikep	a8138fd767	updated lib/README.md part 2	2016-04-25 11:36:44 +02:00
inikep	49794316c2	updated lib/README.md	2016-04-25 11:31:28 +02:00
inikep	f066816998	fix for g++ compilation	2016-04-22 18:54:05 +02:00
inikep	db3964382c	introduced ZSTD_NODECOMPRESS to link only compressor	2016-04-22 18:22:30 +02:00
inikep	a1febea01d	Error functions moved to common/zstd_common.c	2016-04-22 17:14:25 +02:00
inikep	3c7c3527d0	introduced ZSTD_NOCOMPRESS to generate decompressor only	2016-04-22 13:59:05 +02:00
inikep	23a0889301	separation of lib/ into common/, compress/, decompress/, dictBuilder/, legacy/	2016-04-22 12:43:18 +02:00
inikep	4699b3be03	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev	2016-04-21 12:30:58 +02:00
inikep	38654988f3	minor speed improvements 2 ... bench.c: block size has to be bigger than 32 bytes zstdcli.c: support for e.g. -B16k -B16m	2016-04-21 12:18:47 +02:00
inikep	ef51941822	zst_opt.h: minor compression speed improvement	2016-04-21 11:08:43 +02:00
Yann Collet	e02808ffc5	bumped lib version to 0.6.1 ... added ZSTD_VERSION_STRING	2016-04-20 22:46:16 +02:00
inikep	c5e1d295ee	bench.c: force at least one compression and decompression loop ... fix for -i0 with small files	2016-04-19 09:37:59 +02:00
inikep	2c5eeea635	introduced ZSTD_DEAFULT_CLEVEL for (compressionLevel<=0)	2016-04-15 13:44:46 +02:00
inikep	4b3c5eed99	removed ZSTD_compressBegin_targetSrcSize	2016-04-14 13:43:51 +02:00
inikep	c70c90e8a3	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev	2016-04-13 10:48:26 +02:00
Yann Collet	16748729c9	Updated DICT_MAGIC, since internal dictionary format has changed (length encoding)	2016-04-12 23:26:23 +02:00
Yann Collet	c5b6d9741a	Added : zbuff.c to library; reported by Yipei Zhang	2016-04-12 21:46:18 +02:00
Yann Collet	389648cc6d	Fixed : contentSize + copyCCtx()	2016-04-12 19:13:08 +02:00
Yann Collet	541dc7c7d4	Added : fuzzer test, checking contentLength value after copyCCtx() usage	2016-04-12 18:00:20 +02:00
Yann Collet	6236eba991	removed headerBuffer	2016-04-12 15:52:33 +02:00
Yann Collet	0085cd3138	created ZSTD_writeFrameHeader()	2016-04-12 14:14:10 +02:00
Yann Collet	abb5c653f1	minor speed improvement for ZSTD_copyCCtx()	2016-04-11 20:42:31 +02:00
Yann Collet	887e7da785	Fixed copyCCtx() on uninitialized cctx (reported by Oliver Lange)	2016-04-11 20:12:27 +02:00
inikep	8844e21dff	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev	2016-04-11 16:49:27 +02:00
inikep	f7d210b2e9	cache literal prices for ZSTD_btopt	2016-04-11 16:35:13 +02:00
Yann Collet	582933fde5	fixed round-trip corruption, reported by Oliver Lange	2016-04-11 16:25:56 +02:00
Yann Collet	125d81774f	fixed uninitialization error, reported by Maciej Adamczyk	2016-04-11 14:22:56 +02:00
Yann Collet	d673d4cf66	fixed uninitialization error, reported by Maciej Adamczyk	2016-04-11 14:10:23 +02:00
Yann Collet	ccbba296e1	Fix infinite loop with invalid data (fuzzer test)	2016-04-10 14:42:26 +02:00
Yann Collet	793c649ead	updated cLevels	2016-04-09 20:32:00 +02:00
Yann Collet	b144bc3e1c	Fixed uninitialized warning, reported by Maciej Adamczyk	2016-04-09 17:26:22 +02:00
Yann Collet	9990986bdc	fixed minor warnings when compiling with gcc -g (aka -std=gnu90)	2016-04-09 16:17:18 +02:00
Yann Collet	029267ab3f	Integrated Legacy v0.5.x decoder (provided by @inikep)	2016-04-09 09:42:27 +02:00
Yann Collet	7eff39f7ea	fixed decoding error	2016-04-09 01:51:36 +02:00
Yann Collet	ea63bb7b5e	converted fuzzer timer to clock_t	2016-04-08 15:25:32 +02:00
Yann Collet	78267d1abe	updated cLevel for block <= 256K	2016-04-08 12:36:19 +02:00
Yann Collet	72cc704514	Merge branch 'dev060' of github.com:Cyan4973/zstd into dev060	2016-04-08 02:02:31 +02:00
Yann Collet	0dbf2874ee	faster level 1 at 256 KB	2016-04-08 02:02:12 +02:00
Yann Collet	04a4fd638b	Merge pull request #164 from inikep/dev ... only repcode 0 for lazy, lazy2 and btlazy2 strategies	2016-04-07 20:46:13 +02:00
inikep	afe1f7994b	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev ... # Conflicts: # lib/zstd_compress.c	2016-04-07 19:50:03 +02:00
Yann Collet	9e8b09a7bd	Fixed memory initialization issue, reported by Maciej Adamczyk	2016-04-07 19:35:23 +02:00
inikep	2bf79f9bff	use extDict only when needed	2016-04-07 19:24:33 +02:00
inikep	64d7bcb0dc	only repcode 0 for lazy, lazy2 and btlazy2 strategies	2016-04-07 19:14:09 +02:00
inikep	a69640cca1	Revert "removed void ZSTD_compressBlock_greedy" ... This reverts commit 9296400a69.	2016-04-07 18:22:29 +02:00
Yann Collet	d7bee33caf	Fixed make install (reported by Oliver Lange)	2016-04-07 18:16:42 +02:00
Yann Collet	5d39357bc4	minor compression speed optimization	2016-04-07 17:19:00 +02:00
inikep	9296400a69	removed void ZSTD_compressBlock_greedy	2016-04-07 16:58:04 +02:00
Yann Collet	485371600a	Implemented default coding table for lengths and offset	2016-04-07 15:24:29 +02:00
Yann Collet	79459da14b	Merge pull request #163 from inikep/dev ... 3 repcodes and MINMATCH=3	2016-04-07 14:37:30 +02:00
inikep	81a345b0ce	minor speed improvements	2016-04-07 11:35:17 +02:00
inikep	e47f33f980	reordered code	2016-04-06 20:58:00 +02:00
inikep	2db1eb78e9	reordered functions	2016-04-06 17:14:19 +02:00
inikep	e5b6e9e551	fixed support for searchLength=4	2016-04-06 13:15:38 +02:00
inikep	75716851d4	MINMATCH=3 for lazy strategy	2016-04-06 12:34:42 +02:00
inikep	7bc19b6b3e	MINMATCH=3	2016-04-06 09:46:01 +02:00
inikep	f8a339d274	working ZSTD_compressBlock_opt_extDict_generic	2016-04-05 23:58:51 +02:00
inikep	83417bbbd2	code formatting	2016-04-05 21:30:03 +02:00
inikep	5ce00aee37	ZSTD_compressBlock_opt_extDict_generic in progress	2016-04-05 21:03:43 +02:00
inikep	faa8d8aae1	introduced ZSTD_compressBlock_greedy	2016-04-05 19:01:10 +02:00
inikep	908fcb384a	3 repcodes with ZSTD_compressBlock_lazy_extDict_generic	2016-04-05 18:16:38 +02:00
inikep	a4385aa23b	ZSTD_REP_NUM=3 only	2016-04-05 17:11:56 +02:00
inikep	d5359b2e55	Merge remote-tracking branch 'refs/remotes/origin/repcodes' into dev	2016-04-05 17:07:36 +02:00
inikep	ba73d1bb6b	support for ZSTD_REP_NUM=3	2016-04-05 10:30:05 +02:00
inikep	74e3a7ba4c	removed kSlotNew	2016-04-05 10:18:37 +02:00
inikep	b3532793a9	experiments with ZSTD_decodeSequence	2016-04-05 10:08:25 +02:00
Yann Collet	46bdb6267e	Merge pull request #161 from inikep/dev ... Dev	2016-04-05 09:59:59 +02:00
inikep	2ec2698ef9	working 4 reps	2016-04-05 09:34:13 +02:00
inikep	19140cd80e	zstd_stats.h moved to .debug/	2016-04-05 08:52:57 +02:00
inikep	d6f208badf	improved logging	2016-04-04 21:15:23 +02:00
inikep	78e52044de	Merge remote-tracking branch 'refs/remotes/origin/dev' into repcodes ... # Conflicts: # lib/zstd_compress.c # lib/zstd_decompress.c # lib/zstd_internal.h # lib/zstd_opt.h # programs/bench.c	2016-04-04 16:28:40 +02:00
inikep	cb70c8ab90	fixed -Werror,-Wconversion warning	2016-04-04 15:43:45 +02:00
inikep	0c7456c5b7	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev ... # Conflicts: # lib/zstd_compress.c # lib/zstd_static.h	2016-04-04 14:54:53 +02:00
Yann Collet	b59bf963a8	fixed Visual conversion warnings	2016-04-04 14:53:16 +02:00
inikep	227dd4bc74	changed price estimation for optimal parser	2016-04-04 14:34:02 +02:00
Yann Collet	8a57b928ba	changed contentLog -> chainLog	2016-04-04 13:49:18 +02:00
Yann Collet	1005fc1276	reduced memory usage of very hich compression levels in 32-bits mode	2016-04-04 13:28:28 +02:00
inikep	19bd48f516	Merge from Cyan4973/dev060	2016-04-04 12:10:00 +02:00
Yann Collet	1535414927	Fixed 32-bits windowLog limitation ... Fixed some macro errors in zbufftest	2016-04-04 04:22:53 +02:00
Yann Collet	5a88a70f97	fixed buffer sync error in zbuff	2016-04-03 20:24:25 +02:00
Yann Collet	ef36390e4a	refined exception for hashlog < min	2016-04-02 00:46:40 +02:00
Yann Collet	40358d01aa	Fix : rare bugs when hashLog < min in bt mode	2016-04-02 00:40:09 +02:00
Yann Collet	1d803d8deb	fixed zbuff error 775-3 (internal buffer management)	2016-04-01 18:57:41 +02:00
Yann Collet	efb18309ef	fixed minor visual 32-64bits conversion warning	2016-04-01 18:54:13 +02:00
Yann Collet	afa60e0f14	fixed decoding error, reported by Maciej Adamczyk	2016-04-01 15:49:41 +02:00
Yann Collet	27caf2afca	minor variable isolation	2016-04-01 15:48:48 +02:00
Yann Collet	9bb87e5995	fixed harmless comparison warning	2016-03-30 21:28:15 +02:00
Yann Collet	51d500425f	fixed paramgrill	2016-03-30 20:42:19 +02:00
Yann Collet	dd6466a211	fixed clang conversion warning	2016-03-30 20:06:26 +02:00
Yann Collet	3b71925c2d	separate params into compressionParams and frameParams	2016-03-30 19:48:05 +02:00
Yann Collet	21588e370b	changed validateParams() into checkParams() + adjustParams()	2016-03-30 16:50:44 +02:00
Yann Collet	b44be74244	Fixed zdict ... more dictionary compression tests	2016-03-26 20:52:14 +01:00
Yann Collet	74bd11954b	fixed ARM compatibility	2016-03-26 17:50:26 +01:00
Yann Collet	b9151406de	fixed 32-bits compatibility	2016-03-26 17:18:11 +01:00
inikep	97c88e716c	fix for -Wshorten-64-to-32 warning in ZSTD_statsUpdatePrices	2016-03-25 13:39:14 +01:00
Yann Collet	433a5cce7e	improved decompression speed (680)	2016-03-25 11:43:48 +01:00
inikep	5cc4efdaf8	created zstd_stats.h	2016-03-25 10:52:25 +01:00
Yann Collet	646693e3be	support for alternate offset (fusion)	2016-03-24 02:42:15 +01:00
Yann Collet	b21ce15efe	minor variables isolation	2016-03-24 02:02:32 +01:00
Yann Collet	7cbe79ab25	fixed write overflow found by fuzzer	2016-03-23 22:31:57 +01:00
Yann Collet	72d706a020	fixed crash at -O3 with customized block size (wrong alignment)	2016-03-23 20:44:12 +01:00
inikep	eaba91a161	added ZSTD_compressBegin_targetSrcSize	2016-03-23 20:30:26 +01:00
Yann Collet	862a85976f	Added BMI instructions	2016-03-23 18:45:23 +01:00
inikep	7adceef974	hashLog3 added to ZSTD_CCtx	2016-03-23 15:53:38 +01:00
Yann Collet	6cf45dac81	shared const BIT_mask	2016-03-23 14:41:54 +01:00
Yann Collet	3c017867de	minor optimization (DSpeed 665)	2016-03-23 14:09:51 +01:00
Yann Collet	2512597576	minor reordering (DSpeed 665)	2016-03-23 14:00:09 +01:00
Yann Collet	afab02098a	improved decoding speed (660)	2016-03-23 13:57:49 +01:00
inikep	472638c861	added mili_sleep and setHighPriority	2016-03-23 12:28:28 +01:00
Yann Collet	9c34df93b6	new decodeSequence, merging parts	2016-03-23 01:54:25 +01:00
Yann Collet	add08d6f61	minor variation - DSpeed at 640	2016-03-23 01:32:41 +01:00
Yann Collet	be391438ff	first working version with both encoder and decode alternate LL + ML coding scheme. ... decompression speed highly impacted	2016-03-22 23:19:28 +01:00
inikep	2872b6f01f	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev ... # Conflicts: # lib/zstd_compress.c # programs/bench.c	2016-03-22 14:38:34 +01:00
Yann Collet	fadda6c875	first prototype with ML encoding scheme (but not yet decoding scheme)	2016-03-22 12:14:26 +01:00
inikep	2b942881ac	improved 4reps in ZSTD_compressBlock_lazy_generic ... kSlotNew = 0	2016-03-22 11:56:22 +01:00
Yann Collet	1dcf930f72	Merge branch 'dev060' into alternateLLcode	2016-03-22 00:29:57 +01:00
inikep	f2fa0e1198	final repcodes for ZSTD_btopt	2016-03-21 20:14:03 +01:00
inikep	378be62a9e	better repcodes	2016-03-21 19:48:54 +01:00
inikep	fc1d7bd93b	changed update logic of repcodes	2016-03-21 19:02:16 +01:00
inikep	e7ec8f2b9b	improved ratio	2016-03-21 17:23:02 +01:00
Yann Collet	be4605d851	improved decompression speed	2016-03-21 14:29:26 +01:00
Yann Collet	e91477c171	minor variable isolation	2016-03-21 14:22:38 +01:00
Yann Collet	b0aec17a90	code simplification (but reduce decompression speed ...)	2016-03-21 13:24:16 +01:00
Yann Collet	6c62b7dfc8	support default LL distribution	2016-03-21 00:39:19 +01:00
Yann Collet	d64f435f63	handles litLength >= 65535	2016-03-21 00:07:42 +01:00
Yann Collet	597847a2ae	first emulation	2016-03-20 19:14:22 +01:00
Yann Collet	ecabfe3777	Improved index overflow protection	2016-03-20 16:20:06 +01:00
Yann Collet	5244738046	minor code refactoring	2016-03-20 16:00:00 +01:00
Yann Collet	de406eebcd	minor code refactor	2016-03-20 15:46:10 +01:00
Yann Collet	516ba88022	updated huff0	2016-03-20 05:40:39 +01:00
Yann Collet	a5c2c08c68	code refactoring	2016-03-20 01:09:18 +01:00
Yann Collet	70e4577179	minor code refactoring	2016-03-19 18:08:32 +01:00
Yann Collet	c6eea2b2f2	Improved speed for small files	2016-03-19 17:18:00 +01:00
Yann Collet	e74215e3c7	performance fix for small files	2016-03-19 16:09:09 +01:00
inikep	1269e2f7db	working kSlotNew for ZSTD_btopt	2016-03-19 15:32:21 +01:00
Yann Collet	bb604486f1	fixed clang warning	2016-03-19 15:18:42 +01:00
Yann Collet	37f3d1b2ab	minor refactor	2016-03-19 15:11:42 +01:00
Yann Collet	90a03a95a1	minor performance fix (decompression)	2016-03-19 14:47:31 +01:00
Yann Collet	01e5b95199	code refactor	2016-03-19 14:14:31 +01:00
Yann Collet	d1d210f3fb	minor code refactor	2016-03-19 12:12:07 +01:00
Yann Collet	4db09efde4	simplifications	2016-03-18 22:23:49 +01:00
Yann Collet	39c596ca58	minor simplifications	2016-03-18 21:40:56 +01:00
inikep	fade9b7ae3	ZSTD_btopt with 4 repcodes	2016-03-18 16:10:35 +01:00
inikep	1c1e6916e4	improved timer accuracy	2016-03-18 12:49:44 +01:00
Yann Collet	a910dc8238	minor code refactoring	2016-03-18 12:37:45 +01:00
inikep	5b15865413	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into repcodes ... # Conflicts: # lib/zstd_decompress.c # programs/bench.c	2016-03-18 11:14:58 +01:00
inikep	eca2ec0b3b	introduced ZSTD_REP_MOVE and ZSTD_REP_INIT	2016-03-18 11:03:43 +01:00
inikep	c56e32e782	4 repcodes when litLen==0	2016-03-17 19:53:38 +01:00
inikep	3e743052cd	kSlotNew	2016-03-17 17:24:17 +01:00
Yann Collet	3fd164e656	minor refactoring	2016-03-17 13:43:32 +01:00
Yann Collet	b40287aaef	Improved decompression speed (by @inikep)	2016-03-17 12:11:16 +01:00
inikep	7d1742825b	support for ZSTD_REP_NUM == 1 and ZSTD_REP_NUM == 4	2016-03-17 11:21:00 +01:00
inikep	5945308983	working 4 reps	2016-03-16 15:35:14 +01:00
Yann Collet	569b81adb1	changed BLOCKSIZE into ZSTD_BLOCKSIZE_MAX	2016-03-16 15:26:51 +01:00
inikep	62d694795d	ZSTD_compressBlock_lazy supports more reps	2016-03-16 12:57:07 +01:00
inikep	0ac4a551b9	ZSTD_REP_NUM removed from ZSTD_storeSeq	2016-03-16 12:03:07 +01:00
inikep	df164e6d35	introduced U32 rep[ZSTD_REP_NUM]	2016-03-16 11:56:59 +01:00
inikep	4569667c85	ZSTD_REP_NUM	2016-03-16 11:40:09 +01:00
Yann Collet	6cb41356e8	update doc / comments	2016-03-15 15:47:38 +01:00
Yann Collet	346bffbc68	Added frame content size test within fuzzer	2016-03-15 15:24:52 +01:00
Yann Collet	09b21ee2f6	fix fileio bug with new zbuff ... simplified zbuff decoding	2016-03-15 12:56:03 +01:00
inikep	d700a1a8b9	ZSTD_setAdditionalParam	2016-03-15 12:18:44 +01:00
Yann Collet	1c2c2bcaff	minor refactoring	2016-03-15 01:33:36 +01:00
Yann Collet	d1b26849e5	Frame content size supported	2016-03-15 01:24:33 +01:00
inikep	5fdd0b46d3	added additionalParam	2016-03-14 19:51:11 +01:00
Yann Collet	fb7973580f	minor comments update	2016-03-13 11:08:40 +01:00
Yann Collet	03ea59b17b	Decompression can decode frame content size	2016-03-12 01:25:40 +01:00
Yann Collet	0e491c01fe	created ZSTD_frameParams	2016-03-11 21:58:04 +01:00
inikep	cc6c283633	removed hashLog3 from ZSTD_parameters	2016-03-10 15:17:24 +01:00
inikep	260a9e3886	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev	2016-03-10 15:04:12 +01:00
inikep	b0c640d934	small ratio improvement with MINMATCH=3	2016-03-10 13:18:36 +01:00
Yann Collet	da5be72fda	update compression level parameters	2016-03-10 11:01:44 +01:00
Yann Collet	dcc000f0eb	minor doc clarifications	2016-03-10 01:09:41 +01:00
Yann Collet	e3193c4081	update compresson levels	2016-03-09 16:57:09 +01:00
Yann Collet	4f013a2281	Merge pull request #142 from inikep/dev ... Dev	2016-03-09 16:29:42 +01:00
inikep	b5a519f43c	improved compression speed ... introduced cache in ZSTD_setLog2Prices	2016-03-09 15:45:01 +01:00
Yann Collet	95cd0c2e26	frame format can support window sizes up to 128 MB	2016-03-08 18:24:21 +01:00
Yann Collet	ff14b351f5	Merge pull request #141 from inikep/dev ... fix update for zstd_opt	2016-03-08 14:41:04 +01:00
Yann Collet	d3a6660ad1	Update clevel table for <= 128K	2016-03-08 14:12:46 +01:00
Yann Collet	2e91dde43e	improved memory size evaluation by paramgrill	2016-03-08 12:22:11 +01:00
inikep	48413765b4	fixed update of hashTable for matches with length of 4+	2016-03-08 11:03:55 +01:00
Yann Collet	dd54bbc184	Fixed large NbSeq > 32 K ... Added a test in Fuzzer to check NbSeq > 32 K	2016-03-08 02:35:34 +01:00
Yann Collet	370b08e840	fix compression ratio for blocks <= 16 KB	2016-03-08 00:03:59 +01:00
Yann Collet	fb7ab8b2b8	Merge pull request #140 from inikep/dev ... mml3	2016-03-07 18:06:48 +01:00
inikep	4f7f79ef9d	fixed update of nextToUpdate	2016-03-07 16:14:58 +01:00
inikep	e2446b0e36	added support for MINMATCH=3 on big endian architecture ... changed ZSTD_HASHLOG3_MIN to 2 (4 bytes for MINMACH > 3)	2016-03-07 10:07:08 +01:00
inikep	89c9e1a847	added missing "case 3:" in ZSTD_BtGetAllMatches	2016-03-06 23:21:52 +01:00
Yann Collet	1df2594bfe	new compression level scale for <= 16KB	2016-03-05 18:43:21 +01:00
inikep	01060bc477	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev060' into dev ... # Conflicts: # lib/zstd_compress.c	2016-03-04 20:10:09 +01:00
inikep	f3c6503e55	resolve conflict in zstd_internal.h	2016-03-04 20:04:25 +01:00
inikep	e29caf7d1c	code cleanup	2016-03-04 19:52:23 +01:00
inikep	805d2a7465	removed zstd_opt_internal.h	2016-03-04 19:31:57 +01:00
Yann Collet	944d0d2285	updated fse version	2016-03-04 19:26:59 +01:00
inikep	4ab9c913a5	MEM_readMINMATCH replaced with inline function	2016-03-04 19:17:31 +01:00
Yann Collet	4c01580362	finer searchLength validation	2016-03-04 17:07:54 +01:00
Yann Collet	d409db68d3	Support for nbSeq > 32767 ... tamed -Wstrict-aliasing warning	2016-03-04 14:45:31 +01:00
Yann Collet	1435335342	Merge pull request #138 from inikep/dev ... reduce stack usage with optimal parsing mode	2016-03-03 02:08:34 +01:00
inikep	5cccd77c55	fix clang -Wcast-align warning (part 2)	2016-03-02 20:37:49 +01:00
inikep	51bb9a0064	fix clang -Wcast-align warning	2016-03-02 19:17:13 +01:00
inikep	87d4f3daa4	priceTable moved to heap	2016-03-02 15:56:24 +01:00
inikep	338533f741	improved price function initialization	2016-03-02 14:36:41 +01:00
Yann Collet	a997654670	Merge pull request #136 from inikep/dev ... merging mml3 mode	2016-03-01 17:31:03 +01:00
inikep	a4dde25498	fixed repcode before lowLimit	2016-03-01 14:14:35 +01:00
inikep	2d55563b92	better logs	2016-02-29 22:07:40 +01:00
inikep	f647d99d61	fixed bug with 3 or more succesive rep codes	2016-02-29 12:33:08 +01:00
inikep	f414647c6b	fixed ZSTD_copyCCtx	2016-02-25 22:31:07 +01:00
inikep	59493e8669	fixed paramgrill	2016-02-25 19:15:08 +01:00
inikep	afa9c9f5ab	turn off RTF optimization	2016-02-25 16:47:57 +01:00
Yann Collet	ee0229287c	changed version to v0.6.0	2016-02-25 14:53:35 +01:00
inikep	02137f8c42	more accurate gain function	2016-02-24 18:09:36 +01:00
inikep	ee55628c9d	combined price function	2016-02-24 14:40:30 +01:00
Yann Collet	3152a8c87a	finer dictionary completion	2016-02-23 21:28:59 +01:00
inikep	4a981f7937	improved ZSTD_GETPRICE	2016-02-23 19:08:20 +01:00
inikep	e0010e9baf	priceFunc	2016-02-23 16:25:04 +01:00
inikep	15174b0cfb	statistics of encoded sequences	2016-02-23 12:41:56 +01:00
inikep	9f754d23dc	dummy levels 22-25	2016-02-22 17:00:04 +01:00
inikep	6b3739c8e5	MINMATCH=3 or 4 selected in ZSTD frame	2016-02-22 15:53:42 +01:00
inikep	84f43e2359	support for searchLength from 3 to 7 for optimal parser	2016-02-22 11:34:07 +01:00
inikep	c950b78ce2	removed backward match extension	2016-02-22 10:11:39 +01:00
inikep	d3b8d7a4e5	removed ZSTD_opt parser	2016-02-22 10:06:17 +01:00
inikep	6291c54006	litlen bounded to 128 bytes	2016-02-19 18:24:14 +01:00
Yann Collet	f8482131d7	better compatibility with Visual <= 2012	2016-02-19 17:33:43 +01:00
Yann Collet	191b52a8dc	silence array subscript warning	2016-02-19 15:16:15 +01:00
inikep	fcafb26b2e	speed optimiaztions	2016-02-19 11:59:44 +01:00
inikep	c3a9a9ca56	ZSTD_rescaleFreqs and removed minml	2016-02-19 11:05:25 +01:00
inikep	cc52a97214	added hashLog3	2016-02-19 10:09:35 +01:00
Yann Collet	4c64d51425	modified level 18	2016-02-18 01:01:42 +01:00
Yann Collet	d56a4192f6	Added : README within /lib directory, to help file selection	2016-02-17 17:47:29 +01:00
Yann Collet	6062b15a07	performance fix for hash chain mode	2016-02-16 17:41:03 +01:00
Yann Collet	accfd80c37	Added : option -t to test compressed files integrity	2016-02-15 19:33:16 +01:00
Yann Collet	0068be94d8	Faster literals cost evaluation (suggested by @inikep)	2016-02-15 18:42:13 +01:00
Yann Collet	e93add0439	Removed if (matchIndex >= current) return 0; as recommended by @inikep	2016-02-15 17:44:14 +01:00
Yann Collet	b8a6f6824b	refactored fix to be more generic and applicable to both btlazy2 and btopt	2016-02-15 17:06:29 +01:00
Yann Collet	b37e0a9241	performance tweak : bt search w/ long repeated segments	2016-02-15 14:07:07 +01:00
Yann Collet	cefef8cde7	Change : zstd_opt_bt => zstd_btopt	2016-02-15 07:21:54 +01:00
Yann Collet	0d78a33374	updated CLevel confs for 256K	2016-02-15 06:49:09 +01:00
Yann Collet	422fd563f3	modified : compression Levels for blocks <= 16KB	2016-02-13 15:07:24 +01:00
Yann Collet	38fba56483	Changed compression level settings	2016-02-13 11:20:23 +01:00
Yann Collet	e8bd73df80	Fixed : error when compressing one file	2016-02-13 03:09:48 +01:00
Yann Collet	23e8aee224	Introduced new compression level settings ... Changed position of parameter TargetLength	2016-02-13 02:08:46 +01:00
Yann Collet	82f4d7da03	Fixed Visual solution	2016-02-12 22:04:49 +01:00
Yann Collet	6f3acbac0d	notificationLevel into ZDICT_param_t	2016-02-12 20:19:48 +01:00
Yann Collet	09ab681328	fixed libzstd install	2016-02-12 19:00:30 +01:00
Yann Collet	62ae5fb0d2	zstd_buffered => zbuff	2016-02-12 18:59:11 +01:00
Yann Collet	f4c9d75370	dictBuilder => zdict	2016-02-12 18:45:02 +01:00
Yann Collet	f062436fa6	cli : added ability to compress multiple files into stdout (-c)	2016-02-12 15:56:46 +01:00
Yann Collet	71eafdd23f	Integrated dictionary builder into zstd cli	2016-02-12 02:31:57 +01:00
Yann Collet	fb92a78ac1	Fixed clang warnings	2016-02-12 00:32:38 +01:00
Yann Collet	7d36028078	Integrate dictBuilder into libzstd	2016-02-12 00:10:25 +01:00
Yann Collet	82260ddd8a	fixed asan read error in ZSTD_HcGetAllMatches	2016-02-11 07:14:25 +01:00
Yann Collet	04b12d8bcb	paramgrill can select manually targetLength ... new compression levels	2016-02-11 06:23:24 +01:00
Yann Collet	bd828d9cd1	changed : sufficientLength => targetLength	2016-02-11 04:38:55 +01:00
Yann Collet	4488661678	minor cosmetics	2016-02-11 04:17:50 +01:00
Yann Collet	490aa68b18	restored fast zstd_opt variant	2016-02-10 17:18:24 +01:00
Yann Collet	b79a0b34f3	paramgrill supports sufficientLength ... stronger variation for ZSTD_opt	2016-02-10 17:07:37 +01:00
Yann Collet	62ae262771	restored currentMl>0 condition, as suggested by @inikep	2016-02-10 15:44:07 +01:00
Yann Collet	3b63f7fbb0	change zstd_opt.c into zstd_opt.h, since it's included	2016-02-10 15:05:12 +01:00
Yann Collet	0db4115f35	fixed debug print macros on Windows	2016-02-10 14:50:22 +01:00
Yann Collet	2f94d2f166	minor refactor	2016-02-10 14:26:30 +01:00
Yann Collet	768c6bc705	fix : inlining difficulties with gcc 4.6 (travis)	2016-02-10 14:01:49 +01:00
Yann Collet	70e8c389f1	Refactoring for portability	2016-02-10 13:37:52 +01:00
Yann Collet	302177a0c8	Merge pull request #125 from inikep/dev ... Dev	2016-02-10 00:57:00 +01:00
inikep	ce6730a740	fix for ZSTD_HcGetAllMatches_generic	2016-02-09 23:26:30 +01:00
inikep	be77f3342d	Revert "Revert "ZSTD_compressBlock_opt_extDict_generic"" ... This reverts commit 2b714dfd26.	2016-02-09 23:00:41 +01:00
inikep	2b714dfd26	Revert "ZSTD_compressBlock_opt_extDict_generic" ... This reverts commit b498fc1c99.	2016-02-09 22:47:51 +01:00
inikep	b498fc1c99	ZSTD_compressBlock_opt_extDict_generic	2016-02-09 22:23:48 +01:00
inikep	e75621f98d	dumpsEnd	2016-02-09 21:12:23 +01:00
inikep	31314ae0d1	int to U32	2016-02-09 20:47:39 +01:00
Yann Collet	6bff748e43	Fixed minor warnings in legacy modes	2016-02-09 17:55:01 +01:00
Yann Collet	4f0a3931c8	added missing patches	2016-02-07 04:00:27 +01:00
inikep	da730ff092	a bugfix for backward matches	2016-02-06 15:18:26 +01:00
inikep	dbade7c263	remove ZSTD_compressBlock_opt2_generic	2016-02-05 20:29:31 +01:00
inikep	f2fee4c316	added params.sufficientLength	2016-02-05 19:45:25 +01:00
inikep	5b0749715d	backward match extension	2016-02-05 19:25:40 +01:00
inikep	2712b69582	Merge remote-tracking branch 'refs/remotes/origin/v05x' into dev	2016-02-05 09:22:10 +01:00
inikep	3379c5df36	ZSTD_OPT_DEBUG	2016-02-05 09:21:20 +01:00
inikep	f967753bc2	ssize_t -> int	2016-02-05 09:07:22 +01:00
Yann Collet	982ffc773d	changed enum type name to ZSTD_ErrorCode	2016-02-05 02:33:10 +01:00
inikep	55a444764b	sufficient_len = 32	2016-02-04 22:22:47 +01:00
inikep	34608aaa29	ssize_t	2016-02-04 22:18:02 +01:00
inikep	17712afdbc	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into v05x	2016-02-04 21:49:45 +01:00
inikep	b3df71e538	code cleaning	2016-02-04 21:45:26 +01:00
inikep	4fa4098628	last match will start at a minimum distance of 8 from oend	2016-02-04 21:16:07 +01:00
inikep	08f500ca10	cur_min	2016-02-04 20:53:54 +01:00
inikep	6f07186745	clean code	2016-02-04 17:57:50 +01:00
inikep	59c9146258	40099096	2016-02-04 17:48:33 +01:00
inikep	baff8d6c1f	gitignore	2016-02-04 16:02:05 +01:00
inikep	40dd4739e3	removed llen	2016-02-04 15:52:35 +01:00
inikep	c9483f0982	clean code	2016-02-04 15:28:55 +01:00
Yann Collet	953ce72f4a	minor comments refactoring	2016-02-04 15:28:14 +01:00
inikep	158677c16a	new copy rep	2016-02-04 15:25:37 +01:00
inikep	d1ff7bf89a	cur_rep	2016-02-04 14:57:11 +01:00
inikep	eb72f5992a	i != litlen	2016-02-04 14:55:23 +01:00
inikep	997fb064a6	anchor	2016-02-04 13:01:56 +01:00
inikep	75be1a412c	fixes	2016-02-04 11:48:06 +01:00
inikep	cb5cd16d6b	40392170 integer	2016-02-04 11:19:06 +01:00
inikep	27e1c6a51f	ZSTD_resetFreqs	2016-02-04 11:11:08 +01:00
inikep	4196923dc9	40370570 ZSTD_getPriceReal	2016-02-04 10:04:33 +01:00
inikep	70b054559b	MaxLit	2016-02-03 22:56:55 +01:00
inikep	a98149df59	ZSTD_getLiteralPrice 2	2016-02-03 21:26:18 +01:00
inikep	dbbedfbd95	ZSTD_getLiteralPrice	2016-02-03 19:53:29 +01:00
inikep	892776efd6	ZSTD_get_price2	2016-02-03 19:27:44 +01:00
inikep	1292509987	ZSTD_get_price	2016-02-03 19:13:28 +01:00
inikep	749460d9ab	ZSTD_updatePrice	2016-02-03 19:08:48 +01:00
inikep	3bfcfc7bb0	ZSTD_updatePrice	2016-02-03 18:47:30 +01:00
inikep	53721f134c	Merge remote-tracking branch 'refs/remotes/Cyan4973/v05x' into v05x ... # Conflicts: # lib/zstd_compress.c # lib/zstd_static.h	2016-02-03 17:29:40 +01:00
inikep	78e5ea3a32	ZSTD_getPrice	2016-02-03 17:25:42 +01:00
inikep	d02506c186	40699855	2016-02-03 16:54:23 +01:00
inikep	85cc6b509b	minml	2016-02-03 15:58:08 +01:00
inikep	7f6680e658	clean some code	2016-02-03 13:30:34 +01:00
inikep	4cb2009821	40901629	2016-02-03 13:25:34 +01:00
inikep	1b8b1f9d53	auto check of matches	2016-02-03 13:17:28 +01:00
inikep	e9f30eae4b	40970636	2016-02-03 12:53:07 +01:00
Yann Collet	b010b3b627	Fixed decoding error (afl)	2016-02-03 12:39:34 +01:00
Yann Collet	72bff50ecf	Updated error API, following suggestions by @nemequ (#116)	2016-02-03 12:06:24 +01:00
Yann Collet	ae7aa06650	lots of minor refactorings	2016-02-03 02:46:46 +01:00
Yann Collet	7d968c7bdd	move _usingDict() to stable API (zstd.h)	2016-02-03 02:11:32 +01:00
Yann Collet	464fa996e9	added v0.4 legacy support	2016-02-03 01:09:46 +01:00
inikep	5391a718a4	fixed	2016-02-03 00:04:14 +01:00
inikep	55bc9a868a	41374379	2016-02-02 19:47:09 +01:00
inikep	5d4131067b	update	2016-02-02 17:41:21 +01:00
Yann Collet	7d8e6bddb9	changed long length format	2016-02-02 17:30:37 +01:00
inikep	d6e8cc5f0c	41403384	2016-02-02 16:43:31 +01:00
Yann Collet	7a6343fb1b	fixed encoding bugs	2016-02-02 16:00:50 +01:00
inikep	e1b8e53e84	better logging	2016-02-02 15:41:16 +01:00
Yann Collet	2ce4923238	modified buffer management rules	2016-02-02 14:36:49 +01:00
inikep	59140c5480	turn off: last match must start at a minimum distance of 8 from oend	2016-02-02 12:36:09 +01:00
inikep	ef2706ed12	in progress	2016-01-31 15:32:01 +01:00
inikep	6b0715b8fd	no "start"	2016-01-31 15:17:50 +01:00
inikep	d4c3e8c595	fixed logs	2016-01-31 14:28:26 +01:00
inikep	c0dc4c361d	best_off	2016-01-31 12:36:41 +01:00
inikep	9bfb1a3357	working	2016-01-31 12:17:16 +01:00
inikep	ee4be5e981	ZSTD_compressBlock_opt_generic	2016-01-31 11:57:01 +01:00
inikep	e2bfe2407a	GetAllMatches	2016-01-31 11:26:05 +01:00
inikep	c71568fd44	ZSTD_opt	2016-01-31 11:26:04 +01:00
Yann Collet	35f7de52c8	fixed Visual warnings	2016-01-31 02:51:03 +01:00
Yann Collet	61e16ce07c	minor optimization for small files	2016-01-31 02:04:15 +01:00
Yann Collet	e93d6ced17	minor compression gain	2016-01-31 00:58:06 +01:00
Yann Collet	fd416f1eaf	added : ZSTD_compress_usingPreparedCCtx()	2016-01-30 03:14:15 +01:00
Yann Collet	863ec40f1e	minor compression speed improvement	2016-01-28 17:56:33 +01:00
Yann Collet	9cadd0853c	add basic tests	2016-01-28 15:39:52 +01:00
Yann Collet	4ddb1f5566	optimized counting of small segments	2016-01-28 03:24:53 +01:00
Yann Collet	fb810d6053	changed dictionary format	2016-01-28 00:21:13 +01:00
Yann Collet	1c8e194297	modified streaming compression API	2016-01-26 16:31:22 +01:00
Yann Collet	7b51a2948f	modified streaming decompression API	2016-01-26 15:58:49 +01:00
Yann Collet	b923f65076	introduced dictionary format	2016-01-26 03:14:20 +01:00
Yann Collet	bc4c8aa4b7	fixed new MIN_CBLOCK_SIZE	2016-01-25 17:26:01 +01:00
Yann Collet	3742219b4e	fixed decompression bugs	2016-01-25 16:54:05 +01:00
Yann Collet	a1249dc8da	fix visual warnings	2016-01-25 04:22:03 +01:00
Yann Collet	afe0709964	stream control	2016-01-25 04:10:46 +01:00
Yann Collet	59d1f79768	variable litblock header size	2016-01-23 19:28:41 +01:00
Yann Collet	a768a301ba	fixed VS project	2016-01-21 16:04:35 +01:00
Yann Collet	ffec740d37	fixed visual and clang errors	2016-01-21 15:50:11 +01:00
Yann Collet	977f1f3600	modified error system, following suggestions by @nemequ	2016-01-21 15:38:47 +01:00
Yann Collet	74ce8e812c	fixed VS 2015 'ERROR already defined', reported by Rich Geldreich	2016-01-21 14:32:41 +01:00
Yann Collet	7beaa05d81	fixed bt bug	2016-01-21 11:57:45 +01:00
Yann Collet	1840d6af33	bench pay attention to variable block sizes	2016-01-20 15:39:06 +01:00
Yann Collet	fba6aed175	changed : ZSTD_LEGACY_SUPPORT set to 0 by default within zstd_decompress.c (#113)	2016-01-18 12:03:54 +01:00
Yann Collet	a87278a4f0	small compression speed	2016-01-17 00:12:55 +01:00
Yann Collet	3a3b72f25a	Implemented ZSTD_HEAPMODE for zstd_decompress.c ... Reduce cmake version requirement to 2.8.7	2016-01-11 12:56:11 +01:00
Yann Collet	523b594233	fixed read error	2016-01-09 02:10:40 +01:00
Yann Collet	b012510d9f	dictionary with block API	2016-01-09 02:00:10 +01:00
Yann Collet	bf42c8e5d8	block-level API	2016-01-09 01:08:23 +01:00
Yann Collet	c64c100658	fixed : unused variable	2016-01-08 17:54:02 +01:00
Yann Collet	600962711d	added duplication tests	2016-01-08 17:27:50 +01:00
Yann Collet	6e1c4c6c65	fixed context copy	2016-01-07 23:07:44 +01:00
Yann Collet	ecd651bd91	added ZSTD_duplicateCCtx()	2016-01-07 15:35:18 +01:00
Yann Collet	494c786a3b	fixed outstanding strict aliasing warnings	2016-01-06 12:54:02 +01:00
Yann Collet	1fdd82312d	Fixed : strict aliasing warnings in legacy decoders	2016-01-06 12:35:42 +01:00
Yann Collet	218bd31942	minor strict aliasing warning	2016-01-06 02:19:55 +01:00
Yann Collet	3b994cb465	updated fse	2016-01-06 01:58:37 +01:00
Yann Collet	96ffa424eb	minor fast mode improvement	2016-01-02 01:16:28 +01:00
Yann Collet	2cc12cb3cb	minor refactoring	2016-01-01 07:47:58 +01:00
Yann Collet	1358f91ec4	minor refactoring	2016-01-01 07:29:39 +01:00
Yann Collet	72e84cfd0f	minor simplification	2015-12-31 19:08:44 +01:00
Yann Collet	722504c2a0	update version to 0.4.6	2015-12-31 18:32:15 +01:00
Yann Collet	5f9f98768a	Makefile version automatically follows zstd.h	2015-12-30 19:52:08 +01:00
Yann Collet	48da164f14	fixed clang warnings	2015-12-29 23:40:02 +01:00
Yann Collet	ee3f4515c1	faster hc mode with repetitive data (experimental)	2015-12-29 22:26:09 +01:00
Yann Collet	d94efbf95c	fix fast mode on Windows	2015-12-29 14:29:08 +01:00
Yann Collet	1ac364d60b	Merge pull request #99 from KrzysFR/dev ... Update version to 0.4.5 in zstdlib resources	2015-12-21 10:19:26 +01:00
Yann Collet	ed73ae8c67	fix #101	2015-12-19 19:51:42 +01:00
Christophe Chevalier	8972f13b4a	Added missing ZSTDLIB_API for ZSTD_decompress_usingDict and ZSTD_decompressDCtx	2015-12-18 11:20:37 +01:00
Yann Collet	31683c0b28	benchmark can use dictionary	2015-12-18 01:26:48 +01:00
Yann Collet	fdcad6d3e1	added ZSTD_compress_usingDict()	2015-12-17 23:50:15 +01:00
Yann Collet	deb078b9b1	added : -m : decompress multiple files	2015-12-17 20:30:14 +01:00
Yann Collet	f54f57023c	update params for 256K	2015-12-16 19:38:54 +01:00
Yann Collet	0700585fb9	fixed asan warning	2015-12-12 12:54:48 +01:00
Yann Collet	188b6ed6c7	fixed g++ warning	2015-12-12 12:44:44 +01:00
Yann Collet	3a061a4c9f	extended static dictionary to buffered mode	2015-12-12 11:17:42 +01:00
Yann Collet	6c3e2e7855	fixed : high compression modes for Windows 32 bits	2015-12-11 10:44:07 +01:00
Yann Collet	9f5ab1a4f5	fix 32-bits windows decoding on high-compression archives	2015-12-11 00:27:41 +01:00
Christophe Chevalier	2abb04d90d	Added ZSTDLIB_API to new ZSTD_maxCLevel method	2015-12-09 23:55:23 +01:00
Christophe Chevalier	7b05324a71	Only define ZSTDLIB_API when building a Windows DLL	2015-12-09 17:12:12 +01:00
Christophe Chevalier	844baf4fc5	Merge remote-tracking branch 'upstream/dev' into zstdlib_api	2015-12-09 15:45:57 +01:00
Yann Collet	d608088ca3	added : ZSTD_maxCLevel() ... added : 256KB blocks mode	2015-12-09 09:05:22 +01:00
Yann Collet	0cde77bc5e	fixed comment (reported by @annulen)	2015-12-08 14:47:46 +01:00
Christophe Chevalier	d2199e7250	Added ZSTDLIB_API definition to zstd_buffered.h which does not seem to include zstd.h	2015-12-07 19:01:41 +01:00
Christophe Chevalier	c6e845398a	Add ZSTDLIB_API macro to prefix all exported methods with__declspec(dllexport) when building a DLL (on windows)	2015-12-07 17:48:53 +01:00
Yann Collet	4bfe415fa7	validated external dictionary	2015-12-06 13:18:37 +01:00
Yann Collet	e47c4e5f8e	strengthened bufferless streaming decompression	2015-12-05 09:23:53 +01:00
Yann Collet	417890cec2	strengthened streaming bufferless compression	2015-12-04 17:16:37 +01:00
Yann Collet	6a2f03295c	update version to 0.4.2	2015-12-02 15:05:22 +01:00
Yann Collet	120230b895	compress uses less memory for small blocks	2015-12-02 14:00:45 +01:00
Yann Collet	a308259afa	new : parameters for <= 16KB	2015-12-02 13:38:48 +01:00
Peter Harris	f06e238d6f	Fix compression on Big Endian CPUs	2015-12-01 14:58:24 -05:00
Yann Collet	c620b48489	revert install to normal zstd due to travis limitation (likely gcc 4.6)	2015-12-01 01:56:02 +01:00
Yann Collet	44287a3bf4	refactored decompress (#82)	2015-11-30 23:13:56 +01:00
Yann Collet	aead1bf86b	changed to ZSTD_LEGACY_SUPPORT=0, as suggested within #80	2015-11-30 14:18:47 +01:00
Yann Collet	daa1c65500	fixed #81 : removed dead file (reported by @inikep)	2015-11-30 12:25:48 +01:00
Yann Collet	a85a8dd94a	Fixed #80 (reported by @luben) ... Added : tests for ZSTD_LEGACY=disable build mode extended ZSTD_LEGACY=disable to dll build	2015-11-30 11:53:11 +01:00
Yann Collet	375727310c	fixed compression bug	2015-11-29 03:17:04 +01:00
Yann Collet	7fe531e7ec	fixed address space overflow (32-bits mode)	2015-11-29 02:38:09 +01:00
Yann Collet	52732b505d	fixed huff0 decoder (32-bits mode)	2015-11-28 22:03:08 +01:00
Yann Collet	bf7aa3c14e	fixed decoder error (rollbuffer mode)	2015-11-28 18:19:44 +01:00
Yann Collet	ad50c59bb7	fixed decoder error (32-bits mode, malicious input)	2015-11-28 17:09:28 +01:00
Yann Collet	00fd7a2110	protect 32-bits mode	2015-11-28 16:03:22 +01:00
Yann Collet	53fbf01370	minor level change	2015-11-28 14:08:01 +01:00
Yann Collet	800fa6c378	stronger tests for zbuff decompression	2015-11-27 14:30:23 +01:00
Yann Collet	31d1806a77	zbuff properly supports small dst buffers	2015-11-27 14:07:36 +01:00
Yann Collet	944042ab7d	coalesce block+header size (buffered mode)	2015-11-27 13:26:38 +01:00
Yann Collet	d248aa1c62	fixed hang bug (infinite loop) in decoder, buffered mode, malicious input	2015-11-26 22:39:29 +01:00
Yann Collet	661b94fb5d	fixed potential warning on undefine, reported by Maciej Adamczyk	2015-11-26 16:42:06 +01:00
Yann Collet	29a58c94f6	fixed lib makefile	2015-11-26 16:25:56 +01:00
Yann Collet	26fa696713	minor API change	2015-11-26 16:07:08 +01:00
Yann Collet	29a2c838c5	support for legacy 0.3 format	2015-11-26 16:02:04 +01:00
Yann Collet	c2ce890ac2	fixed bug in legacy decoder v0.2, reported by Maciej Adamczyk	2015-11-26 14:12:23 +01:00
Yann Collet	26415d3ff2	fixed leak	2015-11-26 12:43:28 +01:00
Yann Collet	6bcdeac954	fixed asan warning	2015-11-26 11:43:00 +01:00
Yann Collet	5f2ec63852	fixed decompression bug (buffered mode)	2015-11-26 10:32:17 +01:00
Yann Collet	91a97960f7	Added zbufftest	2015-11-26 09:59:49 +01:00
Yann Collet	e4fdad55dc	fixed minor decompression bug in buffered mode	2015-11-25 21:09:17 +01:00
Yann Collet	d3cb690156	fixed leak issue	2015-11-25 15:26:55 +01:00
Yann Collet	377b9e8b94	fixed : potential malloc error	2015-11-25 15:04:37 +01:00
Yann Collet	7b2c31043c	fixed : potential malloc error	2015-11-25 15:02:46 +01:00
Yann Collet	88fcd2916e	Added : zstd buffered API	2015-11-25 14:42:45 +01:00
Yann Collet	c36521571e	fixed large buffers bug	2015-11-24 14:06:07 +01:00
Yann Collet	225179dd05	more tests (bt modes)	2015-11-23 16:52:22 +01:00
Yann Collet	239cc289d3	fixed asan bugs	2015-11-23 16:17:21 +01:00
Yann Collet	03526e18fa	preliminary rollbuffer support for bt mode	2015-11-23 15:29:15 +01:00
Yann Collet	c1e52f0f05	hc simplication	2015-11-23 14:37:59 +01:00
Yann Collet	06eade5102	comments	2015-11-23 14:23:47 +01:00
Yann Collet	5054ee0cc0	rollbuffer refactor	2015-11-23 13:34:21 +01:00
Yann Collet	d7233d6e14	more compression tests	2015-11-22 14:40:51 +01:00
Yann Collet	287b7d9607	small hc speed improvement	2015-11-22 13:24:05 +01:00
Yann Collet	428619ccd7	small cLevel update	2015-11-22 12:46:30 +01:00
Yann Collet	a85c77b9d3	fixed roll buffer lazy2 mode	2015-11-22 12:22:04 +01:00
Yann Collet	b7fc88eec7	fixed roll buffer lazy mode	2015-11-22 03:12:28 +01:00
Yann Collet	734aa92ed5	fixed roll buffer greedy mode	2015-11-22 03:01:33 +01:00
Yann Collet	9a24e5986b	fixed roll buffer in fast mode	2015-11-22 02:53:43 +01:00
Yann Collet	007c1c6ff0	minor cr improvement	2015-11-22 02:42:28 +01:00
Yann Collet	7a23179779	small fast compression improvement	2015-11-21 15:27:35 +01:00
Yann Collet	402fdcf1a3	fixed asan	2015-11-20 12:46:08 +01:00
Yann Collet	55aa7f94e3	modified format	2015-11-20 12:04:52 +01:00
Yann Collet	743402cb23	modified format	2015-11-20 12:03:53 +01:00
Yann Collet	7dfd56be4a	restored fix small blocks	2015-11-19 17:46:29 +01:00
Yann Collet	c95f899725	refactored greedy	2015-11-19 17:28:35 +01:00
Yann Collet	b3a2af970e	refactored decompression	2015-11-19 17:13:19 +01:00
Yann Collet	110cc14bab	fixed bug #73 notified by @nemequ	2015-11-19 12:02:28 +01:00
Yann Collet	dcac00e7a6	update library builder	2015-11-19 11:23:44 +01:00
Yann Collet	b2549846ba	more tests	2015-11-18 11:29:32 +01:00
Yann Collet	94b9d8ec8d	fix compression bug in rollbuffer mode	2015-11-17 16:14:55 +01:00
Yann Collet	138e89c9c4	fix compression bug (extDict mode)	2015-11-17 14:26:54 +01:00
Yann Collet	786eb7d42e	fixed repcode bug	2015-11-17 12:23:53 +01:00
Yann Collet	82368cfa1b	comment	2015-11-16 19:10:56 +01:00
Yann Collet	eaa5f75b99	fixed decompression bug	2015-11-15 11:13:41 +01:00
Yann Collet	93a823ccb7	fixed asan warnings	2015-11-13 15:08:43 +01:00
Yann Collet	89db5e001d	rolling buffer preliminary support for fast mode	2015-11-13 11:27:46 +01:00
Yann Collet	0f366c61ed	fixed streaming decoder	2015-11-12 16:19:30 +01:00
Yann Collet	5b78d2f20c	basic rolling buffer support in decoder	2015-11-12 15:36:05 +01:00
Yann Collet	14983e7aed	refactored compression code	2015-11-11 21:38:21 +01:00
Yann Collet	7f6e91fa98	fixed library compilation	2015-11-11 14:39:50 +01:00
Yann Collet	5be2dd25f2	Changed Library source tree ... - no more zstdhc ; zstd.h is enough - ZSTD_compress() now needs a compressionLevel - separated zstd_compress.c and zstd_decompress.c - updated zstdcli, fullbench, fuzzer with new API	2015-11-11 13:43:58 +01:00
Yann Collet	530918b409	level tuning	2015-11-10 10:52:58 +01:00
Yann Collet	9f432926b4	fileio can use small blocks params	2015-11-09 17:42:17 +01:00
Yann Collet	43e0cd512a	small blocks params	2015-11-09 16:38:17 +01:00
Yann Collet	4baee50728	fix bug	2015-11-09 03:19:33 +01:00
Yann Collet	d1ade5a61d	fix 32-bits mode	2015-11-08 15:49:20 +01:00
Yann Collet	e9eba608c2	simplified bt	2015-11-08 15:08:03 +01:00
Yann Collet	31d00ba4b3	fix bt bug	2015-11-08 14:11:29 +01:00
Yann Collet	72a8e751f4	level tuning	2015-11-07 22:24:46 +01:00
Yann Collet	f48e35c206	minor bt improvements	2015-11-07 01:13:31 +01:00
Yann Collet	a81d9ac42d	minor bt CSpeed	2015-11-06 19:03:59 +01:00
Yann Collet	628065cf98	revert	2015-11-06 18:44:54 +01:00
Yann Collet	9b8e453713	minor CRatio improvement	2015-11-06 18:28:02 +01:00
Yann Collet	47cfa9a985	minor CRatio improvement	2015-11-06 17:13:46 +01:00
Yann Collet	eb283710dd	minor greedy CRatio improvement	2015-11-06 16:33:11 +01:00
Yann Collet	fc2afcfa64	small Cspeed increase	2015-11-06 15:40:14 +01:00
Yann Collet	444873f494	minor greedy mod	2015-11-06 11:48:56 +01:00
Yann Collet	805a52a773	fixed asan warning	2015-11-06 10:52:17 +01:00
Yann Collet	8ad12ec343	level tuning	2015-11-06 10:19:00 +01:00
Yann Collet	f12c1304a8	integrate fast mode	2015-11-05 18:16:59 +01:00
Yann Collet	1f44b3f778	added fast mode	2015-11-05 17:32:18 +01:00
Yann Collet	47b68908b2	rename lazy2	2015-11-05 15:14:17 +01:00
Yann Collet	9036105b62	simplified source	2015-11-05 15:03:12 +01:00
Yann Collet	5106a76dc9	merged strats	2015-11-05 15:00:24 +01:00
Yann Collet	43ae161fe9	level tuning	2015-11-05 11:58:02 +01:00
Yann Collet	92309cb37e	level tuning	2015-11-05 00:48:37 +01:00
Yann Collet	3137d1a059	fix fuzzer32 litCSize limit condition	2015-11-04 23:36:36 +01:00
Yann Collet	50c5cdb44c	fixed issues reported by Maciej Adamczyk	2015-11-04 20:35:33 +01:00
Yann Collet	3e3582719c	Fixed issue #62, reported by @luben	2015-11-04 18:19:39 +01:00
Yann Collet	e8455f51ea	fix clang warning	2015-11-04 17:41:20 +01:00
Yann Collet	b241e9deb7	small compression improvement	2015-11-04 13:57:24 +01:00
Yann Collet	59d7063fbc	fix and level tuning	2015-11-04 12:05:27 +01:00
Yann Collet	96b9f0ba4d	btlazy2	2015-11-04 03:52:54 +01:00
Yann Collet	588d1e5fa0	Fixed asan issue reported by Maciej Adamczyk	2015-11-03 10:48:42 +01:00
Yann Collet	050efba81b	level tuning	2015-11-03 09:49:30 +01:00
Yann Collet	3564487614	lazydeep	2015-11-02 16:14:46 +01:00
Yann Collet	7671f393be	v0.3.2	2015-11-02 12:17:39 +01:00
Yann Collet	4e049bca8c	Fixed Visual Studio issue #60, reported by @mailagentrus	2015-11-02 11:09:54 +01:00
Yann Collet	56e1092cb0	level tuning	2015-11-02 01:02:43 +01:00
Yann Collet	9b11b46f8a	Fixed speed regression	2015-11-01 12:40:22 +01:00
Yann Collet	be2010ea1b	lazy strategy	2015-10-31 12:57:14 +01:00
Yann Collet	4b100f43b0	level tuning	2015-10-30 15:49:48 +01:00
Yann Collet	ec43ba4a95	updated visual studio projects	2015-10-30 11:51:26 +01:00
Yann Collet	aa074051d6	decoding support for legacy v0.2 format	2015-10-30 11:21:50 +01:00
Yann Collet	fc04bcc6d7	fixed minor gcc warning	2015-10-30 07:11:07 +01:00
Yann Collet	8f9d49c092	fixed asan warning	2015-10-30 07:00:37 +01:00
Yann Collet	4114f95ce9	fix Visual Studio projects	2015-10-30 06:40:22 +01:00
Yann Collet	44fe99116e	fixed hc alloc	2015-10-29 22:02:40 +01:00
Yann Collet	712def905b	one less hc malloc	2015-10-29 18:41:45 +01:00
Yann Collet	3d9cf7acf2	fix include	2015-10-29 17:15:14 +01:00
Yann Collet	71bcdb5c1f	sync paramgrill with HC starting at level 2	2015-10-29 17:08:03 +01:00
Yann Collet	2acb5d3d48	HC modes can be selected from command line	2015-10-29 16:49:43 +01:00
Yann Collet	76c5c2a4a0	minor tuning	2015-10-28 19:07:05 +01:00
Yann Collet	81e49e6654	fine tune & messages	2015-10-28 15:56:48 +01:00
Yann Collet	b315bc86e1	srcLog	2015-10-27 13:12:25 +01:00
Yann Collet	2c6992e1dc	blocksize & T	2015-10-27 12:18:00 +01:00
Yann Collet	d55a7e0e4d	simplified chain	2015-10-27 03:06:40 +01:00
Yann Collet	8b91abec18	can test param conf	2015-10-27 02:59:12 +01:00
Yann Collet	342892c3ae	repcode+1	2015-10-26 17:44:04 +01:00
Yann Collet	786f5b554a	skip slow conf faster	2015-10-26 15:45:58 +01:00
Yann Collet	3a787da419	Fixed : minor clang warning	2015-10-25 19:42:03 +01:00
Yann Collet	083fcc82fc	HC params	2015-10-25 14:06:35 +01:00
Yann Collet	53fff6cd9d	Individual memlog values	2015-10-24 13:48:37 +01:00
Yann Collet	ed0a781246	repfirst	2015-10-23 19:25:06 +01:00
Yann Collet	83fae977e6	Added asan tests	2015-10-23 15:21:53 +01:00
Yann Collet	fc60883d42	fixed issue #49, reported by Hanno Böck (@hannob)	2015-10-23 12:45:02 +01:00
Yann Collet	eeb8ba18d1	uncoupled maxdlog and chainlog	2015-10-22 16:55:40 +01:00
Yann Collet	f3eca25322	quick first zstdhc version	2015-10-22 15:31:46 +01:00
Yann Collet	353c5d26cf	unified ctx naming convention	2015-10-21 14:39:26 +01:00
Yann Collet	f42803e5b4	restored ZSTD_compressCCtx() def	2015-10-21 13:31:46 +01:00
Yann Collet	8eb50b8acf	Fixed minor gcc5+ warning	2015-10-21 10:01:09 +01:00
Yann Collet	7010c27a13	Added advanced compression functions	2015-10-21 09:07:25 +01:00
Yann Collet	c8a7254192	clang optimizer bug workaround	2015-10-20 03:01:38 +01:00
Yann Collet	4caefaf62f	refactored literal segment	2015-10-19 19:25:44 +01:00
Yann Collet	1fbee1f483	Fixed sanitizer warning	2015-10-19 15:32:47 +01:00
Yann Collet	4d1a0ef515	minor speed improvement	2015-10-19 03:12:23 +01:00
Yann Collet	c7ae9ffde3	small decompression speed improvement	2015-10-19 02:36:34 +01:00
Yann Collet	107c5755f4	small decompression speed improvement	2015-10-19 02:14:43 +01:00
Yann Collet	50174c77fd	missed updated files	2015-10-18 22:26:26 +01:00
Yann Collet	b1f3f4b565	update huff0	2015-10-18 22:18:32 +01:00
Yann Collet	8f86c700cd	Fix uninitialized warnings reported by @nemequ	2015-10-14 17:53:44 +01:00
Yann Collet	59aac5f467	Clarified comments	2015-10-14 16:28:19 +01:00
xaphier	77ee44c7b6	Remove one malloc/free from compression ... By making the buffer part of the zstd context structure, one malloc/free can be removed from the compression.	2015-10-10 12:14:51 +02:00
Yann Collet	be50aaa0ee	Added : Frame concatenation ability	2015-09-10 23:26:09 +01:00
Yann Collet	5abd8203cb	Fix : decompression i/o detects flush write errors	2015-08-27 03:16:04 +01:00
Yann Collet	866a3c3158	Fixed minor visual warning	2015-08-26 23:10:09 +01:00
Yann Collet	352ab88aea	fixed minor issue on decoding invalid data	2015-08-26 22:45:23 +01:00
Yann Collet	6b5198fe3a	Updated FSE	2015-08-26 19:22:01 +01:00
Yann Collet	bf50409e05	Refactor Makefile and .travis tests	2015-08-26 00:19:06 +01:00
Yann Collet	f44b2b02b4	Zstd cli : --version and --help write to stdout	2015-08-25 23:32:45 +01:00
Yann Collet	c705567522	Fixed : compression error on some specific distribution	2015-08-25 22:56:42 +01:00
Yann Collet	1885029ba1	Fixed a few issues found by AFL (American Fuzzy Lop)	2015-08-24 20:17:11 +01:00
Yann Collet	fee8e240c7	Fixed decoding error #11 (reported by @magv)	2015-08-24 15:47:04 +01:00
Yann Collet	d5d9bc3f82	Added : ZSTD decompression : ability to resist to noisy/faulty data source	2015-08-23 23:13:49 +01:00
Yann Collet	d02114e0e1	updated fse	2015-08-21 03:59:31 +01:00
Yann Collet	997f9ee2ef	Added : fuzzer tests : decompressing noisy src	2015-08-21 02:44:20 +01:00
Yann Collet	7d20acd340	Fix : decoder issue in exceptionnal circumstances (dst buffer too small)	2015-08-20 15:55:50 +01:00
Yann Collet	602834f794	Fixed : bug in compression in specific conditions (too small dst size)	2015-08-20 07:46:10 +01:00
Yann Collet	23743530e0	Updated fse	2015-08-19 23:53:56 +01:00
Yann Collet	f4ce8913a3	Added more strigent tests : compresson into too small buffer	2015-08-11 14:18:45 +01:00
Yann Collet	f90314610d	Minor decompression simplification	2015-08-08 02:54:26 +01:00
Yann Collet	8a46c159c5	minor compress sequence refactor	2015-08-08 02:16:11 +01:00
Yann Collet	e9853b2cdb	Fixed : ZSTD_compress() can attempt compression on a too small buffer	2015-08-07 19:07:32 +01:00
Yann Collet	96f3b2c0f7	zstd_compress() no longer requires maxDstSize >= ZSTD_compressBound(srcSize)	2015-08-07 15:50:42 +01:00
Yann Collet	a787550d1c	Updated FSE lib	2015-08-07 15:21:00 +01:00
Yann Collet	77c82b680b	updated fse	2015-08-02 01:19:09 +01:00
Yann Collet	138db2187d	Fixed some minor sanitizer warnings within huff0	2015-07-27 20:19:21 +01:00
Yann Collet	fb8296f55c	Updated fse (faster huff0 32-bits decoding)	2015-07-27 19:34:27 +01:00
Yann Collet	e8c6bb1e42	Integrated huff0 (breaking format change)	2015-07-26 00:23:57 +01:00
Yann Collet	968f275981	Changed allocation strategy to reduce stack usage of ZSTD_compressSequences()	2015-07-07 16:54:25 -08:00
Yann Collet	858db1291f	Fixed : ZSTD_compress() corrently prevents buffer overflow	2015-07-07 01:02:04 -08:00
Yann Collet	dc6c1d920a	Fixed clang warning	2015-07-07 00:44:22 -08:00
Yann Collet	674d91b3da	Added sanitize=undefine test	2015-07-07 00:36:49 -08:00
Yann Collet	17867ced03	Fixed minor clang warnings	2015-07-07 00:14:27 -08:00
Yann Collet	7083b8a54a	Travis CI : added ARM compilation tests	2015-07-06 23:58:59 -08:00
Yann Collet	ee28a0def4	Fixed issue #34 and #31	2015-07-06 23:45:35 -08:00
Yann Collet	94f998b1fc	Fixed a few visual analyzer warnings	2015-07-04 23:10:40 -08:00
Yann Collet	7393c5a51d	Advanced API change : Cctx & DCtx are now incomplete types for stricter type checking	2015-07-04 18:20:42 -08:00
Yann Collet	1efa31fad8	Updated FSE lib	2015-07-04 15:56:41 -08:00
Yann Collet	078a9a2804	Added Wcast-qual compilation flag ... Updated xxHash Removed log traces	2015-06-20 19:37:53 -08:00
Yann Collet	213089c078	Updated FSE ... Added clang and g++ tests	2015-06-18 07:43:16 -08:00
Yann Collet	bbfa7d77c9	Fixed issue #28	2015-03-23 23:15:22 +01:00
Yann Collet	ad68a0e5f7	"type redef fix" for gcc <= 4.4	2015-02-26 00:29:15 +01:00
Yann Collet	26aa1ec946	updated FSE	2015-02-24 09:05:58 +01:00
Yann Collet	a3c75bad5a	Updated FSE ... Faster speed on barely compressible data	2015-02-21 03:31:59 +01:00
Yann Collet	00be3437a8	Some comments, to explain streaming decompression API	2015-02-20 18:53:31 +01:00
Yann Collet	1db6f22f34	Removed useless constants	2015-02-17 13:38:44 +01:00
Yann Collet	c5d46b5c90	New streaming API behavior, to solve issue 19 (https://github.com/Cyan4973/zstd/issues/19)	2015-02-16 18:06:26 +01:00
Yann Collet	fb98fd0bd4	datagen refactoring	2015-02-10 18:15:20 +01:00
Yann Collet	e30c39cd1b	internal refactoring - better separation between match & entropy	2015-02-09 03:50:11 +01:00
Yann Collet	2ddf7e9a28	Fixed : FSE issue with corner case distributions	2015-02-08 20:26:47 +01:00
Yann Collet	d6914bfd5a	fix minor Visual warning	2015-02-01 12:09:46 +01:00
Yann Collet	aacace37f7	Fixed a few clang warnings	2015-02-01 11:57:30 +01:00
Yann Collet	b0802817c0	avx2 mode solely detected using __AVX2__ macro	2015-02-01 10:13:22 +01:00
Yann Collet	e5c25e0ebc	minor fix for Visual	2015-01-31 20:10:53 +01:00
Yann Collet	936e678dda	Fixed : issue 8 regarding huge input buffer (> 4 GB)	2015-01-31 15:09:51 +01:00
Yann Collet	439eb77088	Normalize all the line endings	2015-01-31 10:52:59 +01:00
Yann Collet	1cc58def2d	Restored cast for C++ (fse)	2015-01-29 07:13:54 +01:00
Yann Collet	6434adcf54	Fix : no AVX2 for Visual 2008 and older	2015-01-29 07:00:43 +01:00
Yann Collet	565b81d0ba	fixed : corner case (FSE)	2015-01-29 06:51:30 +01:00
Yann Collet	e98ea20c37	Fix for VS2005 (AVX2 intrinsics)	2015-01-28 01:23:14 +01:00
r-lyeh	0da7941a72	Bugfixed Visual Studio errors when compiled as C++ sources	2015-01-26 10:24:04 +01:00
Yann Collet	fb81417323	Fixed : minor warnings under Visual 32 bits	2015-01-25 13:19:12 +01:00
Yann Collet	759433d679	Updated fullbench tests	2015-01-24 13:31:55 +01:00
Yann Collet	04d8e15e57	fixed : file permissions	2015-01-24 11:22:13 +01:00
Yann Collet	4856a00164	Initial release	2015-01-24 01:58:16 +01:00