2074 Commits

Author	SHA1	Message	Date
Yann Collet	89bc309d90	error out when --adapt is associated with --single-thread ... since they are not compatible	2018-09-19 14:49:13 -07:00
Yann Collet	2f78228f65	Merge branch 'dev' into adapt	2018-09-19 12:43:42 -07:00
Casey McGinty	d019d15f2b	Add build define to manually enable -rdynamic LD flag	2018-09-12 14:00:08 -07:00
Casey McGinty	b703181f93	Disable -rdynamic LD option on Windows	2018-09-11 16:19:34 -07:00
Casey McGinty	b9118ecdab	Update comments, and LD flag usage in Make	2018-09-11 14:49:47 -07:00
Casey McGinty	2224ecd718	Remove dead code and method name typo	2018-09-11 11:56:50 -07:00
Casey McGinty	d4337b6f1d	Move ABRThandler func out of internal lib	2018-09-11 11:39:49 -07:00
Casey McGinty	2a3967b7c4	Refactor abort signal handling ... - Print signal name to term - Add -rdynamic option to generate Linux symbol names in backtrace - Raise default signal after handler to ensure program termination	2018-09-11 10:40:45 -07:00
Casey McGinty	a06574fc97	Print a stack trace on unexpected term signal (e.g. SIGABRT) ... For OSX and Linux, add a signal handler to SIGABRT, SGIFPE, SIGILL, SIGSEGV, and SIGBUS. When the program terminates unexpectedly the handler will print the current stack to the terminal to help determine the location of the failure. On OSX the output will look like: ``` Stack trace: 4 zstd 0x000000010927ed96 main + 16886 5 libdyld.dylib 0x00007fff767d1015 start + 1 6 ??? 0x0000000000000001 0x0 + 1 ``` On Linux the output will look like: ``` Stack trace: ./zstd() [0x4b8e1b] ./zstd() [0x4b928a] ./zstd() [0x403dc2] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5e0fbb0445] ./zstd() [0x405754] ``` As is, the code does not function on WIN32. See also: https://oroboro.com/stack-trace-on-crash/	2018-09-06 18:46:52 -07:00
Yann Collet	deb03d8a26	fixed UTIL_isLink() for Mac OS-X	2018-09-06 17:32:16 -07:00
Yann Collet	50b216146f	Merge pull request #1304 from facebook/largeNbDicts ... contrib/largeNbDicts	2018-09-06 09:50:56 -07:00
Jennifer Liu	21721b75a3	Change default f to 20	2018-09-04 17:15:14 -07:00
Yann Collet	39c55a118f	fixed minor compatibility issues with older compilers	2018-08-30 16:00:57 -07:00
Jennifer Liu	f87383507d	Update comment about default dictionary builder	2018-08-30 15:46:39 -07:00
Yann Collet	4086b2871b	largeNbDicts compatible with multiple source files ... splitting is disabled by default, but can be re-enabled using usual command -B# update commands to look like zstd ones	2018-08-30 14:38:49 -07:00
Yann Collet	0c66a44d1b	first working test program ... measures : - compression ratio with / without dictionary - create one dictionary per block - memory budget for dictionaries - decompression speed, using one different dictionary per block current limitations : - only one file - 4K blocks only - automatic dictionary built with 4K size dictionary can be selected on command line, with -D	2018-08-28 15:47:07 -07:00
Yann Collet	b37a0a6bde	Merge pull request #1298 from facebook/bench ... Refactored bench.c	2018-08-28 12:25:02 -07:00
Yann Collet	0491037db9	Merge branch 'bench' into largeNbDicts	2018-08-28 11:26:46 -07:00
Yann Collet	55affc09de	timedFn : measurement delay is programmable ... instead of hard-coded 1 second per measurement	2018-08-28 11:26:27 -07:00
Yann Collet	d97e92dfad	Merge branch 'bench' into largeNbDicts	2018-08-27 12:12:51 -07:00
Yann Collet	01dcd0fd17	bench: minor api update, for consistency ... BMK_benchTimedFn() BMK_isCompleted_TimedFn() uses TimedFnState	2018-08-26 21:30:18 -07:00
Yann Collet	6782725155	first sketch for largeNbDicts test program	2018-08-26 19:29:12 -07:00
Yann Collet	c3a4baaf6e	fixed minor warnings ... valgrind: memory leak of a few bytes in fullbench static analyzer: uninitialized data passed as result	2018-08-24 23:25:35 -07:00
Yann Collet	2279f3d127	bench: reduce nb of return type ... runOutcome is enough removed timedFnOutcome	2018-08-24 17:28:38 -07:00
Yann Collet	6ce7b08f17	fix minor warnings ... gcc : prototype with 0 parameter must be labelled (void) visual : const property must be identical in both declaration and implementation	2018-08-24 15:59:57 -07:00
Yann Collet	4da5bdf482	fixed zstd -b speed result ... the benchmark was displaying the speed of last run instead of the best of all previous runs.	2018-08-23 18:13:49 -07:00
Yann Collet	1f9ec13621	introduced MB_UNIT ... so that all benchmarking programs use the same speed scale	2018-08-23 16:03:30 -07:00
Yann Collet	d39a25c5ed	update fullbench.c to work with new bench.h	2018-08-23 15:00:09 -07:00
Yann Collet	2e45badff4	refactored bench.c ... for clarity and safety, especially at interface level	2018-08-23 14:21:18 -07:00
Jennifer Liu	9d6ed9def3	Merge fastCover into DictBuilder (#1274) ... * Minor fix * Run non-optimize FASTCOVER 5 times in benchmark * Merge fastCover into dictBuilder * Fix mixed declaration issue * Add fastcover to symbol.c * Add fastCover.c and cover.h to build * Change fastCover.c to fastcover.c * Update benchmark to run FASTCOVER in dictBuilder * Undo spliting fastcover_param into cover_param and f * Remove convert param functions * Assign f to parameter * Add zdict.h to Makefile in lib * Add cover.h to BUCK * Cast 1 to U64 before shifting * Remove trimming of zero freq head and tail in selectSegment and rebenchmark * Remove f as a separate parameter of tryParam * Read 8 bytes when d is 6 * Add trimming off zero frequency head and tail * Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed) * Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary * Change nbDmer to always read 8 bytes even when d=6 * Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER * Update comments and benchmarking result * Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover * Add dictType enum and fix bug about passing zParam when converting to coverParam * Combine finalize and skip into a single parameter * Update acceleration parameters and benchmark on 3 sample sets * Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets * Initialize variables outside of for loop in benchmark.c * Update benchmark result for hg-manifest * Remove cover.h from install-includes * Add explanation of f * Set default compression level for trainFromBuffer to 3 * Add assertion of fastCoverParams in DiB_trainFromFiles * Add checkTotalCompressedSize function + some minor fixes * Add test for multithreading fastCovr * Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish * Free segmentFreqs * Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment * Add FASTCOVER_MEMMULT * Minor fix * Update benchmarking result	2018-08-23 12:06:20 -07:00
Yann Collet	77e805e3db	bench: changed creation/reset function to timedFnState ... for consistency	2018-08-21 18:19:27 -07:00
Yann Collet	801e3bcd97	Merge pull request #1290 from edenzik/ezik/1119-safe-strcpy-in-fileio ... Fixed unsafe string copy and concat in `fileio.c`.	2018-08-21 13:18:44 -07:00
Eden Zik	78af534f82	Fixed unsafe string copy and concat in fileio.c. ... Per warnings from flawfinder: "Does not check for buffer overflows when copying to destination [MS-banned] (CWE-120). Consider using snprintf, strcpy_s, or strlcpy (warning: strncpy easily misused).". Replaced called to strcpy and strcat in `fileio.c` to calls with a specified size (`strncpy` and `strncat`). Tested the changes on OSX, Linux, Windows. On OSX + Linux, changes were tested with ASAN. The following flags were used: 'check_initialization_order=1:strict_init_order=1:detect_odr_violation=1:detect_stack_use_after_return=1' To reproduce warning: ./flawfinder.py ./programs/fileio.c	2018-08-20 22:15:24 -04:00
Yann Collet	105677c6db	created ZSTDMT_toFlushNow() ... tells in a non-blocking way if there is something ready to flush right now. only works with multi-threading for the time being. Useful to know if flush speed will be limited by lack of production.	2018-08-17 18:11:54 -07:00
Yann Collet	09e63c58ac	fix : no longer slow down on input saturation ... only slows down when all buffers are full	2018-08-17 16:27:43 -07:00
Yann Collet	8b674d7dc7	ensured compression level is maxed at ZSTD_maxCLevel()	2018-08-17 16:01:56 -07:00
Yann Collet	b4e7f71055	Merge branch 'dev' into adapt	2018-08-17 15:54:13 -07:00
Yann Collet	42a02ab745	fixed minor warnings issued by scan-build	2018-08-15 14:36:02 -07:00
George Lu	e89f1fb45c	Fix scan-build warnings in bench.c	2018-08-14 14:44:47 -07:00
Yann Collet	3e4617ef54	frameProgression reports nbActiveWorkers and output flushed	2018-08-14 11:49:25 -07:00
Yann Collet	973a8d42c7	Merge pull request #1236 from GeorgeLu97/paramgrillconstraints ... ParamgrillConstraints	2018-08-13 15:44:50 -07:00
Yann Collet	0853f86044	adaptive mode uses default window size of 8 MB	2018-08-13 13:13:22 -07:00
Yann Collet	33f7709c71	fileio: changed parameter type from ptr to plain structure ... safer : this parameter is read-only, we don't want original structure to be modified	2018-08-13 13:02:03 -07:00
Yann Collet	f3aa510738	rateLimiter does not "catch up" when input speed is slow	2018-08-13 11:38:55 -07:00
Yann Collet	e7a49c6683	introduced command --adapt	2018-08-11 20:48:06 -07:00
Yann Collet	9d26cb6a75	slow down faster when output speed is limited	2018-08-09 17:44:30 -07:00
Yann Collet	3d7b533f68	Merge branch 'dev' into adapt	2018-08-09 15:57:36 -07:00
Yann Collet	754942cb79	fixed assert() condition	2018-08-09 15:57:19 -07:00
Yann Collet	2dd76037be	zstd cli can increase level when input is too slow	2018-08-09 15:51:30 -07:00
Yann Collet	79a35ac20d	minor code comments improvements	2018-08-09 15:16:31 -07:00
Yann Collet	51e71a5ec7	added zstdgrep documentation ... presenting `zstdgrep` limit regarding dictionary compression with workaround recommended by @tobwen (#1268)	2018-08-09 12:28:25 -07:00
George Lu	bfe8392e23	Remove ctx from benchMem	2018-08-09 12:07:57 -07:00
George Lu	8278a49cb6	const srcPtrs	2018-08-09 10:42:58 -07:00
George Lu	3d230db853	Change speed representation from floating point to integral	2018-08-09 10:42:58 -07:00
George Lu	dd270b2f75	Renaming / Style fixes	2018-08-09 10:42:58 -07:00
George Lu	e148db366e	Separate capacity vs size ... Also: Make suggested fixes -varInds_t -reorder some arguments -remove code duplication -update README / -h -Fix memory leaks	2018-08-09 10:42:58 -07:00
George Lu	df026e159f	Fix windows implicit casting bugs	2018-08-09 10:42:58 -07:00
George Lu	7b5b3d7ae3	BenchMem with block compressed sizes passed back up	2018-08-09 10:42:58 -07:00
George Lu	3adc217ea4	Total Changes: ... Add different constraint types (decompression speed, compression memory, parameter constraints) Separate search space by strategy + strategy selection Memoize results Real random restarts Support multiple files Support Dictionary inputs Debug Macro for extra printing	2018-08-09 10:42:58 -07:00
George Lu	eb21b7f482	Not crashing	2018-08-09 10:42:58 -07:00
George Lu	5f49034520	Working V1	2018-08-09 10:42:58 -07:00
George Lu	cffb6da339	Parses additional parameters ... Additional constraint checking Minor fixes more param parsing Add Memory Change paramVariation work on feasibility reformat bench Changed Paramgrill to use bench.c benchmarking customlevel macro Printing Flag Minor changes Explicit casting Makefile fix casting, type fix Printing Flag Minor Changes comments, helper fn's	2018-08-09 10:42:58 -07:00
Yann Collet	5808027abf	Merge branch 'dev' into fix1241	2018-08-03 16:08:33 -07:00
Yann Collet	2fdab1629b	fix unused variable warning	2018-08-03 08:30:01 -07:00
Yann Collet	5203f01774	fix : zstd cli can be built with build macro ZSTD_NOBENCH ... which disables bench.c module	2018-08-03 07:54:29 -07:00
cyan4973	3f535007e4	fix %zu support under minGW ... and relevant test on Appveyor	2018-07-30 16:56:18 +02:00
George Lu	09ccd977c3	no zero	2018-07-26 15:17:58 -07:00
Yann Collet	effa84c8d1	Merge pull request #1230 from terrelln/train-out ... zstdcli: Allow -o before --train	2018-07-18 16:34:10 +02:00
Nick Terrell	4e706d7f2c	fileio: Error in compression on read errors ... We can write a corrupted file if the input file errors during a read. We should return a non-zero error code in this case.	2018-07-17 15:26:30 -07:00
Nick Terrell	58b8219475	zstdcli: Allow -o before --train ... Only set the default value if `outFileName` is unset. Fixes #1227.	2018-07-16 12:45:34 -07:00
Nick Terrell	45821fac0c	Merge pull request #1225 from jennifermliu/dev ... Split samples when building dictionary for COVER	2018-07-13 13:26:15 -07:00
Jennifer Liu	612b346ed5	Add explanation for split=100	2018-07-11 15:50:28 -07:00
Jennifer Liu	5021441d86	Change default splitPoint to 100	2018-07-10 11:19:33 -07:00
Jennifer Liu	bfad1af031	Update doc for split==100	2018-07-05 11:05:31 -07:00
Jennifer Liu	0881184c89	Some edits based on pull request comments	2018-07-03 17:53:27 -07:00
Yann Collet	689bfecd48	Merge pull request #1188 from GeorgeLu97/BenchModule ... Bench module	2018-07-02 13:33:27 -07:00
Jennifer Liu	8afcb8eea7	Update documentation	2018-07-01 19:59:37 -07:00
Jennifer Liu	84e8b2a305	Fix another declaration issue	2018-06-29 18:02:02 -07:00
Jennifer Liu	348e5f77a9	Add split=# to cli	2018-06-29 17:54:41 -07:00
Yann Collet	b5207aadfa	make build tests more unforgiving ... `-Werror` will ensure they fail if there is the slightest warning. fix a minor warning specific to `zstd_decompress` variant.	2018-06-29 17:10:56 -07:00
W. Felix Handte	712a9fd972	Allow Invoking zstd --list When stdin is not a tty ... Also now returns an error when no inputs are given. New proposed behavior: ``` felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l; echo $? No files given 1 felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l Makefile.zst; echo $? Frames Skips Compressed Uncompressed Ratio Check Filename 1 0 3.08 KB 10.92 KB 3.544 XXH64 Makefile.zst 0 felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l <Makefile.zst; echo $? zstd: --list does not support reading from standard input No files given 1 felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l Makefile.zst <Makefile.zst; echo $? Frames Skips Compressed Uncompressed Ratio Check Filename 1 0 3.08 KB 10.92 KB 3.544 XXH64 Makefile.zst 0 felix@odin:~/prog/zstd (list-stdin-check)$ ```	2018-06-29 15:33:44 -04:00
Yann Collet	a2c3a4cd0e	Merge pull request #1214 from jennifermliu/dev ... Make --fast=0 fail	2018-06-27 18:53:39 -07:00
Yann Collet	1fd621ff6d	minor man page update ... regarding advanced parameter `tlen` which was recently changed. `0` in association with `ZSTD_fast` now means "normal fast mode".	2018-06-27 18:49:02 -07:00
Jennifer Liu	1ab57a7ce1	Redirect failed test result to INTOVOID and update comment about parsing fast command	2018-06-27 16:27:45 -07:00
Jennifer Liu	aef8486fee	Make fast=0 fail	2018-06-27 14:27:27 -07:00
cyan4973	f741fb8fcd	minor fixes for MSYS2 compilation	2018-06-26 01:22:45 -07:00
George Lu	50d612f4f0	Interleave compression/decompression ... Fix Bugs	2018-06-25 15:01:03 -07:00
George Lu	d6121ad0e1	Opaque State ... And minor fixups (comments/alignment/checks/fix memory leak)	2018-06-25 08:07:43 -07:00
George Lu	ab26f24c9c	benchFunction Timed Wrappers ... Add BMK_benchFunctionTimed Add BMK_init_customResultCont.. Change benchMem to use benchFunctionTimed Minor Fixes/Adjustments	2018-06-21 16:23:55 -07:00
George Lu	a8eea99ebe	Incremental Display + Fn Separations ... Seperate syntheticTest and fileTableTest (now renamed as benchFiles) Add incremental display to benchMem Change to only iterMode for benchFunction Make Synthetic test's compressibility configurable from cli (using -P#)	2018-06-21 16:23:18 -07:00
Yann Collet	93702a7a62	Merge pull request #1198 from facebook/msdebug ... made Visual Studio compatible with DEBUGLEVEL >= 2	2018-06-20 12:26:31 -07:00
cyan4973	ae0b7ffa0a	made Visual Studio compatible with DEBUGLEVEL >= 2	2018-06-20 09:45:02 -07:00
Yann Collet	6768cf53fd	Merge pull request #1190 from terrelln/ldm-adjust ... Adjust advanced parameters to source size	2018-06-19 14:40:56 -07:00
Yann Collet	c0b6ce95b1	Merge pull request #1179 from supertopher/dev ... Improves UX for --list command's lack of support for pipes	2018-06-19 14:36:30 -07:00
Nick Terrell	1d0fcde45d	Use debug.h in fileio.c	2018-06-18 15:51:21 -07:00
Nick Terrell	3841dbac84	Adjust advanced parameters to source size ... In the new advanced API, adjust the parameters even if they are explicitly set. This mainly applies to the `windowLog`, and accordingly the `hashLog` and `chainLog`, when the source size is known.	2018-06-18 15:49:31 -07:00
George Lu	a3c8b59990	Fix cli no print ... Change looping behavior to match old	2018-06-18 15:38:14 -07:00
George Lu	e482e328cd	Reorder Arguments ... make initFn nullable	2018-06-18 13:21:42 -07:00
George Lu	0d1ee22990	Requested Changes ... Add Comment Simplify Interface (Remove resultSet) Reorder Arguments Remove customBench displayLevel Reorder bench.h Change benchFiles return type to match advanced Rename stuff	2018-06-18 12:01:12 -07:00
George Lu	8522346322	Make Fullbench use new function ... Rearrange Args Add nothing function Use new function, change locals to match New Display Comment cleanup Change builds	2018-06-15 11:37:49 -04:00
George Lu	20f4f32379	Add to bench ... -Remove global variables -Remove gv setting functions -Add advancedParams struct -Add defaultAdvancedParams(); -Change return type of bench Files -Change cli to use new interface -Changed error returns to own struct value -Change default compression benchmark to use decompress_generic -Add CustomBench function -Add Documentation for new functions	2018-06-14 14:23:24 -04:00
Topher Lubaway	6bca3fb4bf	Reduce noise in diff ... putting the code block back on the exact line it came from	2018-06-13 14:32:59 -07:00
Topher Lubaway	ec24f98cca	Removes duplicate IS_CONSOLE from PR ... I misunderstood that this function was included already	2018-06-13 13:39:23 -07:00
Yann Collet	c986dbf241	Merge pull request #1168 from GeorgeLu97/paramgrillfeatures ... Have paramgrill share bench.c benchmarking function	2018-06-13 11:38:29 -04:00
George Lu	01d940b670	Requested changes ... -Remove g_displaylevel/setNotificationLevel function -Add extern "C" -Remove averaging -Reorder arguments More fixes -Added BMK_return_t (result + possible error) -Correct comment' -Nullcheck ctx, dctx when allocated -Remove extra assert	2018-06-12 17:02:44 -04:00
Topher Lubaway	b024e1e1f4	Keep windows specific headers ... Accidentially deleted this existing windows only header	2018-06-12 10:16:27 -07:00
Topher Lubaway	88ae51acb3	Multi-OS support for --list detecting stream input ... IS_CONSOLE stolen wholesale from Options.cpp not sure if i should have extracted that code for DRY-ness tested in OSX and functionality seems appropriate unstested in a windows environment	2018-06-12 07:59:17 -07:00
Topher Lubaway	881defaeb3	Only check for tty in non-windows environments ... unistd.h is for unix standard tools. There does not appear to be a simple isatty for windows this we only run the logic and header include in non-windows environments	2018-06-11 15:26:35 -07:00
Topher Lubaway	5ca1d5c6f4	Properly brackets isatty if statement ... ¯\_(ツ)_/¯ this is my first commit in c	2018-06-11 12:19:15 -07:00
Topher Lubaway	4c16608e3c	Improves UX for --list command's lack of support for pipes ... --list does not support piped input This checks for a terminal and exits 1 with a well formatted error message if the STDIN is not from a terminal	2018-06-11 10:13:00 -07:00
Ryan Schmidt	b567ce9d68	Fix name of macOS	2018-06-09 14:31:17 -05:00
George Lu	0e808d608b	Make paramgrill use bench.c benchmarking	2018-06-08 12:01:05 -07:00
Yann Collet	d3615c28db	Merge pull request #1159 from GeorgeLu97/suffixlist ... Unknown Suffix Error	2018-06-01 14:00:10 -07:00
George Lu	8984cc93d6	update display	2018-05-31 18:04:05 -07:00
George Lu	547096d672	update man	2018-05-31 18:03:52 -07:00
George Lu	c9b1068298	removed strcats	2018-05-31 17:47:29 -07:00
George Lu	5ff30fe2e5	Unknown Suffix Error ... Changed so only compiled formats are printed in list of supported extensions	2018-05-31 16:13:36 -07:00
George Lu	140f59d38e	Added --format=zstd ... title	2018-05-31 15:29:35 -07:00
Yann Collet	174bd3d4a7	Merge pull request #1131 from facebook/zstdcli ... minor: control numeric argument overflow	2018-05-14 11:53:58 -07:00
Yann Collet	9cd5c63771	cli: control numeric argument overflow ... exit on overflow backported from paramgrill added associated test case	2018-05-12 14:29:33 -07:00
Yann Collet	b824d213cb	fix #1115	2018-05-12 10:21:30 -07:00
cyan4973	62487b5e76	fixed decoding bogus lz4 frame ... FIO would keep presenting data after an LZ4F decoding error resulting in a NULL pointer dereference when associated with older liblz4 version (< v1.8.1.2)	2018-04-23 18:50:16 -07:00
Yann Collet	1da629f2ad	Merge pull request #1104 from terrelln/fast-train ... Allow negative compression levels in training	2018-04-09 14:16:20 -07:00
Nick Terrell	569e2abccd	Allow negative compression levels in training ... * Set `dictCLevel` in `zstdcli.c`. * Only set to default level if the compression level `== 0`, not `<= 0`.	2018-04-09 12:12:03 -07:00
Björn Ketelaars	e5ea8d272a	fix typo in programs/zstd.{1,1.md} ... s/nodictID/no-dictID/g	2018-04-05 06:44:46 +02:00
Yann Collet	7188862d32	Merge pull request #1086 from hagemt/hagemt-patch-1 ... Correct small typo in manual (man file and markdown)	2018-03-30 20:45:10 -06:00
Tor E Hagemann	c7a5e60bc6	Update zstd.1.md	2018-03-30 15:25:32 -07:00
Tor E Hagemann	292d370ab4	Update zstd.1	2018-03-30 14:53:57 -07:00
Yann Collet	525f3fab33	restored ability to manually set overlapLog	2018-03-28 11:33:41 -06:00
Yann Collet	01082a39bd	restored simple status line during zstd compression ... the more advanced one, featuring amount of data buffered, is triggered on `-v`.	2018-03-22 17:49:46 -07:00
Yann Collet	153bc1c004	removed limit ZSTD_TARGETLENGTH_MAX ... this makes it possible to specify extremely large negative compression levels, achieving the side effect as "no compression". It will also be possible to define larger targetlength for ultra compression mode. There is no adverse side effect due to removing this limit.	2018-03-21 15:50:05 -07:00
Yann Collet	353117c5d7	implemented ZSTD_DCtx_loadDictionary*() ... this required updating ZSTD_createDDict_advanced() to accept a dictContentType parameter (raw, full, auto).	2018-03-20 13:40:29 -07:00
Yann Collet	4c5cbac179	Merge pull request #1041 from facebook/fasterFast ... Negative compression levels	2018-03-13 21:32:46 -07:00
Yann Collet	bd7bb94361	Merge pull request #1044 from baldurk/remove-utf8-characters ... Remove non-ASCII characters in header file comments	2018-03-13 13:22:07 -07:00
Baldur Karlsson	430a2fec19	Remove non-ASCII characters in header file comments ... * Replaced a non-breaking space and an en dash with a plain space and a hyphen. * This means the files are simple ASCII and less likely to run into codepage issues.	2018-03-13 20:05:53 +00:00
Jesse Talavera-Greenberg	2f70fbf2a3	Made -H's printout specify the semantics of -T0	2018-03-12 20:43:32 -04:00
Yann Collet	a57d43d4d4	updated documentation of targetLength	2018-03-12 11:35:01 -07:00
Yann Collet	f24566b597	minor bench improvements ... - do not test level 0, as it is converted into level 3, which feels strange when compressing multiple levels - Use direct synchronous mode when a single worker is requested.	2018-03-12 04:02:57 -07:00
Yann Collet	6a9b41b731	create command --fast[=#] ... access negative compression levels from command line for both compression and benchmark modes. also : ensure proper propagation of parameters through ZSTD_compress_generic() interface. added relevant cli tests.	2018-03-11 20:01:23 -07:00
Yann Collet	a70f7e10fa	Merge branch 'benchDecode' into longOffsetMode	2018-03-05 14:09:00 -08:00
Yann Collet	03e7e14192	fix benchmark issue when measuring only decoding speed ... zstd bench module can focus on decompression speed _only_. This is useful when trying to measure performance on large input data compressed using a high level as compression time becomes problematic (too long). This mode is triggered by command : zstd -b -d Problem was : in such a mode, measured decoding speed was > 10% slower than in nominal mode (compression + decompression), making decompression benchmark mode much less useful. This patch fixes the issue. It's not completely clear why, but moving the `memcpy()` operation sooner in the pipeline fixed it. I can still measure some difference, but it is in the < 2% range, so it's much more tolerable. also : it doesn't matter anymore in which order are selected commands `-b` and `-d`. The combination always triggers bench_decodeOnly mode.	2018-03-05 13:57:41 -08:00
Yann Collet	41bd10446e	Merge branch 'dev' into longOffsetMode	2018-03-05 13:10:10 -08:00
Yann Collet	b91ddf0ae6	Merge branch 'dev' into longOffsetMode	2018-03-05 11:59:54 -08:00
Conrad Meyer	606374269c	FIO_addFInfo: Fully initialize output 'total' struct ... Silence a Coverity warning about 'windowSize' being uninitialized. (Yes, nothing that calls this routine actually uses the windowSize value. Still, appeasing Coverity is pretty harmless in this case.)	2018-02-28 15:23:05 -08:00
Yann Collet	25d00d10fc	fixed minor conversion warning	2018-02-20 16:52:28 -08:00
Yann Collet	3538a535bf	use TIMELOOP_NANOSEC ... as suggested by @terrelln	2018-02-20 15:33:56 -08:00
Yann Collet	d3364aa39e	improve benchmark measurement for small inputs ... by invoking time() once per batch, instead of once per compression / decompression. Batch is dynamically resized so that each round lasts approximately 1 second. Also : increases time accuracy to nanosecond	2018-02-20 14:58:40 -08:00
Yann Collet	5cb1144872	fixed --single-thread ... was incorrectly set to -T0 (use as many cores as possible) previously	2018-02-13 14:56:35 -08:00
Yann Collet	04a3f85ce7	fixed gcc warning on a switch code path	2018-02-09 16:16:27 -08:00
Yann Collet	75689838e4	specify new command --single-thread	2018-02-09 15:55:41 -08:00
Yann Collet	4beaeaace5	Merge branch 'dev' into flexibleLevel	2018-02-09 09:15:05 -08:00
Yann Collet	4b525af53a	zstdmt: applies new parameters on the fly ... when invoked from ZSTD_compress_generic()	2018-02-02 15:58:13 -08:00
Yann Collet	90eca318a7	fileio: create dedicated function to generate zstd frames ... like other formats	2018-02-02 14:24:56 -08:00
Yann Collet	549d26ae71	Merge pull request #1005 from systemcrash/dev ... Update zstd.1	2018-02-02 10:04:40 -08:00
Yann Collet	6c492af284	fixed minor conversion warning	2018-02-01 20:16:00 -08:00
Yann Collet	209df52ba2	Changed nbThreads for nbWorkers ... This makes it easier to explain that nbWorkers=0 --> single-threaded mode, while nbWorkers=1 --> asynchronous mode (one mode thread on top of the "main" caller thread). No need for an additional asynchronous mode flag. nbWorkers>=2 works the same as nbThreads>=2 previously.	2018-02-01 19:29:30 -08:00
Yann Collet	4b6a94f0cc	clarified comments on LDM parameters	2018-02-01 17:07:27 -08:00
Yann Collet	2bfc79ab8d	removed bitstream.h dependency	2018-02-01 16:13:04 -08:00
Yann Collet	823a28a1f4	Merge pull request #1000 from facebook/progressiveFlush ... Progressive flush	2018-01-30 22:49:47 -08:00
systemcrash	d13a75c969	Update zstd.1	2018-01-29 18:38:02 +01:00
Yann Collet	9f8ed23b5b	bumped version number to v1.3.4 ... also added a paragraph on using compression level with training mode as this is a recurrent question (see for example #1004)	2018-01-27 22:23:26 -08:00
ne-sted	50aea2f293	cli: fix align of defaults	2018-01-24 15:07:22 +02:00
Yann Collet	cb5eba8e20	add zcat symlink support, suggested by @wtarreau ... added some test also updated relevant doc + fixed a mistake in `lz4` symlink support : lz4 utility doesn't remove source files by default (like zstd, but unlike gzip). The symlink must behave the same.	2018-01-19 11:26:35 -08:00
Yann Collet	70f81d6030	zstdmt uses POOL_tryAdd() to call a new worker ... so that it's no longer a blocking call. This makes it possible to stream out data gradually, while waiting for a worker to become available.	2018-01-19 10:01:40 -08:00
Yann Collet	4d08ba8b77	fileio: READY_FOR_UPDATE() is now a function-like macro ... as suggested by @terrelln	2018-01-18 11:27:13 -08:00
Yann Collet	aa79c18e3f	fixed a few access contention ... passes thread sanitizer test	2018-01-17 17:18:19 -08:00
Yann Collet	394eec697b	Introduce ZSTD_getFrameProgression() ... Produces 3 statistics for ongoing frame compression : - ingested - consumed (effectively compressed) - produced Ingested can be larger than consumed due to buffering effect. For the time being, this patch mostly fixes the % ratio issue, since it computes consumed / produced, instead of ingested / produced. That being said, update is not "smooth", because on a slow enough setting, fileio spends most of its time waiting for a worker to complete its job. This could be improved thanks to more granular flushing i.e. start flushing before ongoing job is fully completed.	2018-01-17 16:39:02 -08:00
Yann Collet	58dd7de640	zstdmt: fixed an endless loop on allocation failure ... this happened on 32-bits build when requiring a too large input buffer, typically on wlog=29, creating jobs of 2 GB size. also : zstd32 now compiles with multithread support enabled by default (can be disabled with HAVE_THREAD=0)	2018-01-17 12:10:15 -08:00
Yann Collet	3e1e57db27	fix fileio progression status update ... The compression % is no longer correct, since it's no longer possible to make direct correlation between nb bytes read and nb bytes written due to large internal buffer inside CCtx (exacerbated with --long). The current "fix" is to no longer display the %. A more complex solution will have to count exactly how much data has been consumed and compressed internally, within CCtx buffers.	2018-01-16 17:35:00 -08:00
Yann Collet	10c213761a	cli: fix for no-MT mode ... when cli is compiled without MT support, invoking ZSTD_p_nonBlockingMode result in an error code. This patch only sets ZSTD_p_nonBlockingMode when ZSTD_MULTITHREAD is set, meaning there is MT support. The error code could also be intentionnally ignored (there is no side effect).	2018-01-16 17:28:11 -08:00
Yann Collet	1dba98d563	introduced parameter ZSTD_p_nonBlockingMode ... This new parameter makes it possible to call streaming ZSTDMT with a single thread set which is non blocking. It makes it possible for the main thread to do other tasks in parallel while the worker thread does compression. Typically, for zstd cli, it means it can do I/O stuff. Applied within fileio.c, this patch provides non-negligible gains during compression. Tested on my laptop, with enwik9 (1000000000 bytes) : time zstd -f enwik9 With traditional single-thread blocking mode : real 0m9.557s user 0m8.861s sys 0m0.538s With new single-worker non blocking mode : real 0m7.938s user 0m8.049s sys 0m0.514s => 20% faster	2018-01-16 16:15:47 -08:00
Yann Collet	58ecf13e02	zstdmt : can compress at block granularity ... offering perspective of more accurate progression report.	2018-01-13 13:18:57 -08:00
Yann Collet	1edf33764e	Merge pull request #974 from terrelln/dstfile ... [fileio] Improve safety of output file modifications	2018-01-10 19:02:48 +01:00
Yann Collet	752880ffed	Merge pull request #963 from facebook/benchfix ... fix: bench can accept hlog custom parameter	2018-01-06 06:57:02 +01:00
Nick Terrell	ed9611dc62	[fileio] Don't call FIO_remove() on stdout or /dev/null	2018-01-05 11:50:24 -08:00
Nick Terrell	282ad05e0a	[fileio] Use FIO_remove() everywhere for safety	2018-01-05 11:44:45 -08:00
Nick Terrell	fd63140e1c	[util] Refuse to set file stat on non-regular file	2018-01-05 11:44:25 -08:00
Pádraig Brady	e0596715dc	zstd: fix crash when not overwriting existing files ... This fixes the following crash: $ touch exists $ programs/zstd -r examples/ -o exists zstd: exists already exists; not overwritten Segmentation fault (core dumped) * programs/fileio.c (FIO_compressMultipleFilenames): Handle the case where we're not overwriting the destination. Reported at https://bugzilla.redhat.com/1530049	2018-01-02 15:24:09 +00:00
Yann Collet	c707c6e9f2	fix: bench can accept hlog custom parameter ... was ignored during initialization	2017-12-27 13:32:05 +01:00
Yann Collet	cc9e026866	Merge pull request #952 from terrelln/merge-end ... [fileio] Merge end loop for small optimization	2017-12-15 10:27:53 -08:00
Yann Collet	2cff66b62f	version bump to v1.3.3	2017-12-14 16:11:20 -08:00
Nick Terrell	f48d34edba	[fileio] Merge end loop for small optimization	2017-12-14 15:52:24 -08:00
Yann Collet	a0ac8c895c	Merge pull request #950 from facebook/srcSizeAdaptation ... fix adaptation on srcSize	2017-12-14 14:48:31 -08:00
Yann Collet	2e97a6d464	fixed minor declaration-after-statement warning	2017-12-13 18:50:05 -08:00
Yann Collet	5432ef6921	fixes adaptation on srcSize ... This patch restores capability for each file to receive adapted compression parameters depending on its size. The bug breaking this feature was relatively silly : setting a parameter with a value "0" is supposed to be a no-op. Unfortunately, it would pin down compression parameters as if they were manually set, preventing later automatic adaptation. Unfortunately, I'm currently short of a test case that could check this situation and trigger an error. Compression parameters selection between tableID 0,1,2,3 is largely internal, leaving no trace to outside world, not even in frame header.	2017-12-13 17:45:26 -08:00
Nick Terrell	4680e85bdf	Allow -o with multiple files	2017-12-13 17:44:34 -08:00
Yann Collet	4d0dfafa7b	Merge pull request #949 from terrelln/rrm ... [fileio] Refuse to remove non-regular file	2017-12-13 17:36:39 -08:00
Nick Terrell	82bc8fe0cc	[fileio] Refuse to remove non-regular file	2017-12-13 13:38:26 -08:00
Nick Terrell	b5e7f6c0f3	[fileio] Fix window size MB calculation ... Test command: ``` head -c 10000 /dev/zero | ./zstd -c --zstd=wlog=12 | ./zstd -M2048 -t ```	2017-12-13 10:57:01 -08:00
Yann Collet	31293330d0	It's still necessary to check PLATFORM_POSIX_VERSION for clock_gettime() ... glibc/uclibc is not enough	2017-12-04 16:31:59 -08:00
Yann Collet	0097469238	removed a few redundant #include	2017-12-04 16:02:42 -08:00
Yann Collet	e46194bbf9	fix #911 : changed detection macro for clock_gettime() ... The new macro might be a bit too restrictive. Systems which do not support new test will simply default to <time.h>'s `clock_t clock()`, suffering lesser benchmark accuracy. Should it matter, the detection macro will have to be upgraded.	2017-12-04 15:57:01 -08:00
Yann Collet	55faa5492d	fileio: fixed LZ4F invocation from assert()	2017-12-04 11:26:59 -08:00
Yann Collet	af2fbbcb0d	Merge pull request #939 from facebook/shorterCircleCI ... Faster CircleCI tests	2017-12-04 11:22:30 -08:00
Yann Collet	71f012e5bf	zstdcli: fixed minor warning when bench module not enabled ... one variable defined but not used	2017-12-01 17:42:46 -08:00
Yann Collet	a1b24e6262	Merge pull request #938 from terrelln/time ... Use util.h for timing	2017-12-01 16:40:38 -08:00
Nick Terrell	dab8cfa3c7	Combine definitions of SEC_TO_MICRO	2017-11-30 19:40:53 -08:00
Nick Terrell	9a2f6f477b	Use util.h for timing	2017-11-30 14:57:25 -08:00
Yann Collet	2f22a6ec50	Merge branch 'dev' into opt3	2017-11-28 15:03:58 -08:00
Yann Collet	0a0a212934	zstd_opt: changed cost formula ... There was a flaw in the formula which compared literal cost with match cost : at a given position, a non-null literal suite is going to be part of next sequence, while if position ends a previous match, to immediately start another match, next sequence will have a litlength of zero. A litlength of zero has a non-null cost. It follows that literals cost should be compared to match cost + litlength==0. Not doing so gave a structural advantage to matches, which would be selected more often. I believe that's what led to the creation of the strange heuristic which added a complex cost to matches. The heuristic was actually compensating. It was probably created through multiple trials, settling for best outcome on a given scenario (I suspect silesia.tar). The problem with this heuristic is that it's hard to understand, and unfortunately, any future change in the parser would impact the way it should be calculated and its effects. The "proper" formula makes it possible to remove this heuristic. Now, the problem is : in a head to head comparison, it's sometimes better, sometimes worse. Note that all differences are small (< 0.01 ratio). In general, the newer formula is better for smaller files (for example, calgary.tar and enwik7). I suspect that's because starting statistics are pretty poor (another area of improvement). However, for silesia.tar specifically, it's worse at level 22 (while being better at level 17, so even compression level has an impact ...). It's a pity that zstd -22 gets worse on silesia.tar. That being said, I like that the new code gets rid of strange variables, which were introducing complexity for any future evolution (faster variants being in mind). Therefore, in spite of this detrimental side effect, I tend to be in favor of it.	2017-11-28 14:07:03 -08:00
W. Felix Handte	baff9dd15e	Fix LZ4 Compression Buffer Overflow ... Fixes issue where, when `zstd --format=lz4` is fed an input larger than 128KB, the read overruns the input buffer. This changes Zstd to use LZ4 with chained 64KB blocks. This is technically a breaking change in that some third party LZ4 implementations may not support linked blocks. However, progress should not be allowed to be stopped by such petty concerns as backwards compatibility!	2017-11-28 12:07:26 -05:00
Yann Collet	743b23878e	install: changed variable MANDIR into MAN1DIR ... MANDIR still exists, and is now the parent of MAN1DIR	2017-11-27 13:47:35 -08:00
Yann Collet	2fd765498a	updated man page ... following patch #931 by @scottchiefbaker	2017-11-24 17:20:54 -08:00
Yann Collet	c857ee850a	minor update	2017-11-24 16:44:28 -08:00
Scott Baker	31a191b178	Include information about the benchmark output/methodology ... Addresses #930	2017-11-22 20:34:25 -08:00
Yann Collet	daebc7fe26	bench: slightly adjusted display format ... adapt accuracy depending on value. makes it possible to have higher accuracy for small value, notably small compression speed. This capability is expected to be useful while modifying optimal parser.	2017-11-18 15:54:32 -08:00
Nick Terrell	a6052af0e8	[zstd] Fix rare bug with signal handler	2017-11-17 16:38:56 -08:00
Yann Collet	5b957ba899	minor interface adjustments	2017-11-17 01:21:40 -08:00
Yann Collet	d898fb7ba6	bench: added cli command -S to benchmark multiple files separately ... Currently, all files are joined by default, they are compressed separately but benchmarked together, providing a single final result. Benchmarking files separately make it possible to accurately measure difference for each file. This is expected to be useful while tuning optimal parser.	2017-11-17 00:22:55 -08:00
Yann Collet	8accfa7fcc	bench: realTime is a global parameter ... like most parameters not directly related to compression	2017-11-17 00:02:37 -08:00
Yann Collet	9a11f70dc3	merged repcode search into BT match search ... this version has same speed as branch `opt` which is itself 5-10% slower than branch `dev` (no identified reason) It does not compress exactly the same as `opt` or `dev`, maybe because it doesn't stop search after repcodes, leading to sometimes better compression, sometimes worse (by a small margin). warning : _extDict path does not work for the time being This means that benchmark module works, but file module will fail with large files (and high compression level). Objective is to fuse _extDict path into current one, in order to have a single parser to maintain.	2017-11-13 02:23:48 -08:00
Yann Collet	6f1dfa8adf	removed line with // comment ... this is for a different topic (better parameter adaptation for small files + dictionary and/or custome parameters)	2017-11-01 17:01:45 -07:00
Yann Collet	428e8b3bf4	fix : ZSTD_compress_generic(,,,ZSTD_e_end) automatically sets pledgedSrcSize ... as per documentation, on ZSTD_setPledgedSrcSize() : > If all data is provided and consumed in a single round, > this value (pledgedSrcSize) is overriden by srcSize instead. This wasn't applied before compression level is transformed into compression parameters. As a consequence, small input missed compression parameters adaptation. It seems to work fine now : compression was compared with ZSTD_compress_advanced(), results were the same.	2017-11-01 13:15:23 -07:00
Nick Terrell	b495140f67	Update BUCK files ... * Correct XXH namespace (Fixes #901) * Multithreading always enabled * GZIP/LZ4/LZMA always enabled * Legacy support always fully enabled	2017-10-25 12:47:57 -07:00
Yann Collet	91535d71ec	fixed missing zstdmt_compress.h dependency ... we lose a warning message : when a job size is chosen < minimum job size for multithreading, it is automatically resized to minimum size. If this information is really useful, it should be present in zstd.h now.	2017-10-19 12:09:34 -07:00
Yann Collet	eac42534fe	bench: fixed Visual warning regarding struct initialization ... also : removed dependency on zstdmt_compress.h removed several unused macros fileio : small code refactoring to reduce some variable scope	2017-10-19 11:56:14 -07:00
Yann Collet	d3b9547aa4	IO and bench : ZSTD_NEWAPI is the only remaining code path ... removed the other 2 code paths (single thread, and ZSTDMT ones) keeping only the new advanced API, for easier code coverage. It shall also fix identified issue with Visual Studio which doesn't have ZSTD_NEWAPI defined.	2017-10-18 17:01:53 -07:00
Yann Collet	300e1df0a3	fixed wrong test to display compression status	2017-10-18 11:41:52 -07:00
Yann Collet	18b795374a	UTIL_getFileSize() returns UTIL_FILESIZE_UNKNOWN on failure ... UTIL_getFileSize() used to return zero on failure. This made it impossible to distinguish a failure from a genuine empty file. Both cases where coalesced. Adding UTIL_FILESIZE_UNKNOWN constant has many consequences on user code, since in many places, the `0` was assumed to mean "error". This is no longer the case, and the error code must be actively checked.	2017-10-17 16:14:25 -07:00
Yann Collet	32c9f715ae	fixed : Visual build compressing stdin with multi-threading enabled fails ... It was multiple reasons stacked : - Visual use a different code path, because ZSTD_NEWAPI is not defined - fileio.c sends `0` as `pledgedSrcSize` to mean `ZSTD_CONTENTSIZE_UNKNOWN` (fixed) - ZSTDMT_resetCCtx() interpreted `0` as "empty" instead of "unknown" (fixed)	2017-10-17 14:07:43 -07:00
Yann Collet	fc8d293460	dictionary compression use correct file size estimation ... when determining compression parameters to compress one file only. For multiple files, it still "bets" that files are going to be small. There was also a bug recently added in ZSTD_CCtx_loadDictionary_advanced() making it incapable to use pledgedSrcSize to determine compression parameters.	2017-10-14 01:21:43 -07:00
Yann Collet	9ef32b3cf1	minor : zstd -l -v display each file name	2017-10-14 00:02:32 -07:00
Yann Collet	dd18d73e7e	fileio: content size is enabled by default	2017-10-13 16:32:18 -07:00
Nick Terrell	6dd958eea2	[zstdcli] Add window size to verbose list ... ``` > zstd --list -v file1 file2 file3 *** zstd command line interface 64-bits v1.3.2, by Yann Collet *** Window Size: 512.00 KB (524288 B) Compressed Size: 0.02 KB (19 B) Check: XXH64 Window Size: 8192.00 KB (8388608 B) Compressed Size: 0.02 KB (19 B) Check: XXH64 Window Size: 512.00 KB (524288 B) Compressed Size: 0.01 KB (15 B) Check: None ```	2017-10-04 12:26:28 -07:00
Yann Collet	3b27ed41fd	Merge branch 'srcSize' into dev	2017-10-02 16:34:14 -07:00
Yann Collet	4946993f87	removed isRegularFile parameter ... no longer useful : size of src is determined for each file.	2017-10-02 12:29:25 -07:00
Yann Collet	7f580f9ee8	interruption handler and variable are static	2017-10-02 11:39:05 -07:00
Yann Collet	fe5444bc66	removed the statement for all versions of Visual Studio	2017-10-02 02:02:16 -07:00
Yann Collet	51d82d5516	same error in Visual Studio 2012 ...	2017-10-02 01:12:40 -07:00
Yann Collet	ed7ae4c9bd	The issue also impacts Visual Studio 2010	2017-10-02 00:45:28 -07:00
Yann Collet	6e7ba3df2f	added (void)sig to avoid compilers complaining that sig is not used.	2017-10-02 00:19:47 -07:00
Yann Collet	82bc200f82	conditionnally removed invocation that generates a buggy warning with Visual Studio 2008	2017-10-02 00:02:24 -07:00
Yann Collet	bd18095edc	blindfix for Visual : minor casting issue ... should not happen since SIGIGN is provided by <signal.h>, so it should work "ouf of the box"	2017-10-01 15:32:48 -07:00
Yann Collet	00fc1ba8dd	cli: add Ctrl-C support, requested by @mike155 in #854 ... Now, pressing Ctrl-C during compression or decompression will erase operation artefact (unfinished destination file) before leaving execution.	2017-10-01 12:10:26 -07:00
Yann Collet	e580dc6a4a	Merge pull request #860 from felixhandte/zstd-lz4-support-tests ... Add Default LZ4 Support When Available	2017-09-29 22:32:54 -07:00
Yann Collet	8afb151c9b	cli: fixed wrong initialization in MT mode ... It's not good to mix old and new API ZSTD_resetCStream() doesn't just set pledgedSrcSize : it also sets the CCtx for a single thread compression. Problem is, when 2+ threads are defined in cctx->requestedParams, ZSTD_compress_generic() will want to start MT compression, since initialization is supposed to have already happened (thanks to ZSTD_resetCStream()) except that the underlying ZSTDMT_CCtx* object is not created, resulting in a segfault. This is an invalid construction (correct one is to use ZSTD_CCtx_setPledgedSrcSize()). I haven't found a nice way to mitigate this impact if someone makes the same mistake. At some point, removing the old API to keep only the new API within fileio.c will limit these risks.	2017-09-29 22:14:37 -07:00
Yann Collet	fbd5ab7027	minor fix : no longer use fake srcSize during resource creation ... srcSize is read and provided at each file, not at resource creation. This used to be useful with older API, because it could not re-adapt parameters between sessions. At some point, it will be better to remove the old code, and only keep the new_api. It works fine by now.	2017-09-29 19:40:27 -07:00
Yann Collet	db1668a43b	fix : srcSize written in frame header when multiple files compressed ... This information used to be disabled when nbFiles>1. It was badly initialized later in the code, resulting in an error.	2017-09-29 18:05:18 -07:00
Yann Collet	8afcc80e07	decode more data before triggering error ... fixes #874 : when a frame is not properly terminated by a "last block" signal, zstd -d used to detect it immediately and error out. This version will decode and flush the last block, and only then issue an error.	2017-09-29 15:54:09 -07:00
W. Felix Handte	dc27c36495	Update documentation to reflect other format support	2017-09-28 19:43:12 -04:00
W. Felix Handte	d0519d4b0c	Add CLI Program Name Detection for LZ4	2017-09-28 19:18:15 -04:00
Yann Collet	54a827fff0	Merge branch 'dev' into newFormats ... Fixed conflicts in zstdmt_compress.c	2017-09-27 16:39:40 -07:00
Yann Collet	3182ea2e64	Merge pull request #866 from facebook/list ... improved --list display	2017-09-27 16:34:29 -07:00
Nick Terrell	c233bdbaee	Increase maximum window size ... * Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail on my Mac. * Maximum window size in 64-bit mode is 2GB, since that is the largest power of 2 that works with the overflow prevention. * Allow `--long=windowLog` to set the window log, along with `--zstd=wlog=#`. These options also set the window size during decompression, but don't override `--memory=#` if it is set. * Present a helpful error message when the window size is too large during decompression. * The long range matcher defaults to a hash log 7 less than the window log, which keeps it at 20 for window log 27. * Keep the default long range matcher window size and the default maximum window size at 27 for the API and CLI. * Add tests that use the maximum window size and hash size for compression and decompression.	2017-09-26 14:00:01 -07:00
Yann Collet	3095ca8c56	fixed minor conversion warnings for g++ on Linux ... U64 is not considered equivalent to unsigned long long	2017-09-26 13:53:50 -07:00
Yann Collet	56f1f0e3dd	write summary for --list on multiple files	2017-09-26 11:21:36 -07:00
Yann Collet	62568c9a42	added capability to generate magic-less frames ... decoder not implemented yet	2017-09-25 14:26:26 -07:00
W. Felix Handte	360238733a	Adds LZ4 support by default if LZ4 is available ... Simple makefile change + quick typename change Test: make clean make # successfully produces binary without lz4 support make clean # with flags to pick up my lz4 build make MOREFLAGS="-L/home/felixh/prog/lz4/lib -I/home/felixh/prog/lz4/lib" # successfully produces binary with lz4 support echo "TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU" | \ ./lz4/lz4 | \ LD_LIBRARY_PATH=/home/felixh/prog/lz4/lib ./zstd/zstd -d # successfully prints TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU	2017-09-22 13:28:56 -07:00
Yann Collet	b0c0e3a3fb	Merge pull request #853 from terrelln/blog ... [zstdcli] Fix LDM advanced options parsing	2017-09-18 15:21:23 -07:00
Nick Terrell	1fe762e236	[zstdcli] Fix LDM advanced options parsing	2017-09-18 14:49:35 -07:00
Yann Collet	92889709f9	fix #851 : sudo zstd -t file.zst changes /dev/null permissions ... reported by @mike155	2017-09-18 13:41:54 -07:00
Yann Collet	1722055799	add comment on using -B# to split input file for dictionary training	2017-09-15 16:23:50 -07:00
Yann Collet	c68d17f2da	ensures that sampleSizes table is large enough ... as recommended by @terrelln	2017-09-15 15:31:31 -07:00
Yann Collet	25a60488dd	fixed 64-to-32 conversion warnings	2017-09-15 11:55:13 -07:00
Yann Collet	a9694231ca	fixed minor conversion warning	2017-09-15 10:16:26 -07:00
Yann Collet	086b9597d9	added ability to split input files for dictionary training ... using command -B# This is the same behavior as benchmark module, which can also split input into arbitrary size blocks, using -B#.	2017-09-14 16:45:10 -07:00
Yann Collet	77c137b3ae	minor comment refactor	2017-09-14 15:12:57 -07:00
Yann Collet	f1571dad8f	Merge pull request #838 from stellamplau/ldm-mergeDev ... Add long distance matcher	2017-09-13 13:24:08 -07:00
Yann Collet	8f26dc3f9c	blindfix for Visual ... LARGE_INTEGER is not an integer : https://msdn.microsoft.com/en-us/library/windows/desktop/aa383713(v=vs.85).aspx Do not take any risk with the structure definition : use int init = 0; like Mac code	2017-09-12 21:21:17 -07:00
Yann Collet	bc41c7f0eb	fixed minor prototype warning	2017-09-12 19:32:26 -07:00
Yann Collet	c95c0c9725	modified util::time API ... for easier invocation. - no longer expose frequency timer : it's either useless, or stored internally in a static variable (init is only necessary once). - UTIL_getTime() provides result by function return.	2017-09-12 18:12:46 -07:00
Nick Terrell	6ab4d5e904	[bench] Use higher resolution timer on POSIX ... The timer used was only accurate up to 0.01 seconds. This timer is accurate up to 1 ns. It is a monotonic timer that measures the real time difference, not on CPU time.	2017-09-12 16:46:44 -07:00
Stella Lau	eb3327c10a	Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev	2017-09-11 15:00:01 -07:00
Yann Collet	3128e03be6	updated license header ... to clarify dual-license meaning as "or"	2017-09-08 00:09:23 -07:00
Yann Collet	baa37c3362	programs/Makefile : better support for GNU conventions ... see https://www.gnu.org/prep/standards/html_node/Command-Variables.html	2017-09-06 16:53:59 -07:00
Stella Lau	eeff55dfa8	Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev	2017-09-06 15:56:32 -07:00
Stella Lau	8c33cfe0bc	Add ldm documentation in README	2017-09-06 15:21:01 -07:00
Stella Lau	c706de5395	Rename and add short ldm parameters in cli	2017-09-05 21:11:18 -07:00
Stella Lau	643d28c701	Add ldm options to 'man zstd'	2017-09-05 11:27:15 -07:00
Stella Lau	67d4a6161c	Add ldmBucketSizeLog param	2017-09-02 21:55:29 -07:00
Stella Lau	a1f04d518d	Move hashEveryLog to cctxParams and update cli	2017-09-01 15:05:47 -07:00
Yann Collet	0558850735	bench stops immediately on decoding error	2017-09-01 11:46:15 -07:00
Stella Lau	17d8e0bdcc	Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate	2017-09-01 10:19:38 -07:00
Stella Lau	8081becadc	Add long distance matching as a CCtxParam	2017-09-01 09:18:58 -07:00
Yann Collet	d7ad99b2ab	Merge branch 'longRangeMatcher' into dev	2017-08-31 18:08:37 -07:00
Yann Collet	c7818fc676	Merge branch 'modTests' into dev ... fixed conflict	2017-08-31 17:00:16 -07:00
Yann Collet	4299c27132	improved console log of utils.h ... removed a warning when compiling on Windows	2017-08-31 16:58:47 -07:00
Yann Collet	8e298382a8	changed target allarch into allzstd ... allzstd contains only zstd-related tests. allmost = allzstd + zwrapper tests (which require zlib)	2017-08-31 14:30:52 -07:00
Stella Lau	6a546efb8c	Add long distance matcher ... Move last literals section to ZSTD_block_internal	2017-08-31 12:53:19 -07:00
Yann Collet	b0cb081dc8	last batch of header files changed to reflect new license (#825) ... only remains to update contrib/linux-kernel (@terrelln)	2017-08-31 12:20:50 -07:00
Stella Lau	c88fb9267f	Replace 'byReference' with enum	2017-08-29 11:55:02 -07:00
Bernhard M. Wiedemann	cf689b84f9	Sort input file list ... in order to make builds reproducible in spite of indeterministic filesystem readdir order. See https://reproducible-builds.org/ for why this is good.	2017-08-26 17:08:00 +02:00
Stella Lau	6f1a21c7e9	Remove formatting-only changes	2017-08-23 10:24:19 -07:00
Yann Collet	232d62b637	fixed a few headers that were too hastily copy/pasted during last license change	2017-08-21 11:24:32 -07:00
Stella Lau	91b30dbe84	Remove test parameter	2017-08-21 10:09:06 -07:00
Stella Lau	f181f33bdf	Disable tests and refactor	2017-08-21 01:59:08 -07:00
Stella Lau	023b24e6d4	Add cctx param tests	2017-08-20 22:55:07 -07:00
Yann Collet	1c108c811e	cli : Display supported formats on -vV command ... Requested and inspired by patch from @ib (#771)	2017-08-19 13:33:50 -07:00
Yann Collet	2ecd34ee5e	fixed unused variables warnings	2017-08-19 01:23:49 -07:00
Yann Collet	23706fb743	updated doc on compilation variables	2017-08-19 01:14:36 -07:00
Yann Collet	9203003d5f	fixed zstd-nolegacy ... and added it to allVariants for CI testings	2017-08-19 01:01:53 -07:00
Yann Collet	4b387729b6	fixed zstd-small ... and added it to shortest for CI tests	2017-08-19 00:48:29 -07:00
Yann Collet	7729ab83bb	Merge branch 'dev' into variants	2017-08-19 00:37:06 -07:00
Yann Collet	32e943b3ef	Merge branch 'dev' of github.com:facebook/zstd into dev	2017-08-19 00:36:37 -07:00
Yann Collet	582e19b01c	Merge pull request #801 from facebook/newLicense ... New license	2017-08-19 00:34:34 -07:00
Yann Collet	288e263ec4	added zstd-decompress to the list of variants tested	2017-08-18 18:39:39 -07:00
Yann Collet	166645e7b3	fixed zstd-compress ... file-information is dependent on decompression functions. it should only be enabled when ZSTD_NODECOMPRESS is not set. also : added zstd-compress compilation test into `make shortest`	2017-08-18 18:30:41 -07:00
Yann Collet	f207b39f55	blindfix for Windows conversion warning ... long type is 32-bits on Windows 64, while it's 64-bits on Unix. 64-to-32 shortening conversion for long is a specific Windows issue.	2017-08-18 17:06:12 -07:00
Yann Collet	32fb407c9d	updated a bunch of headers ... for the new license	2017-08-18 16:52:05 -07:00
Yann Collet	88d2f72df9	fixed --list command in presence of special blocks ... block type RLE is special, compressed size is always 1. block type 3 is "reserved", aka not supported.	2017-08-18 16:18:20 -07:00
Yann Collet	c523c93b26	improved and fixed --list command, original patches by @ib (#772) ... accepts all skippable frame identifiers. display in MB or KB, depending on frame size. fixed combination of skippable and zstd frames.	2017-08-18 15:57:53 -07:00
Yann Collet	3f54d788e9	removed --list from cli help (-h), reported by Agostino Sarubbo (@asarubbo) (#800) ... redundant with shorter -l. both -l and --list do the same thing, and are documented in man page.	2017-08-18 15:15:31 -07:00
codicodi	0fb7b23fcc	fix typo in lz4 support code	2017-08-14 14:04:24 +02:00
Nick Terrell	d0b27483ae	[zstdcli] Fix -t in streaming mode	2017-07-18 14:45:49 -07:00
Yann Collet	3a60efd3a9	policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) ... Previously, ZSTDMT would refuse to create the compressor. Also : increased ZSTDMT_NBTHREADS_MAX to 256, updated doc, and added relevant test	2017-07-13 10:17:23 -07:00
Yann Collet	3345a91964	cli : use new advanced API by default	2017-07-05 17:34:15 -07:00
Yann Collet	b8280fe929	minor code refactoring	2017-07-03 15:14:55 -07:00
Yann Collet	368b974387	fixed gz error reporting	2017-07-03 13:47:46 -07:00
Yann Collet	c9f21c8a71	gzip : only display inflateEnd error when there is no error before	2017-07-03 13:45:09 -07:00
Yann Collet	6c35112d08	gzip decoding does no longer exit() on invalid input	2017-07-03 13:24:50 -07:00
Yann Collet	e97ff3b54a	decoding malformed lz4 frames does no longer exit()	2017-07-03 11:27:29 -07:00
Yann Collet	c34185df13	fileio : decoding malformed lzma frame does no longer exit() ... makes it possible to continue decoding file list	2017-07-03 10:27:16 -07:00
Yann Collet	1d39550471	Merge pull request #744 from terrelln/dev ... [pzstd] Remove appveyor tests	2017-07-01 05:58:33 -07:00
Nick Terrell	56e3964d85	[man] Specify that strategies start at 1	2017-06-30 16:29:37 -07:00
Yann Collet	811deaea6f	Merge pull request #736 from terrelln/cover-default-api ... [zdict] Make COVER the default algorithm	2017-06-28 20:25:36 -07:00
Nick Terrell	5b7fd7c422	[zdict] Make COVER the default algorithm	2017-06-26 21:09:22 -07:00
Nick Terrell	d88a441406	[zstdcli] Document HAVE_LZMA and zstd-noxz	2017-06-26 11:25:00 -07:00
Nick Terrell	6aeb50ea0d	[zstdcli] Add unlzma and unxz symlinks	2017-06-26 11:24:36 -07:00
Nick Terrell	849ecf3510	[zstdcli] Support xz by default when liblzma is available	2017-06-23 17:11:38 -07:00
Yann Collet	ef269c1b68	Merge pull request #725 from facebook/advancedAPI2 ... New Advanced API	2017-06-23 09:50:47 -07:00
Yann Collet	991f9dfcde	switched fileio.c to ZSTD_DEBUG trigger macro	2017-06-21 15:16:13 -07:00
Paul Cruz	5be0f5544e	added newline to align output	2017-06-21 12:41:10 -07:00
Paul Cruz	6f5fe71041	added error check for when file could not be opened	2017-06-21 12:37:23 -07:00
Paul Cruz	74a725da69	reversed calculation of ratio	2017-06-21 12:27:20 -07:00
Yann Collet	7bd1a2900e	added ZSTD_dictMode_e to control dictionary loading mode	2017-06-21 11:50:33 -07:00
Paul Cruz	a73c2a444a	added tests with null files, changed condition to check that the file is more than 0 bytes	2017-06-20 14:33:08 -07:00
Yann Collet	b44ab82f7a	ensure new ZSTD_strategy starts at value 1	2017-06-20 14:11:49 -07:00
Paul Cruz	0765602903	fixed error where extremely small files were not being detected as not compressed with zstd	2017-06-20 14:04:46 -07:00
Paul Cruz	3701dde3ba	changed to use LONG_SEEK	2017-06-20 13:44:05 -07:00
Paul Cruz	93a5510962	added progress metric to display	2017-06-20 13:26:25 -07:00
Paul Cruz	bda57754a2	moved signifcant operations to fileio.c	2017-06-20 12:43:10 -07:00
Paul Cruz	c594507871	removed double if statements for same condition	2017-06-20 11:59:45 -07:00
Paul Cruz	72e0ac9929	minor update to logic	2017-06-20 11:58:27 -07:00
Paul Cruz	cea55a9217	size_t const value	2017-06-20 11:56:11 -07:00
Paul Cruz	bed0dc95bc	changed displayInfo alignment and const	2017-06-20 11:47:24 -07:00
Paul Cruz	cb2dbe6434	changed fileInfo initialization, updated error code	2017-06-20 09:35:21 -07:00
Yann Collet	c08e649e95	first implementation of bench.c with new API ZSTD_compress_generic() ... Doesn't speed optimize this buffer-to-buffer scenario yet. Still internally defers to streaming implementation. Also : fixed a long standing bug in ZSTDMT streaming API.	2017-06-19 18:25:35 -07:00
Yann Collet	fe234bf48b	fix attempts : fullbench for VS2008	2017-06-19 15:23:19 -07:00
Paul Cruz	55b9b28dad	made style changes, added description in man file	2017-06-19 09:55:55 -07:00
Paul Cruz	4a0e4c856c	fixing more spacing on loops	2017-06-15 18:08:22 -07:00
Paul Cruz	b3e62446c6	added in correct error return from main in zstdcli.c	2017-06-15 17:46:49 -07:00
Paul Cruz	0d7c4d766a	initialized info manually to avoid error	2017-06-15 17:30:18 -07:00
Paul Cruz	d3b34e4a25	changed spacing on if statements	2017-06-15 17:12:41 -07:00
Paul Cruz	b316691f91	added const	2017-06-15 17:03:59 -07:00
Paul Cruz	60a2e55e2e	added case for when file is not compressed with zstd (incorrect magic number)	2017-06-15 17:00:59 -07:00
Paul Cruz	acaefb531b	refactoring brackets to allow for const values	2017-06-15 16:27:38 -07:00
Paul Cruz	0757eae6ff	rearranging code to allow for consts	2017-06-15 16:12:04 -07:00
Paul Cruz	e49afae2ab	changed implicit precision loss again	2017-06-15 16:03:40 -07:00
Paul Cruz	54e0b6c66b	got rid of imlpicit precision loss	2017-06-15 16:01:52 -07:00
Paul Cruz	b766211e10	made another function static	2017-06-15 15:55:49 -07:00
Paul Cruz	e7f02fc58a	closed file at end of function, created another variable to keep exit points simpler	2017-06-15 15:53:02 -07:00
Paul Cruz	8b3ff7c9bc	changed info to static allocation	2017-06-15 15:35:37 -07:00
Paul Cruz	96cabf7233	cleaned up if statements	2017-06-15 15:25:20 -07:00
Paul Cruz	9437cc7498	added --list command	2017-06-15 15:07:54 -07:00
Paul Cruz	4495e9a826	allow analysis even if file does not have zst extension	2017-06-15 15:02:54 -07:00
Paul Cruz	a9b77c83e5	cleaning up code for analyzing frames	2017-06-15 14:13:28 -07:00
Paul Cruz	e208992529	cleaning up code	2017-06-15 12:27:32 -07:00
Paul Cruz	fc428ab350	changing formatting again	2017-06-14 13:26:19 -07:00
Paul Cruz	607ee9f4bc	print message when no files are given, rearrange code for readability	2017-06-14 13:23:36 -07:00
Paul Cruz	ebd60349f5	changed U64 to unsigned long to get rid of build test errors	2017-06-14 13:11:18 -07:00
Paul Cruz	81fa33b55a	cleaning up code	2017-06-13 10:06:49 -07:00
Paul Cruz	a56dcbfcf8	changed print format for non-verbose version	2017-06-12 17:47:33 -07:00
Paul Cruz	c828b52111	yet more refactoring	2017-06-12 17:19:53 -07:00
Paul Cruz	28dbf8ee6c	more refactoring for C90	2017-06-12 17:09:08 -07:00
Paul Cruz	618a7b66e2	refactoring for C90 syntax	2017-06-12 16:53:50 -07:00
Paul Cruz	6e33c74133	formatting issues resolved	2017-06-12 16:29:00 -07:00
Paul Cruz	173a739717	code refactorings with explicit conversion	2017-06-12 16:14:04 -07:00
Paul Cruz	f3d6ab28a2	formated information for verbose output	2017-06-12 15:59:28 -07:00
Paul Cruz	6e0204470a	displayed decompressed size	2017-06-12 15:51:59 -07:00
Paul Cruz	dcd6ba6dc6	incremented decompressedSize instead of setting value	2017-06-12 15:40:47 -07:00
Paul Cruz	6996bd2598	removed useless lines	2017-06-12 15:24:50 -07:00
Paul Cruz	9cb602ee25	added in logic for parsing through blocks/frames	2017-06-12 15:22:48 -07:00
Paul Cruz	786b7cac27	added code to analyze the first frame header	2017-06-12 13:46:39 -07:00
Paul Cruz	a3d54cf73d	added line spacing for clarity	2017-06-12 10:58:34 -07:00
Yann Collet	bb0aaf9579	minor man update on -B# option in benchmark mode	2017-06-12 05:19:15 -07:00
Yann Collet	9e6a2eaab6	added MT support to NEWAPI	2017-06-11 18:39:46 -07:00
Paul Cruz	0f06f4f266	added display for compressed size	2017-06-06 09:21:42 -07:00
Paul Cruz	4128f67a4a	added in check suffix check to ensure file was compressed with zstd	2017-06-05 15:00:06 -07:00
Paul Cruz	901435e9ef	setup basic functions for adding --list functionality	2017-06-05 14:45:31 -07:00
cyan4973	8bcbf42617	fixed g++ prototype mismatch	2017-06-04 23:52:00 -07:00
Yann Collet	6d4fef36de	Added ZSTD_compress_generic() ... Used in fileio.c (zstd cli). Need to set macro ZSTD_NEWAPI to trigger it.	2017-05-17 18:36:15 -07:00
Yann Collet	23c256e44b	removed useless variable from CCtx ... CStream's pledgedSrcSize is no longer necessary srcSize control is realized within bufferless interface.	2017-05-16 18:10:11 -07:00
Yann Collet	e12ae02357	minor coding style update ... reduced some long lines	2017-05-16 17:32:33 -07:00
Yann Collet	2d4d31c18a	removed gcc compilation flag -Wbad-function-cast ... It makes it more difficult to directly cast the result of a function, requiring to store the result in an intermediate variable. It does not necessarily help readability, and this restriction can be difficult to overcome in some constructions, like some macros. also : fixed minor Visual conversion warnings in datagencli.c	2017-05-16 11:34:38 -07:00
Yann Collet	46ac9ade9b	fixed vla warning on linux	2017-05-15 18:15:08 -07:00
Yann Collet	83d0c764dc	added several compilation flags	2017-05-15 17:15:46 -07:00
Nick Terrell	a99ece9e79	Update zstd.1.md to use btultra	2017-05-10 16:51:18 -07:00
Nick Terrell	5f2c7213c7	Merge remote-tracking branch 'upstream/dev' into btopt ... * upstream/dev: (305 commits) added test for ZSTD_estimateCStreamSize() changed variable name, for clarity fixed ZSTD_estimateCStreamSize() shortened ZSTD_createCStream_Advanced() fixed symbols test added ZSTD_estimateDStreamSize() changed name frameParams into frameHeader regroup memory usage function declarations separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize() bumped version number added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() Updated ZSTD_freeCCtx() updated ZSTD_estimateCCtxSize() Updated ZSTD_sizeof_CCtx() merged CCtx and CStream as a single same object cli : -d and -t do not stop after a failed decompression added dev branch CircleCI badge added dev branch Appveyor badge keep dev branch status only creates a binary archive without the `programs` directory ...	2017-05-10 16:49:58 -07:00
Yann Collet	01a1abfdb5	cli : -d and -t do not stop after a failed decompression ... The problematic srcfile will be named on console/log, but decompression/test will continue onto next file in the list.	2017-05-05 19:15:24 -07:00
Yann Collet	a00e9599f1	removed -g from DEBUGFLAGS ... It inflates binary sizes, which is negative for the Windows build. It also makes it impossible to check if 2 different source codes get nonetheless compiled to the same binary, since checksum will be different, due to integrated source code.	2017-05-04 17:24:29 -07:00
Yann Collet	c08e56861e	updated dict graphs to 2D mode	2017-05-03 14:33:28 -07:00
Yann Collet	710497d8ea	updated programs/README.md, to introduce compilation variables ... make it possible to enable/disable features individually	2017-05-02 17:18:24 -07:00
Yann Collet	f47284fec0	reorganized Makefile for multiple targets	2017-05-02 16:55:57 -07:00
Yann Collet	3791d2105d	added xzstd4 target ... support for all formats, xz/lzma/lz4 included	2017-05-02 16:38:37 -07:00
Yann Collet	2e63a877f3	fixed xzstd ... --format=xz was missing a break, making the execution continue into lz4 error message	2017-05-02 15:40:42 -07:00
Nick Terrell	f376d47c11	[CLI] Switch dictionary builder on CLI to cover	2017-05-02 11:18:27 -07:00
Nick Terrell	020b960e13	[cover] Make optimization faster	2017-05-02 11:02:48 -07:00
Nick Terrell	f2d9ef1dc0	[cover] Optimize case where d <= 8	2017-05-02 11:02:43 -07:00
Nick Terrell	feba969a69	Fix LZ4_MSG in xzstd	2017-05-02 11:02:16 -07:00
Yann Collet	31533bacce	Changed ZSTD_createCDict_advanced() ... It now only uses compressionParameters as argument. It produces many changes throughout user code, though hopefully they tend to be simple : just provide the cParams part from existing ZSTD_parameters. Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters. This change will force them to revisit this strategy and fix it, since frame parameters are effectively silently ignored in current version.	2017-04-27 00:29:04 -07:00
Sean Purcell	eab41c1872	Fix LZ4 wrapper deprecation warnings	2017-04-26 10:17:38 -07:00
Sean Purcell	2c4b6fe6b3	Make lz4 compression/decompression compatible with library r123	2017-04-25 11:00:54 -07:00
Sean Purcell	4de8632957	Add LZ4 compress/decompress support to CLI	2017-04-24 16:48:25 -07:00
Yann Collet	230d7acc7d	cli : add support for --threads=# command ... updated documentation add relevant test case	2017-04-21 11:38:13 -07:00
Sean Purcell	9606256a8d	Fix no thread message	2017-04-18 13:52:00 -07:00
Yann Collet	1c286950af	Merge pull request #663 from iburinoc/zstdmt-default ... Compile CLI using files instead of objs	2017-04-18 02:59:54 -07:00
Sean Purcell	c8b2df7d62	Compile CLI using files instead of objs ... This avoids conflicts between how the library was configured and how the CLI was configured.	2017-04-17 17:13:47 -07:00
Sean Purcell	e6c504dbe6	Update -T0 comment in man page	2017-04-17 17:12:02 -07:00
Sean Purcell	5935c990a0	Add zstdmt and -T0 to man page	2017-04-17 16:05:20 -07:00
Sean Purcell	f6ef4db20e	Install zstdmt even without threading support	2017-04-17 12:21:11 -07:00
Sean Purcell	5a61f36474	Make zstd compile with mt by default	2017-04-17 12:06:02 -07:00
Baptiste Daroussin	7dd14d03b0	Enable multithreading on BSD	2017-04-15 16:25:08 +02:00
Sean Purcell	e4f3235c85	Add 0 initializers to static variables	2017-04-13 16:34:28 -07:00
Sean Purcell	42bac7fa84	Change ifndef's to undef's	2017-04-13 15:35:05 -07:00
Sean Purcell	9227aae001	Fix clang linux compilation	2017-04-13 14:06:40 -07:00
Sean Purcell	3b6207d4bd	Fix compilation on windows	2017-04-13 14:04:10 -07:00
Sean Purcell	f876f1200c	Fix compilation on macOS	2017-04-13 12:33:45 -07:00
Sean Purcell	afa48518e2	-T0 detects number of physical cores	2017-04-13 12:28:28 -07:00
Yann Collet	5c42d0edc8	cli : better status display for zstdmt in 1-thread mode	2017-04-11 16:57:32 -07:00
Yann Collet	0e30059ba1	cli : FIO_createDictBuffer() replaces FIO_loadFile() ... makes it more explicit that it allocates a buffer and that it's meant to be used for dictionary. Also : simplified function a bit, now only works for dictionaries up to DICTSIZE_MAX	2017-04-11 14:41:02 -07:00
Sean Purcell	279be2015b	Let zstd overwrite read-only files	2017-04-06 12:56:40 -07:00
Yann Collet	c2007388a5	fixed bench.c : optional advanced parameters applied ... before creating cdict	2017-04-04 15:35:06 -07:00
Yann Collet	81d6380139	minor bench.c adjustments ... shorter debug messages no need to check decompressedLength==0 twice	2017-04-04 15:21:09 -07:00
Yann Collet	a36330a27d	minor man update, for overlapSize	2017-04-03 14:57:24 -07:00
Yann Collet	805c5a3efb	updated documentation on multithreading modes	2017-04-01 00:36:31 -07:00
Nick Terrell	eb38617175	Clean up default sparse logic	2017-03-31 15:20:50 -07:00
Nick Terrell	96fe545a18	Turn off sparse mode for OS X by default	2017-03-31 15:16:43 -07:00
Nick Terrell	5152fb2cb2	Convert all tabs to spaces	2017-03-29 18:51:58 -07:00
Yann Collet	5bde4be5a1	fix : bench automatically adapts parameters to srcSize	2017-03-29 12:10:38 -07:00
Sean Purcell	efdaf8bb7c	Fix inline compile errors	2017-03-27 12:26:40 -07:00
Sean Purcell	894bf49713	Fix IS_CONSOLE returning 1 for NUL on windows	2017-03-27 12:19:30 -07:00
Yann Collet	0246d5c531	Merge pull request #630 from facebook/advancedCliCommands ... changed advanced commands --maxdict= and --dictID=	2017-03-26 00:13:35 -07:00
Sean Purcell	ef30af843f	Ignore extension in command name matching	2017-03-24 17:06:09 -07:00
Yann Collet	96aa3019b2	changed advanced commands --maxdict= and --dictID= ... now works with the `=` variant, which is the recommended one. Old variant `--dictID #` still works, for compatibility with existing scripts. Long term objective is to remove the old variant..	2017-03-24 16:04:29 -07:00
Yann Collet	f332ece468	dictBuilder fails to create dictionary on certain input ... Properly expressed with an error code (see zstd_errors.h) and a cli return code != 0	2017-03-23 16:24:02 -07:00
Sean Purcell	dee08caabb	Fix displayLevel redefinition issues	2017-03-23 12:09:35 -07:00
Sean Purcell	680e4e0953	Ignore symbolic links unless --force specified	2017-03-23 11:52:09 -07:00
Sean Purcell	042ba122ae	Change g_displayLevel to int and fix DISPLAYUPDATE flush	2017-03-23 11:21:59 -07:00
Nick Terrell	dfc75f74da	Add $(EXT) to binary to fix zlib detection for some MinGW versions	2017-03-22 13:24:52 -07:00
Nick Terrell	741e3e8314	Add $(FLAGS) when testing for zlib/lzma support. ... * If zlib/lzma isn't in the usual spot, it won't be used, even if `$CFLAGS` and `$LDFLAGS` add the location it is in. * Update the test code snippets to not trigger any warnings.	2017-03-22 12:33:08 -07:00
Yann Collet	35c147ab6a	Merge pull request #605 from iburinoc/term ... Fix zstd not erroring on compressing to terminal without forceStdout	2017-03-17 14:02:48 -07:00
Yann Collet	0a189b63fe	fix minor details in man page	2017-03-16 16:33:53 -07:00
Sean Purcell	b8e52d3c83	Fix zstd not erroring on compressing to terminal without forceStdout	2017-03-16 16:06:19 -07:00
Sean Purcell	72a243e3bd	Convert manpage to Markdown, converted with ronn	2017-03-16 15:37:32 -07:00
Yann Collet	3773bd318f	Merge pull request #601 from terrelln/nxz ... Add xz and lzma support.	2017-03-13 22:00:03 -07:00
Nick Terrell	aa8bcf360f	Add xz and lzma support. ... Finish feature started by @inikep. * Add xz and lzma compression and decompression support to target `xzstd`. * Fix bug in gzip decompression that silently accepted truncated files. * Add gzip frame composition tests. * Add xz/lzma compatibility tests. * Add xz/lzma frame composition tests.	2017-03-13 18:45:41 -07:00
Sean Purcell	120df494e9	Update builds to not support legacy v01-v03	2017-03-13 14:44:08 -07:00
Sean Purcell	334cb34edb	ZSTD_LEGACY_SUPPORT defines lowest supported version	2017-03-13 14:32:30 -07:00
Dmitry V. Levin	201e8c8157	programs/Makefile: remove zstd-internal target ... zstd-internal was intended to be a helper target, but it doesn't help at all, what it does in practice is a useless rebuild of zstd every time "make zstd" is invoked. Fixes: 030ac243a0 ("Changed Makefile to generate zstd with .gz support by default")	2017-03-09 22:47:14 +00:00
Nick Terrell	eeb31eed39	s/ZSTD_btopt2/ZSTD_btultra/g	2017-03-09 11:44:25 -08:00
Yann Collet	8b1d004031	added -Wformat-security flag, as recommended by @pixelb	2017-03-05 21:17:32 -08:00
Yann Collet	78208bd8be	fixed : build zstd cli after libzstd	2017-03-01 21:02:06 -08:00
Yann Collet	c1c040eae1	added gzip tests ... also : made sure zstd --format=gzip -V would fail if gzip compatibility is not supported	2017-03-01 16:49:20 -08:00
Przemyslaw Skibinski	8e5032a965	cli : fix : --rm is silent when input is stdin (decompression)	2017-02-28 09:42:37 +01:00
Yann Collet	a81c2e7e44	Merge pull request #573 from facebook/ddict ... Improved DDict memory usage	2017-02-27 20:54:42 -08:00
Yann Collet	ef569bf75f	Merge branch 'dev' of github.com:facebook/zstd into dev	2017-02-27 15:58:38 -08:00
Yann Collet	dccd6b6f65	cli : fix : --rm is silent when input is stdin ... previously, app would produce an error message, and stop.	2017-02-27 15:57:50 -08:00
Przemyslaw Skibinski	862698f479	minor tweaks in FIO_decompressGzFrame	2017-02-27 13:21:05 +01:00
Yann Collet	0b9b894b2d	reduced ZSTD_DDict memory usage ... saved 128 KB	2017-02-27 00:27:30 -08:00
Przemyslaw Skibinski	337ec875b6	minor tweaks	2017-02-22 10:31:30 +01:00
Przemyslaw Skibinski	684858e7b7	fix memory leaks	2017-02-21 18:17:24 +01:00
Przemyslaw Skibinski	74dcd8d15f	bench.c: use a single ticksPerSecond	2017-02-21 12:22:05 +01:00
Przemyslaw Skibinski	1b59333c82	util.h: restore times()	2017-02-21 07:33:45 +01:00
Przemyslaw Skibinski	da4a0f30af	util.h: use mach_absolute_time for macOS	2017-02-20 12:18:15 +01:00
Przemyslaw Skibinski	e052c60540	introduce UTIL_freq_t	2017-02-20 11:27:11 +01:00
Przemyslaw Skibinski	83775d9e05	replace times() with clock_gettime(CLOCK_MONOTONIC, x)	2017-02-20 11:11:50 +01:00
Przemyslaw Skibinski	acb6e57ad2	use FindFirstFileA instead of FindFirstFile	2017-02-15 17:13:35 +01:00
Przemyslaw Skibinski	6e59b3ce01	added UTIL_fseek	2017-02-15 17:03:16 +01:00
Przemyslaw Skibinski	3aaa1dae4e	simplified zlib detection	2017-02-15 09:17:39 +01:00
Yann Collet	f0b9a8dddb	Merge pull request #547 from inikep/dev11 ... Avoid fseek()'s 2GiB barrier with MacOS and *BSD	2017-02-14 12:29:00 -08:00
Yann Collet	c09d16ba8c	preset behavior for gzip, gunzip and gzcat ... when zstd is called through a link named gzip, gunzip or gzcat, provides the same behavior as the related program. gzip compresses using --format=gz both gzip and gunzip enable --rm by default	2017-02-14 10:45:19 -08:00
Przemyslaw Skibinski	ce13d087d9	fix LONG_SEEK	2017-02-14 09:52:52 +01:00
Przemyslaw Skibinski	970419535f	fixed function name (2)	2017-02-14 09:47:29 +01:00
Przemyslaw Skibinski	bf336572bf	Avoid fseek()'s 2GiB barrier with MacOS and *BSD	2017-02-14 09:45:33 +01:00
Przemyslaw Skibinski	abd6302423	Windows resources updated to v1.1.4	2017-02-14 09:39:09 +01:00
Przemyslaw Skibinski	442c75f132	removed UTIL_doesFileExists (replaced with UTIL_isRegFile)	2017-02-14 09:38:51 +01:00
Przemyslaw Skibinski	b876b96ce1	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2017-02-14 09:26:03 +01:00
Przemyslaw Skibinski	98509a70ac	fixed function name	2017-02-14 09:23:32 +01:00
Przemyslaw Skibinski	1a195b3b7a	fixed unitialized variable warning	2017-02-13 22:56:31 +01:00
Przemyslaw Skibinski	48466b36e8	Resolve conflict with branch 'refs/remotes/facebook/dev' ... # Conflicts: # programs/fileio.c	2017-02-13 21:35:39 +01:00
Yann Collet	db2666c10c	Merge pull request #536 from iburinoc/multiframe ... Simple API multiframe decoding	2017-02-13 12:18:16 -08:00
Przemyslaw Skibinski	64f7221958	limit zlib compression level to Z_BEST_COMPRESSION	2017-02-13 21:00:41 +01:00
Przemyslaw Skibinski	5a3bb05bb2	Merge remote-tracking branch 'refs/remotes/facebook/dev' into gz_compress ... # Conflicts: # programs/Makefile	2017-02-13 20:47:01 +01:00
Przemyslaw Skibinski	35bf23c086	MinGW-w64 requires _FILE_OFFSET_BITS 64	2017-02-13 13:57:29 +01:00
ds77	6220bfc924	fix indentation in previous commit	2017-02-13 12:00:59 +01:00
ds77	168d9b8006	fix seeking 2GB+ files under Windows ... Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes fallback workaround using Win32 API).	2017-02-12 16:05:19 +01:00
Yann Collet	acfa151622	Merge pull request #540 from ds77/dev-stat64-fix ... zstdcli: Fix reporting incorrect sizes of large flies on MinGW	2017-02-11 21:08:08 -08:00
Przemyslaw Skibinski	eb132530cd	revert last commit	2017-02-10 21:15:49 +01:00
Przemyslaw Skibinski	645f5b9856	fix for original MinGW	2017-02-10 20:09:28 +01:00
Sean Purcell	5069b6c2c3	Merge branch 'dev' into multiframe	2017-02-10 10:08:55 -08:00
ds77	45f0c207ab	use _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces previously preprocessor checks.	2017-02-10 18:37:57 +01:00
Przemyslaw Skibinski	456330948a	Merge remote-tracking branch 'refs/remotes/facebook/dev' into mingw ... # Conflicts: # programs/platform.h	2017-02-10 13:38:54 +01:00
Yann Collet	bbba42acd1	Merge pull request #537 from terrelln/small-bugs ... Fix small bugs	2017-02-10 04:35:43 -08:00
-	7ec315df0d	fix previous commit ... * struct _stat64 is not defined by (non-w64) MinGW releases, __stat64 should be everywhere * proper detection of _stat64() availability (as in MinGW sys/stat.h)	2017-02-10 13:27:43 +01:00
-	19f61b534e	use _stat64 only when targetting Win2k or later	2017-02-10 10:56:45 +01:00
Przemyslaw Skibinski	429e13099a	fix 64-bit file support for MinGW	2017-02-10 10:36:44 +01:00
Nick Terrell	d08019813b	Improvement from @inikep	2017-02-09 14:20:52 -08:00
ds77	9cde3f8b2e	use _stat64() on MinGW ... On MinGW, use _stat64() and struct _stat64 instead of stat() and struct stat_t. This fixes reporting incorrect sizes for large files.	2017-02-09 22:49:31 +01:00
Przemyslaw Skibinski	896638a8a2	echo replaced with printf	2017-02-09 17:01:17 +01:00
Przemyslaw Skibinski	13127fd05b	don't use "echo -e"	2017-02-09 11:32:21 +01:00
Nick Terrell	a0f9006e5a	#undef _POSIX_C_SOURCE if already defined	2017-02-08 17:25:01 -08:00
Sean Purcell	0f5c95af44	Disambiguate pledgedSrcSize == 0 ... - Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size - Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize	2017-02-08 15:12:46 -08:00
Sean Purcell	4e709712e1	Decompressed size functions now handle multiframes and distinguish cases ... - Add ZSTD_findDecompressedSize - Traverses multiple frames to find total output size - Add ZSTD_getFrameContentSize - Gets the decompressed size of a single frame by reading header - Deprecate ZSTD_getDecompressedSize	2017-02-08 14:50:10 -08:00
Przemyslaw Skibinski	93901fe85c	remove redundant "ifeq ($(HAVE_ZLIB), 1)"	2017-02-08 21:11:18 +01:00
Przemyslaw Skibinski	4f9eaa7bb3	fixed gcc warnings	2017-02-08 18:08:09 +01:00
Przemyslaw Skibinski	cb56306a50	.gz suffix for gzip compressed files	2017-02-08 17:37:14 +01:00
Przemyslaw Skibinski	02018c83cf	added FIO_compressGzFrame	2017-02-08 16:54:23 +01:00
Przemyslaw Skibinski	cfd4dc299a	add "--format=gzip" option	2017-02-08 15:17:55 +01:00
Przemyslaw Skibinski	ca20edd960	fixed zlib detection with MinGW	2017-02-08 14:32:49 +01:00
Przemyslaw Skibinski	cdf5a7bd9f	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2017-02-08 13:49:35 +01:00
Przemyslaw Skibinski	40580ff669	added description of "--priority=rt"	2017-02-08 13:49:06 +01:00
Przemyslaw Skibinski	0665a359aa	"--rt-prio" renamed to "--priority=rt"	2017-02-07 20:12:59 +01:00
Przemyslaw Skibinski	d05014c739	added the "--rt-prio" option	2017-02-07 16:48:01 +01:00
Przemyslaw Skibinski	94abd6a26c	SET_REALTIME_PRIORITY	2017-02-07 16:36:19 +01:00
Nick Terrell	7e3fc73795	Ensure <zlib.h> can be included in HAVE_ZLIB test	2017-02-06 11:54:31 -08:00
Nick Terrell	2cb8ee8784	Change zlib include to be a system include	2017-02-06 11:32:13 -08:00
Yann Collet	f6e9c4f1e1	Merge pull request #529 from facebook/gzstd_default ... Gzstd default	2017-02-05 18:03:36 -08:00
cyan4973	613087c02b	Silence zlib detection routine ... When it fails, $(CC) sends error message into stderr redirected to /dev/null	2017-02-04 23:36:12 -08:00
Nick Terrell	b4016ff02f	Add cover dictionary training to zstd.1 ... Tested with `make install && man zstd` and visual inspection.	2017-02-03 16:42:07 -08:00
Yann Collet	762ddeeb9e	fixed zstdmt compilation under Windows minGW/MSYS2, by @inikep	2017-02-03 14:35:42 -08:00
Yann Collet	21eb80d485	remove zlib detection artefact ... result of compilation test is sent to /dev/null	2017-02-03 14:34:41 -08:00
Yann Collet	b02ac8d613	fixed pointer conversion warnings (C++) ... in gz module	2017-02-03 08:43:06 -08:00
Yann Collet	c2a4632789	release builds use less debug symbols and warnings ... release build are triggered through either `make`, or their specific target `make zstd-release` and `make lib-release`.	2017-02-02 20:54:41 -08:00
Yann Collet	c3cba9d858	fixed silent conversion warnings in GZDECOMPRESS path	2017-02-02 17:12:50 -08:00
Yann Collet	030ac243a0	Changed Makefile to generate zstd with .gz support by default ... .gz support is detected by a runtime test.	2017-02-02 16:49:34 -08:00
Yann Collet	3dc85bae66	minor : fixed zstd-frugal ... fixed a minor unused variable warning when compiling zstd-frugal target	2017-02-02 16:19:09 -08:00
Yann Collet	48bed91606	Merge pull request #527 from facebook/zstdmt ... zstdmt refinements	2017-01-31 16:36:46 -08:00
Yann Collet	8d8513fb64	fixed C constant restrictions	2017-01-30 14:37:08 -08:00
Yann Collet	3672d06d06	zstdmt : section size is set to be a minimum of overlapSize ... the minimum size condition size is applied transparently (no warning, no error) like previous minimum section size condition (1 KB) which still applies.	2017-01-30 13:35:45 -08:00
Yann Collet	6ccd37c8d4	cli : added advanced parameter overlapLog ... as a hidden (undocumented) parameter for now	2017-01-30 13:07:24 -08:00
Yann Collet	6be2337c26	added command --block-size= ... for Multi-threading only. alias : -B#	2017-01-30 11:17:26 -08:00
Yann Collet	cc3d1bc262	Merge pull request #525 from terrelln/covermt ... Multithreaded COVER dictionary training	2017-01-30 10:15:33 -08:00
Yann Collet	5d9b894e46	Fixed status display for zstdmt ... There is a large buffering effect when using zstdmt in MT mode. Consequently, data is read first, pushed to workers, and only later will the compressed result come out. That means there is no longer immediate correlation between amount of data read, and amount of data written. This patch disables the displaying of % compression when multi-threading is enabled. It adds the displaying of total size when it can be determined (it usually can be determined for files, but not for stdin) so the user has a sense of "how far from the end" the compression compressed is. There is no modification to decompression side, since decompression is only single-threaded for now.	2017-01-27 13:30:18 -08:00
Nick Terrell	2fe9126591	Add multithread support to COVER	2017-01-27 11:56:02 -08:00
Nick Terrell	9c018cc140	Add BUCK files for Nuclide support	2017-01-27 10:43:12 -08:00
Przemyslaw Skibinski	eb2d23a90c	improved #232 fix	2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski	92a4dbf2e4	Fixed https://github.com/facebook/zstd/issues/232	2017-01-27 10:43:03 -08:00
Nick Terrell	83c387eb8e	Fix zstdmt_compress.h include	2017-01-26 15:25:32 -08:00
Yann Collet	717c65d690	Merge pull request #519 from inikep/dev11 ... Dev11	2017-01-26 14:23:44 -08:00
Yann Collet	8dafb1acf5	CLI : automatically set overlap size to max (windowSize) for max compression level	2017-01-25 17:01:13 -08:00
Yann Collet	bb0027405a	fixed zstdmt corruption issue when enabling overlapped sections ... see Asana board for detailed explanation on why and how to fix it	2017-01-25 16:25:38 -08:00
Yann Collet	943cff9c37	fixed zstdmt cli freeze issue with large nb of threads ... fileio.c was continually pushing more content without giving a chance to flush compressed one. It would block the job queue when input data was accumulated too fast (requiring to define many threads). Fixed : fileio flushes whatever it can after each input attempt.	2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski	5022a18d51	improved #232 fix	2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski	64fa2dbc5e	Fixed https://github.com/facebook/zstd/issues/232	2017-01-25 13:02:33 +01:00
Yann Collet	512cbe8c10	zstdmt cli and API allow selection of section sizes ... By default, section sizes are 4x window size. This new setting allow manual selection of section sizes. The larger they are, the (slightly) better the compression ratio, but also the higher the memory allocation cost, and eventually the lesser the nb of possible threads, since each section is compressed by a single thread. It also introduces a prototype to set generic parameters, ZSTDMT_setMTCtxParameter() The idea is that it's possible to add enums to extend the list of parameters that can be set this way. This is more long-term oriented than a fixed-size struct. Consider it as a test.	2017-01-24 17:08:53 -08:00
Yann Collet	f8804d1014	convert tabs to space ... joys of using multiple editors from multiple environments ...	2017-01-20 17:23:19 -08:00
Yann Collet	d7e3cb58c5	Resolved merge conflict dev+zstdmt	2017-01-20 16:44:50 -08:00
cyan4973	5fba09fa41	updated util's time for Windows compatibility ... Correctly measures time on Posix systems when running with Multi-threading Todo : check Windows measurement under multi-threading	2017-01-20 12:57:31 -08:00
Yann Collet	458c8a94b4	minor refactoring : cleaner MT integration within bench	2017-01-19 17:44:15 -08:00
Yann Collet	500014af49	zstd cli can now compress using multi-threading ... added : command -T# added : ZSTD_resetCStream() (zstdmt_compress) added : FIO_setNbThreads() (fileio)	2017-01-19 17:04:28 -08:00
Yann Collet	0f984d94c4	changed MT enabling macro to ZSTD_MULTITHREAD	2017-01-19 14:05:07 -08:00
Yann Collet	f22adae984	fixed minor warning (unused variable) in fuzzer	2017-01-19 13:46:30 -08:00
Yann Collet	736788f8e8	added streaming fuzzer tests for MT API ... Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize which is possible when many flushes are issued while there is not enough dst buffer to flush completed ones.	2017-01-19 12:15:29 -08:00
Przemyslaw Skibinski	8a0bc30a2d	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2017-01-17 13:02:29 +01:00
Przemyslaw Skibinski	d72f4b6b7a	added "Makefile is validated"	2017-01-17 12:40:06 +01:00
Yann Collet	5eb749e734	ZSTDMT_compress() creates a single frame ... The new strategy involves cutting frame at block level. The result is a single frame, preserving ZSTD_getDecompressedSize() As a consequence, bench can now make a full round-trip, since the result is compatible with ZSTD_decompress(). This strategy will not make it possible to decode the frame with multiple threads since the exact cut between independent blocks is not known. MT decoding needs further discussions.	2017-01-11 18:21:25 +01:00
Nick Terrell	c220d4c74d	Use COVER_MEMMULT when training with COVER.	2017-01-09 16:49:04 -08:00
Yann Collet	56958500fc	minor man page update	2017-01-09 19:47:09 +01:00
Nick Terrell	3a1fefcf00	Simplify COVER parameters	2017-01-02 17:51:38 -08:00
Nick Terrell	df8415c502	Add COVER to the zstd cli	2017-01-02 14:43:08 -08:00
Yann Collet	6334b04d61	compile object files, for faster recompilation	2017-01-02 03:22:18 +01:00
Yann Collet	747452677d	fixed cmake tests	2017-01-02 02:05:45 +01:00
Yann Collet	cdb2763f4a	new Makefile target zstdmt	2017-01-02 01:43:56 +01:00
Yann Collet	f1cb55192c	fixed linux warnings	2017-01-02 01:11:55 +01:00
Yann Collet	0ec6a95ba1	minor fixes	2017-01-02 00:49:42 +01:00
Yann Collet	c6a6417458	bench correctly measures time for multi-threaded compression (posix only)	2016-12-31 03:31:26 +01:00
Yann Collet	f765a375a5	Merge pull request #504 from terrelln/thread-pool ... [zstdmt] Add thread pool	2016-12-30 15:31:49 +01:00
Yann Collet	e70912c72b	Changed : input divided into roughly equal parts. ... Debug : can measure time waiting for mutexes to unlock.	2016-12-29 01:24:01 +01:00
Yann Collet	8d7432914f	Merge pull request #503 from inikep/dev11 ... Dev11	2016-12-28 16:50:39 +01:00
Yann Collet	ab7a579180	added -T command , to set nb of threads	2016-12-28 16:11:09 +01:00
Przemyslaw Skibinski	75f3a3a335	changed default PREFIX and MANDIR	2016-12-28 12:32:41 +01:00
Yann Collet	3d93f2fce7	first zstdmt sketch	2016-12-27 07:19:36 +01:00
Yann Collet	c07d2e3a31	Merge pull request #499 from inikep/dev11 ... improved *BSD and Solaris compatibility	2016-12-23 21:32:03 +01:00
Przemyslaw Skibinski	63b0014b96	BSD: improved "make install"	2016-12-23 10:05:49 +01:00
Andrew Janke	fce374a100	zstdless: add shebang and quote $@	2016-12-22 17:40:10 -05:00
Przemyslaw Skibinski	b999170311	Solaris: working "make -C lib install"	2016-12-22 20:14:37 +01:00
Przemyslaw Skibinski	aab442133d	Solaris: working "make -C programs install"	2016-12-22 19:26:01 +01:00
Yann Collet	8333106b8a	Merge branch 'dev' of github.com:facebook/zstd into dev	2016-12-21 16:44:24 +01:00
Yann Collet	0d7e84899f	Merge pull request #489 from inikep/v112 ... improved detection of POSIX	2016-12-21 16:42:46 +01:00
Yann Collet	9518744025	Merge branch 'dev' of github.com:facebook/zstd into dev	2016-12-21 16:20:41 +01:00
Yann Collet	1f57c2ed32	added : ZSTD_createCDict_byReference()	2016-12-21 16:20:11 +01:00
Przemyslaw Skibinski	7a8a03c20d	util.h: restore BSD license for Facebook Open-Source	2016-12-21 15:08:44 +01:00
Przemyslaw Skibinski	97a258d71d	updated comments	2016-12-21 14:00:41 +01:00
Przemyslaw Skibinski	e679741b18	_CRT_SECURE_NO_WARNINGS moved to util.h	2016-12-21 13:47:11 +01:00
Przemyslaw Skibinski	2f6ccee6af	platform.h: removed Compiler Options	2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski	101df4f636	fixed Visual Studio compilation	2016-12-21 11:43:11 +01:00
Przemyslaw Skibinski	a35b9448ac	improved MinGW support	2016-12-21 11:18:45 +01:00
Przemyslaw Skibinski	5736db219e	fix basic types redefinition	2016-12-21 09:26:00 +01:00
Przemyslaw Skibinski	20b089e53d	simplified zstdcli.c	2016-12-21 09:19:15 +01:00
Przemyslaw Skibinski	16ae6563a2	executables use new util.h and platform.h	2016-12-21 09:06:14 +01:00
Przemyslaw Skibinski	ead350bdc0	improved util.h and platform.h	2016-12-21 09:04:59 +01:00
Przemyslaw Skibinski	1c1db6b845	windres updated to v1.1.3	2016-12-20 11:21:26 +01:00
Przemyslaw Skibinski	f8046b8e72	Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112 ... # Conflicts: # appveyor.yml	2016-12-19 08:20:26 +01:00
Yann Collet	1496c3dc47	Fix : size estimation when some samples are very large	2016-12-18 11:58:23 +01:00
Yann Collet	d46ecb58a5	added dll compilation tests	2016-12-17 16:28:12 +01:00
Przemyslaw Skibinski	0b37205098	util.h: minor improvement	2016-12-16 17:12:23 +01:00
Przemyslaw Skibinski	b0f3663edc	imporved support for POSIX-type OSes	2016-12-16 15:41:18 +01:00
Przemyslaw Skibinski	3cdfe266cf	use PLATFORM_POSIX_VERSION	2016-12-16 15:00:50 +01:00
Przemyslaw Skibinski	b0e670a054	util.h uses platform.h	2016-12-16 14:25:12 +01:00
Przemyslaw Skibinski	b866e72826	tools use platform.h	2016-12-16 14:24:01 +01:00
Przemyslaw Skibinski	b3843afcf5	introduced platform.h	2016-12-16 14:13:15 +01:00
Przemyslaw Skibinski	25314428c9	zstd.1: added advanced compression options	2016-12-14 16:10:13 +01:00
Przemyslaw Skibinski	c71e552b2e	fixed "strategy" in advanced compression parameters	2016-12-13 20:04:32 +01:00
Przemyslaw Skibinski	98ef0f98df	fixed conversion warning	2016-12-13 14:52:21 +01:00
Przemyslaw Skibinski	8349d675e0	fileio.c: support advanced compression parameters	2016-12-13 13:24:59 +01:00
Przemyslaw Skibinski	897b8bb5eb	bench.c: support advanced compression parameters	2016-12-13 13:03:41 +01:00
Przemyslaw Skibinski	334b8962b0	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2016-12-13 12:19:19 +01:00
Przemyslaw Skibinski	671f28d1e5	added parseCompressionParameters	2016-12-13 12:18:07 +01:00
Dimitry Andric	83cc2fb083	Enable using isatty() and nanosleep() on *BSD	2016-12-12 19:24:51 +01:00
Yann Collet	4c6ce5a57e	Merge pull request #476 from inikep/dev11 ... Dev11	2016-12-12 16:44:55 +01:00
Przemyslaw Skibinski	c855da7533	allow all operationg systems with _POSIX_C_SOURCE >= 200112L	2016-12-12 15:37:43 +01:00
Przemyslaw Skibinski	242c036875	turn on the '-r' option for HP-UX and AIX	2016-12-12 11:59:17 +01:00
Przemyslaw Skibinski	c01ac6c30a	turn on the '-r' option for *BSD and Solaris	2016-12-12 11:23:21 +01:00
Yann Collet	fda539f50b	minor coding style changes	2016-12-12 01:03:23 +01:00
Przemyslaw Skibinski	128acb35db	improved MSYS support	2016-12-08 10:49:59 +01:00
Przemyslaw Skibinski	13de72e0cc	programs\Makefile: use Linux paths	2016-12-08 10:43:55 +01:00
Yann Collet	2a826e592b	compatibility with zstd-frugal (noBench mode)	2016-12-06 17:56:20 -08:00
Yann Collet	e63c631aaf	decode benchmark, multi-files	2016-12-06 17:46:49 -08:00
Yann Collet	d946501d2c	decode benchmark - single file (hidden option)	2016-12-06 16:49:23 -08:00
Yann Collet	6b9a983261	changed gzstd build messages	2016-12-06 11:23:25 -08:00
Przemyslaw Skibinski	7c6bbc3298	updated FIO_passThrough	2016-12-05 18:31:14 +01:00
Przemyslaw Skibinski	6b508b1770	updated test-gzstd	2016-12-05 18:02:40 +01:00
Przemyslaw Skibinski	b493e3b3d3	introduced srcBufferLoaded	2016-12-05 17:39:38 +01:00
Przemyslaw Skibinski	3c69760275	improved FIO_decompressGzFrame	2016-12-05 15:58:23 +01:00
Przemyslaw Skibinski	8489f184f6	improved detection of -lz	2016-12-05 13:47:00 +01:00
Yann Collet	db85a6e09a	added zstdgrep	2016-12-02 15:57:07 -08:00
Yann Collet	efaf104b2d	added zstdless	2016-12-02 15:24:40 -08:00
Yann Collet	743b33f57e	fix zstdcat	2016-12-02 15:18:57 -08:00
Yann Collet	5bd4237beb	minor refactor	2016-12-02 12:40:57 -08:00
Przemyslaw Skibinski	690753ea1a	improved formatting in FIO_decompressSrcFile	2016-12-02 16:20:16 +01:00
Przemyslaw Skibinski	4e49580407	removed testing artifacts	2016-12-02 15:19:00 +01:00
Przemyslaw Skibinski	c5eebca128	rewritten FIO_decompressGzFile	2016-12-02 15:01:31 +01:00
Przemyslaw Skibinski	b0f2ef2119	improved gzip_* functions	2016-12-02 13:50:29 +01:00
Przemyslaw Skibinski	4b504f131a	added gzip_open, gzip_close, gzip_read	2016-12-02 13:11:39 +01:00
Przemyslaw Skibinski	daaf754539	detect stream with ungetc	2016-12-01 13:29:19 +01:00
Przemyslaw Skibinski	19aad42ee1	added FIO_decompressGzFile	2016-12-01 11:56:31 +01:00
Przemyslaw Skibinski	166830ed0a	autodetect -lz	2016-11-30 16:43:07 +01:00
Przemyslaw Skibinski	abfb51f5f2	gzstd: decompresses .gz files	2016-11-30 15:05:54 +01:00
Przemyslaw Skibinski	0e14675df2	fileio.c: detect .gz files	2016-11-30 13:34:21 +01:00
Yann Collet	167c494748	Merge branch 'dev' of github.com:facebook/zstd into dev	2016-11-29 14:05:15 -08:00
Yann Collet	4f5350f610	long matches support overflow	2016-11-29 13:12:24 -08:00
Przemyslaw Skibinski	fd0ac93024	bench.c: use ZSTD_maxCLevel()	2016-11-23 21:45:29 +01:00
Przemyslaw Skibinski	5ddcd9d9ae	bench.c: fixed MAX_CLEVEL	2016-11-21 16:37:56 +01:00
Przemyslaw Skibinski	5a17223691	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2016-11-18 11:47:01 +01:00
Przemyslaw Skibinski	2558b4cdbc	bench.c without dict uses ZSTD_compressCCtx	2016-11-18 11:46:30 +01:00
Przemyslaw Skibinski	3d18088b38	updated windres	2016-11-17 18:04:41 +01:00
Yann Collet	52afb3993e	zbuff API now generates deprecation warnings	2016-11-16 08:50:54 -08:00
Yann Collet	cdff19c4b3	minor comment change	2016-11-11 17:26:54 -08:00
Yann Collet	0018ca28dc	zstd cli : displays total decoded size, even when a stream consists of multiple frames	2016-11-07 14:42:37 -08:00
Przemyslaw Skibinski	d0815583d9	Changed stdinmark and stdoutmark	2016-11-04 11:37:27 +01:00
Przemyslaw Skibinski	38b590ad69	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 ... # Conflicts: # lib/Makefile	2016-11-04 10:10:54 +01:00
Yann Collet	407a11f63e	fixed Visual compatibility	2016-11-03 15:52:01 -07:00
Przemyslaw Skibinski	3a415594b1	fixed MinGW compilation	2016-11-03 12:59:20 +01:00
Przemyslaw Skibinski	1f60eb6b17	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11	2016-11-03 11:38:04 +01:00
Przemyslaw Skibinski	26306fcacf	BMK_SetNbIterations renamed to BMK_SetNbSeconds	2016-11-03 11:38:01 +01:00
Przemyslaw Skibinski	b40884f43d	preserve file modification time for Visual C++	2016-11-03 09:54:53 +01:00
Yann Collet	7347869fb6	fixed make install	2016-11-02 22:28:37 -07:00
Yann Collet	179b19776f	fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h ... Added : ZSTD_isFrame() in experimental section	2016-11-02 17:30:49 -07:00
Yann Collet	f3f13211ae	Fix #419 : no warning when setting custom LDFLAGS	2016-11-02 17:02:45 -07:00
Przemyslaw Skibinski	fcf22e3473	set permissions, access and modification times	2016-11-02 14:08:07 +01:00
Przemyslaw Skibinski	a42794df61	preserve file modification time	2016-11-02 13:08:39 +01:00
Przemyslaw Skibinski	d872b64f52	added UTIL_setModificationTime, UTIL_getModificationTime	2016-11-02 12:52:20 +01:00
Yann Collet	3d197d4c8c	updated man page	2016-10-31 18:10:32 -07:00
Yann Collet	b9550d6d43	fixed benchmark mode typo . ... added benchmark mode tests	2016-10-28 14:43:24 -07:00
Yann Collet	22de81e87e	updated man page	2016-10-28 13:58:31 -07:00
Przemyslaw Skibinski	bd0bd0fb65	generate RES files during each compilation	2016-10-28 18:15:34 +02:00
Przemyslaw Skibinski	589a8f27b7	generate_res.bat moved to programs/windres	2016-10-28 13:17:35 +02:00
Przemyslaw Skibinski	76d66dffd0	generate_res moved to programs/windres	2016-10-28 13:16:06 +02:00
Przemyslaw Skibinski	347903cfb5	Windows resources moved to programs/windres/	2016-10-25 10:34:41 +02:00
Yann Collet	a7a4690b0a	use implicit rules to compile zstd_decompress.c	2016-10-18 16:01:03 -07:00
Yann Collet	33fdd099bb	fix command line interpretation	2016-10-17 17:48:48 -07:00
Yann Collet	70077bc9bb	refactor for long commands	2016-10-14 14:41:17 -07:00
Yann Collet	d7b120ab5c	added long commands --memory= and --memlimit-decompress=	2016-10-14 14:22:32 -07:00
Yann Collet	1122349ac2	added long comment --memlimit=	2016-10-14 14:07:11 -07:00
Yann Collet	d4cda27b63	new command -M#, to limit memory usage during decompression (#403)	2016-10-14 13:32:20 -07:00
Yann Collet	e19111c42f	make creates libzstd binaries (#415)	2016-10-12 11:09:36 -07:00
Nick Terrell	3e82f2b61a	Remove references to GPLv2 license ... License headers added to Makefiles were taken from `zstd/Makefile`.	2016-10-11 15:28:40 -07:00
inikep	23cec875cd	fixed Makefile targets: zstd-small, zstd-decompress, zstd-compress	2016-10-05 11:56:22 +02:00
Nick Terrell	2b4de225e1	Don't redefine macro in util.h	2016-09-22 18:02:39 -07:00
Nick Terrell	5b2c0dbed0	Add include guards to datagen.h	2016-09-22 17:12:50 -07:00
Yann Collet	993060e0f2	cli : better adaptation to small files	2016-09-21 16:46:08 +02:00
Yann Collet	714464f05d	fixed : cli : forgotten mandatory argument	2016-09-21 16:05:03 +02:00
Yann Collet	27b5ac666e	Implemented "command must be followed by argument" protection ... suggested by @terrelln (#375)	2016-09-21 14:31:22 +02:00
Yann Collet	0977f7ece6	minor refactor for clarity	2016-09-21 12:24:43 +02:00
Yann Collet	dbe70bad48	completed change from projects to build	2016-09-19 15:08:43 +02:00
Yann Collet	60038948e6	added -- command in help	2016-09-16 18:52:52 +02:00
inikep	3e70cf8036	Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev	2016-09-15 20:41:40 +02:00
inikep	6173931868	fixed memory leak reported by bryongloden	2016-09-15 18:58:18 +02:00
Yann Collet	a6bdf55759	fixed memory leak	2016-09-15 17:02:06 +02:00
Yann Collet	43eeea4725	fileio uses ZSTD_resetCStream()	2016-09-15 15:38:44 +02:00
Yann Collet	0b01a8ef09	Merge pull request #366 from inikep/dev ... zstd.exe has FileVersion and ProductVersion with 32-bit gcc (MinGW)	2016-09-14 19:42:29 +02:00
Yann Collet	3ecbe6a37c	fileio uses ZSTD_resetDStream()	2016-09-14 17:26:59 +02:00
inikep	c03f15e89d	zstd.exe has FileVersion and ProductVersion with 32-bit gcc (MinGW)	2016-09-14 16:16:24 +02:00
Yann Collet	64a84edef5	added -r support for Mac OS-X	2016-09-13 17:54:37 +02:00
Yann Collet	c4cc9bf973	-r generates an error on systems which do not support it	2016-09-13 17:50:08 +02:00
inikep	9f25fcf804	fixed precision warnigns	2016-09-13 16:38:54 +02:00
inikep	362708d4d2	zstd.exe has FileVersion and ProductVersion	2016-09-13 13:53:43 +02:00
inikep	1c5ba8a5e7	util.h: removed dependency from PATH_MAX	2016-09-13 13:13:10 +02:00
Yann Collet	a3481d6de0	make uninstall	2016-09-12 05:04:26 +02:00
Yann Collet	d77f2ceee6	Merge branch 'dev' of github.com:Cyan4973/zstd into dev	2016-09-12 03:14:03 +02:00
Yann Collet	5c6d244973	Merge branch 'dev' of github.com:facebook/zstd into dev	2016-09-07 14:54:54 +02:00
Yann Collet	ac8bace6b1	support large skippable frames	2016-09-07 14:54:23 +02:00
Yann Collet	03d3f238de	minor comment	2016-09-07 07:01:33 +02:00
Yann Collet	aad9fe5470	don't remove() /dev/null (#316)	2016-09-07 07:00:08 +02:00
Yann Collet	7ae67bb18a	small compression speed gains with using_CDict	2016-09-06 06:28:05 +02:00
Thomas Klausner	b85cdabd50	Enable install targets for NetBSD.	2016-09-04 14:37:57 +02:00
Yann Collet	33a0465a51	fixed a few links	2016-09-02 22:11:49 -07:00
Yann Collet	5b8c024716	fixed zstd-pgo (#329) reported by @octoploid	2016-09-02 09:44:00 -07:00
Yann Collet	60181e3aaf	zstd cli correctly detects console on Mac OS-X	2016-09-01 17:14:02 -07:00
Yann Collet	c843142ad0	zstd -d writes to stdout when input is stdin	2016-09-01 15:05:57 -07:00
Yann Collet	1c59c20903	removed redundant files	2016-08-31 07:15:44 -07:00
Yann Collet	599c69d917	minor Makefile updates	2016-08-30 13:33:20 -07:00
Yann Collet	4ded9e591c	added boilerplate	2016-08-30 11:06:28 -07:00
Yann Collet	09c3c8e885	Merge pull request #307 from inikep/dev08 ... updated README.md	2016-08-29 16:32:33 -07:00
inikep	6416b0d705	updated README.md	2016-08-29 13:04:26 +02:00
Yann Collet	54caf6893e	Removed fileio_legacy; now legacy supports relies entirely on streaming API. ... CLI legacy support now limited to v0.4+	2016-08-28 19:20:13 -07:00
Yann Collet	9097f7b156	fixed versionsTest	2016-08-28 16:25:56 -07:00
Yann Collet	1c69baa1a9	CLI supports Legacy formats v0.4+	2016-08-28 12:47:17 -07:00
Yann Collet	5f53b0335e	fixed continuation context	2016-08-28 10:00:49 -07:00
Yann Collet	ab267e772c	removed fileio specific legacy support; ... now depends on ZSTD_decompressStream() transparent legacy support	2016-08-28 08:46:25 -07:00
Yann Collet	87c18b2ebd	fixed multiple minor warnings for XCode	2016-08-26 01:43:47 +02:00
Yann Collet	0baa64a763	increased maximum memory size for 64-bits bench to 16 GB	2016-08-25 22:54:13 +02:00
inikep	de9d13068f	updated zstd_compression_format.md	2016-08-25 14:59:08 +02:00
inikep	0132375ba6	improved zstd_compression_format.md	2016-08-25 12:20:38 +02:00
inikep	637d33566d	minor README.md corrections	2016-08-25 10:42:49 +02:00
inikep	ab2f7708c1	created programs\README.md	2016-08-25 10:07:20 +02:00
inikep	e416e30019	remove unnecessary comments	2016-08-24 17:32:09 +02:00
inikep	7bcdbcf51d	Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into Other	2016-08-24 17:09:19 +02:00
Yann Collet	d1733f7417	fixed crc bug in rare timing conditions within bench.c	2016-08-21 01:04:46 +02:00
inikep	c629776b31	make clean will remove decompress/zstd_decompress.o	2016-08-19 10:45:59 +02:00
Yann Collet	b571a0eab4	Merge pull request #298 from inikep/Other ... Other	2016-08-18 17:14:03 +02:00
inikep	b6514d2496	improved .gitignore	2016-08-18 16:53:29 +02:00
Yann Collet	49d105cfcf	better warning and error messages in case of dictionary training failure (#292)	2016-08-18 15:02:11 +02:00
inikep	2868d565e4	test-related files moved to tests/	2016-08-18 13:18:11 +02:00
Yann Collet	e80d15304a	Merge pull request #296 from inikep/Other ... Other	2016-08-18 11:48:48 +02:00
inikep	a7bb322a93	removed never referenced functions	2016-08-18 10:30:21 +02:00
inikep	5a54870047	fixed Intel Compiler warnings with Visual Studio ... http://encode.ru/threads/2119-Zstandard?p=49504&viewfull=1#post49504	2016-08-18 09:00:25 +02:00
Yann Collet	bfdc34e5a3	Merge pull request #294 from inikep/Other ... improved Makefiles	2016-08-17 19:41:56 +02:00
inikep	79b4062c81	fixed paramgrill dependency	2016-08-17 16:29:57 +02:00
inikep	d0c270fe85	improved programs\Makefile	2016-08-17 15:59:50 +02:00
inikep	78f3e0630b	disabled %llu warnings with MinGW on Windows	2016-08-17 14:52:11 +02:00
Yann Collet	3d9211e961	Merge pull request #290 from david-y-lam/fix_file_name ... Remove file name truncation in zstd CLI output	2016-08-17 14:05:29 +02:00
Yann Collet	53e17fbd5e	updated streaming API	2016-08-17 01:39:22 +02:00
Yann Collet	94ca85d01b	fixed Visual warning	2016-08-16 15:11:28 +02:00
Yann Collet	6263ba5451	fileio uses new streaming API	2016-08-16 15:11:28 +02:00
Yann Collet	fb5e385164	minor changes	2016-08-16 15:11:28 +02:00
Yann Collet	d7883a2c0e	added zstreamtest	2016-08-16 15:11:28 +02:00
David Lam	0f2704581e	Remove truncation of filename in zstd CLI log lines	2016-08-13 11:26:21 -07:00
David Lam	ac43e9582e	Add more descriptive message to -v option in zstd CLI	2016-08-12 17:47:18 -07:00
Yann Collet	fab023010d	changed test, in compliance with new --ultra policy	2016-08-12 19:00:18 +02:00
Yann Collet	fe0590f63f	new --ultra policy	2016-08-12 18:04:15 +02:00
inikep	7132fb15ba	bench.c: removed benchResult_t	2016-08-10 14:59:18 +02:00
inikep	3d2c58c3a9	fixed compilation with Visual in C++ mode	2016-08-10 14:28:47 +02:00
Yann Collet	666398e7ed	added : xxhash namespace enforced from xxhash.h. ... added : xxhash namespace test. removed : -DXXH_NAMESPACE	2016-08-10 08:16:51 +02:00
Yann Collet	6a21971f4a	bench : implemented avgSize	2016-08-03 00:06:24 +02:00
Yann Collet	bf2bc112bb	bench : controlled display update when loading lot of files	2016-08-02 23:48:13 +02:00
Yann Collet	de4c04f6c2	Fixed : ZSTD_compress* can compress > 4 GB in a single pass, reported by Nick Terrell	2016-08-02 11:27:05 +02:00
Yann Collet	9a69ec4f3b	added : resilience test for ZSTD_getErrorName()	2016-08-01 16:25:58 +02:00
Yann Collet	415251c63b	minor style change	2016-08-01 14:26:49 +02:00
Yann Collet	a9febe81ae	changed bench behavior for slow compression levels	2016-08-01 13:40:52 +02:00
Yann Collet	2a2ba3691c	Merge pull request #266 from jrmarino/master ... Enable build on FreeBSD ports (includes DragonFly BSD) [dev branch]	2016-07-31 02:21:26 +02:00
jrmarino	0d07ec0c0c	Enable build on FreeBSD ports (includes DragonFly BSD) ... Zstd has been introduced to FreeBSD ports (http://www.freshports.org/archivers/zstd/) which DragonFly BSD also uses. FreeBSD and DragonFly use the install targets (albeit modified in some cases) so they must be added to the associated Makefile filters.	2016-07-30 19:11:15 -05:00
Yann Collet	235911e13f	removed "avg" evaluation from bench -q ... removed "sleeping" notification from bench -q	2016-07-31 01:32:48 +02:00
Yann Collet	4c5bbf64f9	fixed : frame concatenation without checksum	2016-07-28 20:30:25 +02:00
Yann Collet	60ba31c570	zbuff uses ZSTD_compressEnd()	2016-07-28 19:55:09 +02:00
Yann Collet	62470b4bab	Changed ZSTD_compressEnd()	2016-07-28 15:29:08 +02:00
Yann Collet	c991cc1828	new frame end, 32-bits checksums	2016-07-28 00:55:43 +02:00
Yann Collet	4110534886	ZSTD_maxCLevel() is promoted to "stable" API (#254, by @FrancescAlted)	2016-07-27 15:09:11 +02:00
Yann Collet	dd25a27702	added tutorial warning messages for dictBuilder	2016-07-27 12:43:09 +02:00
Yann Collet	fbd557d5c2	multi-files -t doesn't stop after detecting magic number read failure	2016-07-26 17:13:58 +02:00
Yann Collet	a1dd6b97d2	fixed null-length round trip	2016-07-26 16:44:09 +02:00
Yann Collet	7adc2328a3	fixed --test on zero-length files, reported by @amnilsson	2016-07-26 15:49:24 +02:00
Yann Collet	9cebec21e1	Merge pull request #259 from inikep/dev08 ... legacy decoder for v0.7 format	2016-07-26 15:13:05 +02:00
inikep	7bc5c6b5cb	util.h: detect if an error occurs in readdir() (thanks to Jim Meyering)	2016-07-26 11:07:37 +02:00
Yann Collet	24a3d90bf0	strengthened integrity tests	2016-07-26 01:33:20 +02:00
Yann Collet	b402490546	fixed #260, reported by @amnilsson	2016-07-26 00:49:47 +02:00
inikep	fca90f8f60	legacy decoder for v0.7 format	2016-07-25 17:49:08 +02:00
Yann Collet	e5a817a2d8	better MOREFLAGS	2016-07-24 17:46:05 +02:00
Yann Collet	10b9c13d07	fixed doc on cLevel default, reported by Oliver Lange	2016-07-24 01:21:53 +02:00
Yann Collet	f8e7b5363f	unified encoding types	2016-07-23 16:31:49 +02:00
Yann Collet	b752298d1d	fixed 32-bits compilation	2016-07-22 14:37:09 +02:00
Yann Collet	32faf6c8e7	fixed conversion warnings	2016-07-22 14:37:09 +02:00
Yann Collet	5e45a5fbb3	force loop-align to 32 for zstd_decompress	2016-07-22 14:37:09 +02:00
Yann Collet	5288ac0cb7	changed filed order	2016-07-22 14:37:09 +02:00
Yann Collet	6fa05a2371	cBlockSize uses little-endian convention	2016-07-22 14:37:09 +02:00
Yann Collet	06dd74114a	minor : long tests display progression status	2016-07-18 04:06:43 +02:00
Yann Collet	85f3919960	moved zstd.h to /lib	2016-07-17 20:42:21 +02:00
Yann Collet	988bcf360a	-v and --verbose increase display level by 1	2016-07-15 19:43:30 +02:00
Yann Collet	d3d2db587e	adapted fuzzer test to new blockSizeMax rule for ZSTD_compressBlock()	2016-07-15 12:20:26 +02:00
Yann Collet	227cc39e15	improved efficiency for large messages with small dictionaries	2016-07-15 11:27:09 +02:00
Yann Collet	e20d5cf117	fixed paramgrill	2016-07-14 20:46:24 +02:00
Yann Collet	17508f1a16	fixed a few minor coverity warnings	2016-07-14 17:18:20 +02:00
Yann Collet	69c2cdb45c	fixed conversion warning	2016-07-14 16:52:45 +02:00
Yann Collet	44f684ded3	fixed minor coverity warning	2016-07-13 20:12:33 +02:00
Yann Collet	5e80dd3261	fixed minor coverity warnings	2016-07-13 19:21:57 +02:00
Yann Collet	158e7703bb	reduced paramgrill dependency to C standard lib only	2016-07-13 16:45:24 +02:00
Yann Collet	2b1a3638e6	changed macro name to ZSTDCLI_CLEVEL_DEFAULT	2016-07-13 15:16:00 +02:00
Yann Collet	3c242e79d3	updated compression levels table	2016-07-13 14:56:24 +02:00
Yann Collet	2cac5b30b9	changed default compression level to 3 (can be modified with macro ZSTDCLI_DEFAULT_CLEVEL)	2016-07-13 14:15:08 +02:00
Yann Collet	696c4d7ef5	new paramgrill mode : -O# : find optimal parameters for a given sample and a given target speed	2016-07-13 13:11:08 +02:00