townforge/zstd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9,629 Commits 1 Branch 0 Tags 29 MiB
cc
Commit Graph

2074 Commits

Author SHA1 Message Date
Yann Collet
80cee8d3fe faster rebuild of zstd 
Building the zstd CLI costs time.
Some part of it is incompressible, leading to substantial iteration delay when testing code modifications.
That's mainly because all source files from the library must be rebuilt from source every time.

The main reason we don't build the CLI from library object files
is that we can't just build the object directly in the lib/ directory
(which they would by default)
since they use different compilation flags.
Specifically, the CLI enables multithreading, while the library doesn't (by default).

This is solved in this commit, by generating the object files locally.
Now, the CLI and the library can employ different sets of flags, without tripping over each other.
All library object files are generated directly into programs/ dir.
This works because no 2 source files have the same name.

Now, modifying a file doesn't require to recompile the entire lib, just the modified files.
The recipe is also compatible with `-j` parallel build, leading to large build time reductions on multi-core systems.
 2020-10-16 00:01:41 -07:00
senhuang42
043b934ba2 Pass cRess_t by const const pointer 2020-10-14 20:19:46 -04:00
senhuang42
467e4383b0 Merge branch 'dev' of github.com:senhuang42/zstd into change_ldm_mt_config 2020-10-14 10:17:50 -04:00
senhuang42
a39614de1f Fix incorrect usage of cycleLog() in --patch-from 2020-10-13 13:00:27 -04:00
Like Ma
cc907770bd Fix building on AIX 5.1 2020-10-09 18:34:00 +08:00
senhuang42
7259b258d1 Add callsites to zstdcli.c and tests to playTests.sh 2020-10-07 13:47:38 -04:00
senhuang42
93cd9d8a6e Add hasStdoutOutput setter to fileio.h 2020-10-07 13:44:25 -04:00
senhuang42
dd3dac9ce0 Fixed logic for stdout output 2020-10-07 13:43:27 -04:00
senhuang42
1ebe360d0f Add new stdoutOutput field 2020-10-07 13:42:34 -04:00
Yann Collet
f7d4943788
Merge pull request #2330 from senhuang42/fix_stdinout_error_messages 
Improve error messages on console input/output
 2020-10-06 15:58:37 -07:00
Yann Collet
83461ce963
Merge pull request #2322 from senhuang42/guard_against_stdin_for_warning_prompts 
Don't let warning messages consume input from stdin
 2020-09-30 08:26:50 -07:00
senhuang42
ce56810a32 Modify error messages on console input/output 2020-09-28 12:15:18 -04:00
senhuang42
02422db841 Fix Stdin typo 2020-09-25 11:51:35 -04:00
Yann Collet
236b98e7de
Merge pull request #2310 from senhuang42/fix_multifile_status_bar 
Fix multifile status bar and summary - clear out extraneous characters
 2020-09-25 07:48:33 -07:00
senhuang42
9f7212a48b Update unit tests 2020-09-24 16:44:33 -04:00
senhuang42
88f4410390 Add more useful failure message when stdin is an input 2020-09-24 16:29:12 -04:00
senhuang42
93d63eaeb8 Expand UTIL_requireUserConfirmation to include stdin input check 2020-09-24 15:58:06 -04:00
senhuang42
432186cbea Add FIO_determineHasStdinInput() function and member to fCtx 2020-09-24 15:55:30 -04:00
senhuang42
0e8ac6b995 Add fCtx to FIO_openDstFile() 2020-09-24 15:49:30 -04:00
senhuang42
7aa3da1cd7 Use IS_CONSOLE macro to detect that we're indeed using a console 2020-09-22 14:15:52 -04:00
senhuang42
6b6cc80196 Support .zstd suffix only for decompression 2020-09-18 12:49:51 -04:00
senhuang42
29b8f2eb08 Fix multifile decompression summary 2020-09-16 10:38:13 -04:00
senhuang42
1d5c6fda9f Add filename truncations to decompression 2020-09-16 10:28:45 -04:00
senhuang42
ab0d332563 Add display clears 2020-09-15 15:53:32 -04:00
senhuang42
cad6bf99af Add padding to remove previous line's leftovers, and keep the printed line to around the same size 2020-09-15 13:01:46 -04:00
Yann Collet
05622992d5
Merge pull request #2292 from senhuang42/multifile_status_update_fio_refactor 
Make multifile (de)compression print out a summary
 2020-09-14 14:06:26 -07:00
Yann Collet
e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads 
Allow environment variable to specify number of threads for compression
 2020-09-14 14:04:19 -07:00
W. Felix Handte
d46306087b Enable Dedicated Dict Search in the CLI 2020-09-10 18:51:52 -04:00
senhuang42
a71963c7b8 nbThreads instead of numThreads 2020-09-09 12:40:00 -04:00
senhuang42
243c8dfb41 Add include guards for init_numThreads() 2020-09-08 09:26:16 -04:00
senhuang42
cc29492c5f Update documentation for environment variable 2020-09-08 08:42:46 -04:00
senhuang42
136a0673f6 Add quick documentation for ZSTD_NUMTHREADS in the code 2020-09-07 18:19:31 -04:00
senhuang42
888c385a49 Change param name to ZSTDCLI_NUMTHREADS_DEFAULT 2020-09-07 18:11:42 -04:00
senhuang42
972e063219 Change default num_threads value 2020-09-07 18:09:48 -04:00
senhuang42
5123496104 Adjust function signatures 2020-09-07 13:13:05 -04:00
senhuang42
28a9dc78c3 Always update bytes status in FIO_ctx_t 2020-09-03 20:23:30 -04:00
senhuang42
60ee0519ca Remove extraneous FIO_ctx_t functions 2020-09-03 20:22:56 -04:00
senhuang42
48bca10792 Address memory leak in CLI from fCtx 2020-09-03 10:14:04 -04:00
senhuang42
23feaecbe5 Fix FIO_removeMultiFilesWarning conflict 2020-09-03 09:49:13 -04:00
senhuang42
49949f1d51 Merge branch 'dev' into multifile_status_update_fio_refactor 2020-09-03 09:44:47 -04:00
senhuang42
202b2954a2 Remove comma-separated decls 2020-09-03 09:28:40 -04:00
senhuang42
a480b02044 Adjust comments and struct to better reflect FIO_ctx_t 2020-09-03 09:26:30 -04:00
senhuang42
7842f43197 Fix 1 file multifile logic, remove unnecessary variable assignments 2020-09-03 09:22:07 -04:00
senhuang42
dbe5305250 Adjusted extra explanation warning to always pop up if applicable 2020-09-02 08:44:42 -04:00
senhuang42
99039988a5 Fixed newline issue and adjusted wording in comment 2020-09-01 13:18:30 -04:00
senhuang42
3a7d625d6b Cleanup comments, add function to set FIO_ctx_t.nbFilesProcessed 2020-09-01 12:54:21 -04:00
senhuang42
a03917c751 Fix the logic in printing final status updates 2020-09-01 12:46:06 -04:00
senhuang42
a6414f1247 Integrate refactor into status print for multifiles, adjust logic for printing as needed 2020-09-01 12:34:43 -04:00
senhuang42
565f116a56 Change name to ZSTD_NUMTHREADS 2020-09-01 09:02:23 -04:00
senhuang42
d54566f334 First pass at refactoring and creating new FIO_ctx_t */ 2020-08-28 11:01:04 -04:00
senhuang42
01828b27e2 Improve documentation, update man 2020-08-27 17:57:20 -04:00
senhuang42
7e867ad61f Fix potential memory leak 2020-08-26 18:52:32 -04:00
senhuang42
ef11aadc0c Fix UTIL_requireUserConfirmation() declaration 2020-08-26 17:20:37 -04:00
senhuang42
7991c55181 Move logic into new function FIO_removeMultiFilesWarning, add support for decompression 2020-08-26 16:50:20 -04:00
senhuang42
3aec385a10 Fix merge conflicts 2020-08-26 15:43:38 -04:00
Yann Collet
a8c66881e5
Merge pull request #2283 from senhuang42/progress_bars_for_multiple_files 
Refreshing progress bar for processing multiple files
 2020-08-26 11:54:50 -07:00
senhuang42
ead2387a7e Remove extraneous FIO_setNbFiles() 2020-08-26 12:05:04 -04:00
senhuang42
fed7e7850b Fix bug in user prompt where line was not flushed on negative response 2020-08-26 11:57:38 -04:00
senhuang42
b6abbc3ce5 Fix single file decompression summary, remove escape code dependency, add currFileIdx field to FIO_prefs, general cleanups/bugfixes 2020-08-26 11:35:07 -04:00
senhuang42
70fa7b37ca Added env var function for threads 2020-08-25 18:49:52 -04:00
senhuang42
96222cd195 Small formatting changes 2020-08-25 18:03:08 -04:00
senhuang42
47a9ad2ae3 Remove decompression status update 2020-08-25 17:54:00 -04:00
senhuang42
acee48d4c1 Small cleanups 2020-08-25 17:34:00 -04:00
senhuang42
a3401ca7d0 Make compatible with displaylevel = 2 and decompression 2020-08-25 17:23:47 -04:00
senhuang42
da38891a87 Add initial live update with displayLevel = 2, add new field to FIO_prefs_t 2020-08-25 16:46:47 -04:00
senhuang42
7a7cd8861a Add initial functionality to support -q 2020-08-25 13:50:44 -04:00
senhuang42
aab11ce3db Unified warning prompts into new function UTIL_requireUserConfirmationToProceed() 2020-08-25 11:25:49 -04:00
senhuang42
dde97de6c4 Only ask to proceed if using --rm, otherwise just display warning. -f bypasses it all. More robust tests 2020-08-24 20:20:39 -04:00
senhuang42
1acf243540 Add a warning whenever (de)compressing multiple files into one source, or into stdout 2020-08-24 19:10:03 -04:00
senhuang42
e1e41addb3 Adjust #ifndef logic 2020-08-24 17:36:36 -04:00
senhuang42
a030560d62 Add new DCtx param: validateChecksum and update unit tests 2020-08-24 17:28:00 -04:00
senhuang42
44c54a3e31 Addressing comments: more comments, cleanup, remove extra function, checksum logic 2020-08-24 16:14:19 -04:00
senhuang42
ffaa0df76d Document change in CLI for --no-check during decompression in --help menu 2020-08-24 09:49:12 -04:00
senhuang42
e3f5f9658a Added CLI tests for --no-check, fixed ignore checksum logic 2020-08-22 16:05:40 -04:00
senhuang42
47685ac856 Move enum into zstd.h, and fix pesky switch() logic 2020-08-21 18:18:53 -04:00
senhuang42
1b34b15e6b Adding CLI capability to invoke decompression with no checksum 2020-08-21 17:49:30 -04:00
W. Felix Handte
953f0a072a Fix MS Build 2020-08-10 17:28:34 -04:00
W. Felix Handte
b02cdf63b0 Clean Up Redundant Checks, Rename FIO_remove() -> FIO_removeFile() 2020-08-10 15:43:17 -04:00
W. Felix Handte
51ac0207af Remove UTIL_statFile() and UTIL_statDir(); Decompose Former Call-Sites 2020-08-10 15:28:02 -04:00
W. Felix Handte
93dda988c8 Remove Unused Function UTIL_fileExist() 2020-08-10 15:22:53 -04:00
W. Felix Handte
76878697a4 Re-Organize and Document Prototypes in util.h 2020-08-10 15:16:14 -04:00
W. Felix Handte
c1449143c5 Share stat() Calls in Uses of UTIL_chmod() 2020-08-05 12:10:42 -04:00
W. Felix Handte
0a8aacb4db Use stat() to Check that File Should be chmod()-ed 
Rather than special-casing a check for `/dev/null`, this uses `stat()` to
avoid `chmod()`-ing any non-regular file. I believe this is the desirable
behavior. `UTIL_chmod()` is never called on directories at the moment, only
output files.
 2020-08-05 12:00:12 -04:00
W. Felix Handte
7238cca1a1 Deduplicate Some Low-Hanging Fruit of Redundant Stat Calls 2020-08-05 01:08:34 -04:00
W. Felix Handte
44fa052599 Introduce Variants of Various UTIL Functions that Take Pre-Populated stat_t Structs 
Instead of calling `stat()`, these functions accept the result of a previous
`stat()` call on the file in question, which will allow us to make multiple
decisions around a file without redundant `stat()` calls.
 2020-08-05 01:00:06 -04:00
W. Felix Handte
b6e24bc4dc Rename UTIL_getFileStat() -> UTIL_statFile() and UTIL_getDirectoryStat() -> UTIL_statDir() 
I want to introduce versions of many of these functions that take pre-
populated `stat_t` objects and use those rather than doing their own redundant
`stat()` internally. These functions will have `...Stat()` suffixes. So this
commit renames these existing functions into the active voice, to avoid
confusion.
 2020-08-05 00:40:16 -04:00
W. Felix Handte
1a1003f996 Mark stat_t Arg to UTIL_setFileStat() const 2020-08-05 00:35:21 -04:00
W. Felix Handte
5fbc6addb6 Additionally Convert UTIL_getFileStat() Calls to UTIL_stat() Where Appropriate 2020-08-05 00:31:48 -04:00
W. Felix Handte
69cb9e7798 Use New Stat Helper 2020-08-05 00:24:32 -04:00
W. Felix Handte
b11bea56a5 Introduce Dedicated Helper to Call stat() 2020-08-05 00:21:21 -04:00
Yann Collet
15c5e20023 updated documentation 2020-07-23 12:10:57 -07:00
Yann Collet
610c41f90f improved 2-args macro to support both syntaxes 
both `--long-commmand=arg` and `--long-command arg`
are supported by macro `NEXT_FIELD()`
so that the long-command only needs to be listed once.

This extends the syntax to support new syntaxes like
`-o=FILE` or `-D=dict`,
though there is no need to advertise this capability for the time being.

Also : added `NEXT_UINT32()`,
which is wrapper around `NEXT_FIELD()`
to read integer parameters.
Use the wrapper to new field, such as `--memlimit`
which can now support both syntaxes too.
 2020-07-20 17:41:32 -07:00
Yann Collet
23941eec04 added tests for newly enabled syntax 
for --patch-from origin
and --filelist list

Also : removed some constrained syntax tests,
as the new argument parsing syntax is more permissive.

For example :
    zstd file -of dest
used to be disallowed.

It's now allowed, and understood as:
    zstd file -o dest -f
 2020-07-17 13:31:15 -07:00
Yann Collet
7280c1dafa extended 2-fields macro usage to number arguments 
leading to further simplications
 2020-07-17 13:09:23 -07:00
Yann Collet
a7e5652672 CLI: simplified handling of separated-fields arguments 
this patch reduces complexity associated with
commands requiring a separated arguments
such as :
-o filename
-D dictionary
--output-dir-flat dir
--output-dir-mirror dir

It used to be a multi-stage process with explicit context,
it's now simplified as a single macro.

Thanks to this simplification,
separated arguments logic has also been extended to
--patch-from XXX
--filelist XXX
which extends existing capability using =XXX
--patch-from=XXX
--filelist=XXX

Separated argument is useful for filenames and directories,
as it benefits from shell expansion
such as ~/dir/file
where the ~ is automatically translated by the shell.
In contrast --long-command=FILE does not interpret FILE,
so ~/ is transmitted as is to the main() function,
generally resulting in incorrect file name.
 2020-07-17 12:46:36 -07:00
Yann Collet
62df5bca4c
Merge pull request #2243 from facebook/uclibc 
fix uclibc's st_mtim
 2020-07-17 09:14:20 -07:00
Fabrice Fontaine
26d01bdb26 programs/util.c: fix build without st_mtime 
Since version 1.4.5 and commit
5af8cb7aea, if st_mtime is not defined,
programs/util.c uses utime without including utime.h which will raise
the following build failure on some of the buildroot autobuilders:

util.c: In function 'UTIL_setFileStat':
util.c:161:24: error: storage size of 'timebuf' isn't known
         struct utimbuf timebuf;
                        ^~~~~~~

Fixes:
 - http://autobuild.buildroot.org/results/be902c5d110f37bce622a2215191f155b7d3e7e0

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
 2020-07-15 21:24:13 +02:00
Yann Collet
1dcc478796 fix uclibc's st_mtim 
reported and suggested by @ewildgoose (#1872)
 2020-07-13 14:16:33 -07:00
Yann Collet
a3296dae38
Merge pull request #2219 from xxie24/output-dir-mirror 
Add output-dir-mirror option
 2020-07-10 01:21:47 -07:00
Martin Liska
5b9d6b992a
Update comment about PGO build. 2020-07-09 09:31:51 +02:00
Martin Liska
4307e870a2
Rename ZSTDMT_NBTHREADS_MAX to ZSTDMT_NBWORKERS_MAX. 
Rename it in zstd man page.
 2020-07-07 00:18:20 +02:00
Xin Xie
9a8ccd4ba3 Add output-dir-mirror option 2020-06-24 22:12:11 -07:00
Bimba Shrestha
b06710889f error when not using --stream-size on pipe 2020-06-18 09:31:06 -07:00
Bimba Shrestha
6653321ba3 passing streamSrcSize when set 2020-06-18 09:28:18 -07:00
Tobias Hieta
1c9b1ddac7 Update version documention with the support for -q 2020-06-11 10:56:47 +02:00
Tobias Hieta
b1cce9a6ad Short version string output 
This adds support for just showing the version string
without the full welcome message if the log level is
less than the default. That means that you pass -q once
and you will just see "1.4.5".

This makes it easier to parse in scripts.
 2020-06-09 08:40:41 +02:00
Bimba Shrestha
e2838d9eb9 Spelling mistakes 2020-06-05 05:11:26 -05:00
Bimba Shrestha
f847909cf2 Using ZSTD_MAX_WINDOWLOG instead of INT_MAX 2020-05-26 09:23:26 -07:00
Bimba Shrestha
01b38ef162 Change --patch-from limit from 4gb to 2gb 2020-05-26 06:52:30 -07:00
Yann Collet
f22b371e31 updated documentation for v1.4.5 
API manual and man pages
 2020-05-14 16:17:03 -07:00
Bimba Shrestha
12071467d3 reverting docs and test 2020-05-13 15:22:07 -05:00
Bimba Shrestha
9b13077cfb Revert to old zstdgrep 2020-05-13 15:12:14 -05:00
Yann Collet
f5f0bb2591
Merge pull request #2125 from Artoria2e5/patch-2 
programs/Makefile: make it actually work with clang
 2020-05-12 13:48:23 -07:00
Yann Collet
f13eb2ec01
Merge pull request #2123 from facebook/zstd_help 
updated `zstd -h`
 2020-05-11 18:54:07 -07:00
Bimba Shrestha
ef467fd428 documenting non-supported grep options for zstdgrep 2020-05-11 12:22:45 -05:00
Yann Collet
93ff2fb329
Merge pull request #2120 from facebook/clevel_doc 
updated zstd CLI documentation
 2020-05-11 09:42:33 -07:00
Mingye Wang
c3398d2dda
programs/Makefile: make it actually work with clang 
Recent versions of clang seem to require a step of conversion from the profraw file to a "compiler-readable" profdata file to let PGO work.
 2020-05-10 16:36:08 +08:00
Yann Collet
37b2dafbed minor clarification on -o file 2020-05-08 14:20:47 -07:00
Yann Collet
e5366bee16 updated zstd -h 
to help readability.
Group arguments into categories,
try to give better visibility to more common arguments.
 2020-05-08 14:14:46 -07:00
W. Felix Handte
376c26bf56 Use Unused Variable 2020-05-08 16:43:39 -04:00
Yann Collet
d564d56440 document additional compilation macros 
ZSTD_NOBENCH, ZSTD_NODICT, ZSTD_NOCOMPRESS, ZSTD_NODECOMPRESS
 2020-05-08 13:35:59 -07:00
Yann Collet
4f2cb944a8 Merge branch 'dev' into clevel_doc 2020-05-08 11:59:45 -07:00
Yann Collet
e873697625 updated documentation regarding ZSTD_CLEVEL 
make it clearer that it's useful for `tar --zstd`.
 2020-05-08 10:51:37 -07:00
Mingye Wang
37d10b6c75
programs/README.md: update help text 
I noticed that the help is outdated as there is no mention of --adapt. Y'all should probably put "update help text" into your release checklist.
 2020-05-08 02:52:18 +08:00
Felix Handte
ad8dbae1b7
Merge pull request #2103 from felixhandte/relative-includes 
Migrate Includes to Relative Paths
 2020-05-06 09:42:23 -07:00
Bimba Shrestha
e7df0d41bb
Merge pull request #2095 from bimbashrestha/grep 
[bugs] zstdgrep/grep inconsistencies
 2020-05-06 11:18:15 -05:00
Bimba Shrestha
6ff16b4246 adding help text 2020-05-05 10:03:57 -07:00
Bimba Shrestha
a06a508bf3 fixing inconsistencies 2020-05-05 09:56:15 -07:00
W. Felix Handte
7dcca6bc64 Also Move programs/ Directory to Relative Includes 2020-05-04 15:20:26 -04:00
Yann Collet
6f62a9caaa fixed zstd-nolegacy target 
when compiled as part of allVariants

(though I'm unsure why it was working before ...)
 2020-04-29 11:56:21 -07:00
Yann Collet
7ea2ae6649 added test linking user program to multi-threaded libzstd 2020-04-28 21:18:29 -07:00
Bimba Shrestha
659ff85cf5 changing to cParams.hashLog 2020-04-21 21:12:50 -05:00
Bimba Shrestha
b640802ff8 updating chainlog bound 2020-04-20 11:34:24 -07:00
Bimba Shrestha
587a20a12a adding newline 2020-04-20 10:25:58 -07:00
Bimba Shrestha
5b0a452cac
Adding --long support for --patch-from (#1959) 
* adding long support for patch-from

* adding refPrefix to dictionary_decompress

* adding refPrefix to dictionary_loader

* conversion nit

* triggering log mode on chainLog < fileLog and removing old threshold

* adding refPrefix to dictionary_round_trip

* adding docs

* adding enableldm + forceWindow test for dict

* separate patch-from logic into FIO_adjustParamsForPatchFromMode

* moving memLimit adjustment to outside ifdefs (need for decomp)

* removing refPrefix gate on dictionary_round_trip

* rebase on top of dev refPrefix change

* making sure refPrefx + ldm is < 1% of srcSize

* combining notes for patch-from

* moving memlimit logic inside fileio.c

* adding display for optimal parser and long mode trigger

* conversion nit

* fuzzer found heap-overflow fix

* another conversion nit

* moving FIO_adjustMemLimitForPatchFromMode outside ifndef

* making params immutable

* moving memLimit update before createDictBuffer call

* making maxSrcSize unsigned long long

* making dictSize and maxSrcSize params unsigned long long

* error on files larger than 4gb

* extend refPrefix test to include round trip

* conversion to size_t

* making sure ldm is at least 10x better

* removing break

* including zstd_compress_internal and removing redundant macros

* exposing ZSTD_cycleLog()

* using cycleLog instead of chainLog

* add some more docs about user optimizations

* formatting
 2020-04-17 15:58:53 -05:00
Yann Collet
458a1a1723 minor refactor 
- fix a few comments
- reorder some parameters, to enforce "mutable references first"
- simplified fwriteSparse()
 2020-04-13 14:09:57 -07:00
Bimba Shrestha
d0412f3aba no percentage on readsize == 0 2020-04-03 12:10:02 -07:00
Bimba Shrestha
bf1856c26f removing max(1, ..) 2020-04-03 12:07:17 -07:00
Nick Terrell
ac58c8d720 Fix copyright and license lines 
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
 2020-03-26 17:02:06 -07:00
Bimba Shrestha
22e7123a9a targetLen -> targetLength 2020-03-23 06:25:08 -07:00
Bimba Shrestha
9388dac2c0 Setting ZSTD_c_contentSizeFlag instead of UNKNOWN filesize 2020-03-09 15:40:18 -05:00
Bimba Shrestha
84d6bd4c7f Adding documentation 2020-03-09 14:50:49 -05:00
Bimba Shrestha
43874730f4 Adding --content-size 2020-03-09 14:19:05 -05:00
Bimba Shrestha
6d8e761ba3 Inverting content size prefs bool 2020-03-09 14:16:10 -05:00
Bimba Shrestha
03ee18d9c5 Adding (void)noContentSize 2020-03-09 14:00:52 -05:00
Bimba Shrestha
167244a64f --no-content-size cli 2020-03-09 13:07:29 -05:00
Nick Terrell
f589e2b5c8 [zstdgrep] Remove -f because zstdcat implies it 2020-03-02 11:30:12 -08:00
Nick Terrell
2c10bbd866 [zstdgrep] Log zstd decompression errors to stderr 2020-03-02 11:25:36 -08:00
Bimba Shrestha
23165b387e Adding hashLog to --show-default-cparams 2020-02-27 13:56:56 -08:00
Yann Collet
d2464776d1 man page : added --verbose long command 
also updated `zstd.1`
stacking prior updates completed on `zstd.1.md`.
 2020-02-26 14:29:30 -08:00
Felix Handte
61eeca4bd6
Merge pull request #2007 from lgarrison/cli-allow-sizet 
Allow some CLI options to accept numeric values up to size_t
 2020-02-21 08:51:37 -08:00
Lehman Garrison
834d06b655 Allow some cli options to accept numeric values up to size_t instead of unsigned int 2020-02-20 14:29:58 -05:00
Bimba Shrestha
c9ce6db069
Merge pull request #1999 from bket/fix_have_lz4 
Fix forgotten portable header prefix
 2020-02-20 10:55:28 -08:00
Philip Jones
4e728e26ca
Fix integer parsing in cli (#2003) 2020-02-18 15:30:59 -08:00
Björn Ketelaars
cdd59c50e7 Fix forgotten portable header prefix 
PR #1987 forgot to prefix another include causing lz4 detection to fail.
 2020-02-15 12:42:00 +01:00
Bimba Shrestha
06a57cf57e
[build-issue] More portable header prefix usage (#) (#1987) 
* make 4.3 build issue fix

* Changing header name and adding comment
 2020-02-06 14:10:51 -08:00
Bimba Shrestha
f627e1a1ae
Displaying level in --show-default-cparams (#1991) 
* Displaying level in --show-default-cparams

* Displaying actual enum value instead of level
 2020-02-06 13:51:43 -08:00
Bimba Shrestha
075783c1f0
Merge pull request #1980 from bimbashrestha/params 
Adding --show-default-cparams to cli
 2020-02-03 14:43:55 -08:00
Bimba Shrestha
d1389ac3ff Adding assert(NB_STRATEGIES == upperLimit) 2020-02-03 09:52:39 -08:00
Bimba Shrestha
f33baa21c6 Removing assert and changing ratio cSize 2020-01-31 11:54:14 -08:00
Bimba Shrestha
19473390d1 Adding (void)ZSTD_strategyMap 2020-01-31 10:54:02 -08:00
Bimba Shrestha
38d010eea0 Adding static assert and using it in cli 2020-01-31 10:47:17 -08:00
Bimba Shrestha
0fa6c90e89 Moving to same category as -v 2020-01-31 10:38:19 -08:00
Bimba Shrestha
f4ae6c0b94 Moving documentation and elaborating 2020-01-30 17:54:42 -08:00
Bimba Shrestha
9d9cfa9ef9 Adding assert and using ZSTD_STRATEGY_MAX as ZSTD_NB_STRATEGIES 2020-01-30 17:54:01 -08:00
Bimba Shrestha
6ad2f0d7d6 Adding back one cast and using UTIL_FILESIZE_UNKNOWN to check unknown file size 2020-01-30 17:16:02 -08:00
Bimba Shrestha
6bf05a53ab Documenting in the man file 2020-01-30 17:11:38 -08:00
Bimba Shrestha
c203341292 Removing more casts 2020-01-30 17:07:58 -08:00
Bimba Shrestha
2a0771cfdb Removing unnecessary cast 2020-01-30 17:05:33 -08:00
Bimba Shrestha
b0ceab4dc0 Moving default cparams back, checking for unknown file size, using getCParams() 2020-01-30 16:52:07 -08:00
Bimba Shrestha
b660ef87f8 Convertion nit U32 -> size_t 2020-01-30 14:29:04 -08:00
Bimba Shrestha
71754a26f8 Update comment 2020-01-30 14:14:27 -08:00
Bimba Shrestha
2f10019b92 Adding --show-default-cparams (show cparams before compressing 2020-01-30 14:12:03 -08:00
Nick Terrell
7ec87cfb28 Update comment in timefn 2020-01-22 11:17:30 -08:00
Nick Terrell
768adc0774 Fix timefn on android 2020-01-21 18:57:16 -08:00
Nick Terrell
fa48af545f Add --single-thread to zstd --help 2020-01-17 13:54:49 -08:00
Nick Terrell
f6d00c059f [util] Fix readLineFromFile on Cygwin 2020-01-13 14:37:22 -08:00
Nick Terrell
f27b4b4ec5
Merge pull request #1948 from lazka/cygwin-fix-c99-build 
Fix the c99/cmake build under Cygwin/MSYS2
 2020-01-13 11:35:28 -08:00
Christoph Reiter
ddd4c39736 Fix the c99/cmake build under Cygwin/MSYS2 
When building zst under cygwin or msys2 with std=c99 the build would fail because
of an undefined fileno()/_fileno(), which is used by the IS_CONSOLE() macro.

When building with -std=c99 (gcc otherwise defaults to gnu, which implies POSIX),
which is the default of the cmake build, then including unistd.h wont define
_POSIX_VERSION and all other headers also wont expose POSIX API.

To fix this make sure to define _POSIX_C_SOURCE with the version we want before including
unistd.h and so that _POSIX_VERSION is set to the version provided by the system.

Since Cygwin/MSYS2 just follow POSIX we can also remove their special cases for
defining IS_CONSOLE().

And, for completeness, also explicitly include stdio.h which is what actually declares fileno().

Tested with the normal make file and cmake under MSYS2 and Cygwin.
 2020-01-11 01:39:10 +01:00
Bimba Shrestha
f25a6e9f8f Adding new cli endpoint --patch-from= (#1940) 
* Adding new cli endpoint --diff-from=

* Appveyor conversion nit

* Using bool set trick instead of direct set

* Removing --diff-from and only leaving --diff-from=#

* Throwing error when both dictFileName vars are set

* Clean up syntax

* Renaming diff-from to patch-from

* Revering comma separated syntax clean up

* Updating playtests with patch-from

* Uncommenting accidentally commented

* Updating remaining docs and var names to be patch-from instead of diff-from

* Constifying

* Using existing log2 function and removing newly created one

* Argument order (moving prefs to end)

* Using comma separated syntax

* Moving to outside #ifndef
 2020-01-10 14:25:24 -08:00
Christoph Reiter
d0dcaf56c2 Make UTIL_countPhysicalCores() work under Cygwin (#1941) 
Cygwin currently uses the fallback implementation which just returns 1 every time,
which leads to bad performance when zstd is called with -T0 for example.

Instead use the POSIX implementation used for the BSDs which works just fine under Cygwin.

Tested under Cygwin and MSYS2.
 2020-01-07 15:48:26 -08:00
Bimba Shrestha
ef1684f29a [doc] Adding -M# decompression operation modifier documentation (#1938) 
* Adding -M# operation modifier documentation

* Adding long name, typo and update desc
 2020-01-03 16:36:47 -08:00
W. Felix Handte
e2a99db2dd Also Define _ATFILE_SOURCE 2019-12-23 13:24:28 -05:00
W. Felix Handte
5666835ea7 Add Comment 2019-12-23 13:24:28 -05:00
W. Felix Handte
5af8cb7aea Use statbuf->st_mtim Again 2019-12-23 13:24:28 -05:00
Yann Collet
901e545173
Merge pull request #1907 from selavy/check-fread-r1895 
Check for fread failure
 2019-12-03 15:31:11 -08:00
Yann Collet
4b4de7493f refactoring : remove global variable g_displayOut 
displaying into stderr or stdout is now an explicit operation,
no longer depending on some global state set elsewhere in the code.
 2019-12-02 16:19:03 -08:00
Yann Collet
a49417b5af fix recent issue combining -r with empty list of input files 
This would resize the table of input filenames to zero,
delivering an empty table,
to which it was no longer possible to add stdin.
 2019-12-02 14:28:18 -08:00
Yann Collet
d3ec368e13
Merge pull request #1910 from facebook/travisTest 
mingw cross compilation test
 2019-11-27 18:45:16 -08:00
Yann Collet
93ec5cfd05
Merge pull request #1883 from Ericson2314/fix-mingw-cross 
Fix MinGW cross
 2019-11-27 14:48:26 -08:00
Peter Lesslie
8eb499d354 Check for fread failure 
On failure fread may return either a short read or 0. Need to use
ferror to detect error versus eof.
 2019-11-26 20:47:58 -06:00
Yann Collet
d5b4a7ea58 removed scanbuild workaround 2019-11-26 17:46:57 -08:00
Yann Collet
96ee20758c assembleFNT() can no longer fail 2019-11-26 15:44:33 -08:00
Yann Collet
aaab618ae9 pushed aside stdio.h too 
since only UTIL_DISPLAY() depended on it.
 2019-11-26 15:25:32 -08:00
Yann Collet
7543cd055c moved UTIL_DISPLAY() inside util.c 2019-11-26 15:21:58 -08:00
Yann Collet
a684b82774 util: isolated some dependencies 
from *.h to *.c
so that they don't get transitively included
into users of util.h.
 2019-11-26 15:16:53 -08:00
Yann Collet
2d9fad4f52 fixed minor VS warning, on parameter difference 
complaining about a `const` property on one side but not the other.
 2019-11-26 14:53:37 -08:00
Yann Collet
f622c0adf3 switched UTIL_refFilename() to an assert() 2019-11-26 14:48:23 -08:00
Yann Collet
c71bd45a3b Merge branch 'dev' into ahmed_file 2019-11-26 11:20:26 -08:00
Yann Collet
5e657aca90 silence scan-build false positive 
blind attempt
 2019-11-25 15:50:58 -08:00
Yann Collet
9a3de0a535 changed name from createX to assembleX 
shows that the resulting object just takes ownership of provided buffer.
 2019-11-25 15:34:55 -08:00
Yann Collet
b1de3ecdd5 fix mode_t on Windows 2019-11-25 13:59:35 -08:00
Yann Collet
9a22140ef4 created UTIL_chmod() 
protecting "/dev/null" from having its permissions changed.

also : minor : improved consistency of util.h API
 2019-11-25 13:45:22 -08:00
Yann Collet
7aaac3f69c fix #1904 
/dev/null permissions were modified when using sudo rights.
This fixes this bug during decompression.

More importantly, this patch  adds a test, triggered in TravisCI,
ensuring unaltered /dev/null permissions.
 2019-11-25 10:35:36 -08:00
Yann Collet
d4ce04c3e7
Merge pull request #1897 from sergey-dryabzhinsky/patch-1 
WIP: Fix typo in util.c
 2019-11-22 17:25:55 -08:00
Yann Collet
c8fd2cf4c9
Merge pull request #1887 from bimbashrestha/doc-fix 
Fixing hashLog error 2^26=64Mib
 2019-11-19 16:14:20 -08:00
Sergey Dryabzhinsky
612a06eb3e
Update util.c 2019-11-19 23:24:00 +03:00
Sergey Dryabzhinsky
f62cf1fff5
Fix typo in util.c 
There must be mtim*e*
 2019-11-19 23:15:28 +03:00
Bimba Shrestha
b032a6e1d9 Updated some more outdated bounds 2019-11-19 09:55:13 -08:00
John Ericson
10552eaffe Fix MinGW cross install from Unix 
Needed a bunch of `$(EXT)`
 2019-11-14 14:19:07 -05:00
John Ericson
0ede342acc Fix MinGW cross build from Unix 
generate_res.bat seems rather pointless, am I missing something? I just
inlined it into the Makefile.
 2019-11-14 14:19:06 -05:00
Bimba Shrestha
982830b081 Fixing hashLog error 2^26=64Mib 2019-11-14 10:48:55 -08:00
John Ericson
106278e7e5 Use lowercase windows.h 
Helps for cross builds, doesn't matter on Windows itself.
 2019-11-12 19:03:01 -05:00
Yann Collet
2ed5344e84 added target zstd-dictBuilder 
check that the dictBuilder can be compiled
with only lib/common and lib/compress as dependencies.
 2019-11-06 15:46:58 -08:00
Yann Collet
9df49dc50a Visual compiler bug work-around 2019-11-06 15:23:44 -08:00
Yann Collet
be34969e5c minor man page fix 2019-11-06 15:07:35 -08:00
Yann Collet
a7e33e3e10 updated fuzz tests to use FileNamesTable* abstraction 2019-11-06 14:42:13 -08:00
Yann Collet
31a0abbfda updated pzstd and largeNbDicts to use the new FileNamesTable* abstraction 2019-11-06 09:10:05 -08:00
Yann Collet
b09f59390b refactor code to only use FileNamesTable* 2019-11-05 17:02:43 -08:00
Yann Collet
76b9e42b0b refactoring (simplification) of util.h public API 2019-11-05 14:59:45 -08:00
Yann Collet
81c51483e9 Merge branch 'dev' into ahmed_file 2019-11-05 14:44:12 -08:00
Yann Collet
47034cd6c3
Merge pull request #1850 from bimbashrestha/named_pipes 
Gating named pipe support on windows
 2019-10-30 12:18:58 -07:00
Shashank Tavildar
3c1649f139 Removed the optimization check 2019-10-29 15:59:20 -07:00
Shashank Tavildar
6d0b7bd6ce Changed extension comparision logic, added new test cases 2019-10-29 13:36:09 -07:00
Shashank Tavildar
c5060997e9 Added check to perform comparison only if extension is present 2019-10-29 12:56:04 -07:00
Shashank Tavildar
9ab6a747d4 Created utility function to extract extension from filename, fixed tests 2019-10-29 12:27:54 -07:00
Bimba Shrestha
da73b11b14
Merge branch 'dev' into named_pipes 2019-10-29 01:17:13 -04:00
Shashank Tavildar
0f2bff2faf Addressing comments, removing cyclic dependency with header file, updating tests 2019-10-28 18:21:47 -07:00
Shashank Tavildar
0e9a37daeb Fixing tests and moving flag to advanced usage 2019-10-28 15:22:26 -07:00
Yann Collet
65f2d97f33 updated UTIL_createFileNamesTable() inline documentation 2019-10-28 15:20:40 -07:00
Shashank Tavildar
f422914619 Resolving local merge conflicts 2019-10-28 15:15:13 -07:00
Yann Collet
d9c634e13b return final \0 directly from readLine() 2019-10-28 15:03:32 -07:00
Shashank Tavildar
02433e0b15 Addressing comments: -Created a list of extensions defined in fileio.h, -Updated test 2019-10-28 14:54:54 -07:00
Shashank Tavildar
48f856640e Added --exclude-compressed flag feature that skips compression of precompressed files 2019-10-28 13:57:40 -07:00
Yann Collet
d7f258d845 change command name to --filelist= 
updated documentation and man page
 2019-10-28 13:54:36 -07:00
Yann Collet
a78a8759a2 Merge branch 'dev' into ahmed_file 2019-10-28 13:43:28 -07:00
Yann Collet
12efa1ed89 yet another minor visual conversion warning 
this time for 32-bit systems
 2019-10-26 00:27:32 -07:00
Yann Collet
3e5c81ea36 fixed another minor conversion warning on Visual 
and made CONTROL() use exit() instead of abort().
 2019-10-26 00:01:11 -07:00
Yann Collet
74d872e987 fix minor conversion warning on 32-bit 2019-10-25 18:26:30 -07:00
Yann Collet
b40eaced94 minor cosmetic refactoring 2019-10-25 18:16:45 -07:00
Yann Collet
5fb84ca2cf no need to track tableBuf 
free() is compatible with NULL,
let's free() unconditionnally
 2019-10-25 17:34:29 -07:00
Yann Collet
8e414b586d test resilience on garbage file 
and clean traces
 2019-10-25 17:01:26 -07:00
Yann Collet
1ead0c5d5a improved --file=FILE implementation 
pass basic tests
 2019-10-25 16:36:59 -07:00
Shashank Tavildar
55ee7d56e4 Added --exclude-compressed flag feature that skips compression of precompressed files 2019-10-25 15:49:11 -07:00
Bimba Shrestha
5e59d51991 Merge branch 'master' of https://github.com/bimbashrestha/zstd into named_pipes 2019-10-25 15:43:41 -07:00
Bimba Shrestha
857268b32c Gating named pipe support in hedaer file 2019-10-25 15:15:28 -07:00
Bimba Shrestha
0b52d878b2 Cleaning up gate and adding comment to flag 2019-10-25 14:06:50 -07:00
Bimba Shrestha
5b45ff4f1c Gating named file support on windows 2019-10-25 11:32:38 -07:00
W. Felix Handte
4eccc82875 Minor Fixes 2019-10-25 13:59:19 -04:00
Yann Collet
14e9010bb5
Merge pull request #1844 from AhmedAbdellah19/adding_read_files_from_file_feature 
Adding --file=FILE feature
 2019-10-25 10:11:47 -07:00