Commit Graph

678 Commits

Author SHA1 Message Date
Yann Collet
9fb4a42c7b fix #2549 2021-03-20 17:29:41 -07:00
Stephen Kitt
adb54293ab
Stop using deprecated reset?Stream functions
These are replaced by the corresponding context resets. When
converting resetCStream, CCtx_setPledgedSrcSize isn't called if the
source size is "unknown".

This helps reduce the reliance on "static only" symbols, as well as
reducing the use of deprecated functions.

Signed-off-by: Stephen Kitt <steve@sk2.org>
2021-02-23 21:56:01 +01:00
W. Felix Handte
a774c57973 Use umask() to Constrain Created File Permissions
This commit addresses #2491.

Note that a downside of this solution is that it is global: `umask()` affects
all file creation calls in the process. I believe this is safe since
`fileio.c` functions should only ever be used in the zstd binary, and these
are (almost) the only files ever created by zstd, and AIUI they're only
created in a single thread. So we can get away with messing with global state.

Note that this doesn't change the permissions of files created by `dibio.c`.
I'm not sure what those should be...
2021-02-17 15:27:39 -05:00
Yann Collet
0d793a675a removed internal dependency from CLI
ZSTD_cycleLog() is a very short function,
creating a rather large dependency onto libzstd's internal just for it is overkill.
Prefer duplicating this 2-lines function.

This PR makes the zstd CLI one step closer to being linkable to the dynamic library (see #2450)
More steps are still needed to reach this goal.
2021-01-06 01:35:52 -08:00
Nick Terrell
66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
sen
f54109c589
Merge pull request #2378 from senhuang42/free_cress_ptr
[minor] Pass cRess_t by const ptr instead of by value
2020-11-01 20:33:25 -05:00
senhuang42
043b934ba2 Pass cRess_t by const const pointer 2020-10-14 20:19:46 -04:00
senhuang42
467e4383b0 Merge branch 'dev' of github.com:senhuang42/zstd into change_ldm_mt_config 2020-10-14 10:17:50 -04:00
senhuang42
a39614de1f Fix incorrect usage of cycleLog() in --patch-from 2020-10-13 13:00:27 -04:00
senhuang42
dd3dac9ce0 Fixed logic for stdout output 2020-10-07 13:43:27 -04:00
senhuang42
1ebe360d0f Add new stdoutOutput field 2020-10-07 13:42:34 -04:00
Yann Collet
83461ce963
Merge pull request #2322 from senhuang42/guard_against_stdin_for_warning_prompts
Don't let warning messages consume input from stdin
2020-09-30 08:26:50 -07:00
Yann Collet
236b98e7de
Merge pull request #2310 from senhuang42/fix_multifile_status_bar
Fix multifile status bar and summary - clear out extraneous characters
2020-09-25 07:48:33 -07:00
senhuang42
9f7212a48b Update unit tests 2020-09-24 16:44:33 -04:00
senhuang42
93d63eaeb8 Expand UTIL_requireUserConfirmation to include stdin input check 2020-09-24 15:58:06 -04:00
senhuang42
432186cbea Add FIO_determineHasStdinInput() function and member to fCtx 2020-09-24 15:55:30 -04:00
senhuang42
0e8ac6b995 Add fCtx to FIO_openDstFile() 2020-09-24 15:49:30 -04:00
senhuang42
6b6cc80196 Support .zstd suffix only for decompression 2020-09-18 12:49:51 -04:00
senhuang42
29b8f2eb08 Fix multifile decompression summary 2020-09-16 10:38:13 -04:00
senhuang42
1d5c6fda9f Add filename truncations to decompression 2020-09-16 10:28:45 -04:00
senhuang42
ab0d332563 Add display clears 2020-09-15 15:53:32 -04:00
senhuang42
cad6bf99af Add padding to remove previous line's leftovers, and keep the printed line to around the same size 2020-09-15 13:01:46 -04:00
Yann Collet
05622992d5
Merge pull request #2292 from senhuang42/multifile_status_update_fio_refactor
Make multifile (de)compression print out a summary
2020-09-14 14:06:26 -07:00
W. Felix Handte
d46306087b Enable Dedicated Dict Search in the CLI 2020-09-10 18:51:52 -04:00
senhuang42
5123496104 Adjust function signatures 2020-09-07 13:13:05 -04:00
senhuang42
28a9dc78c3 Always update bytes status in FIO_ctx_t 2020-09-03 20:23:30 -04:00
senhuang42
60ee0519ca Remove extraneous FIO_ctx_t functions 2020-09-03 20:22:56 -04:00
senhuang42
23feaecbe5 Fix FIO_removeMultiFilesWarning conflict 2020-09-03 09:49:13 -04:00
senhuang42
49949f1d51 Merge branch 'dev' into multifile_status_update_fio_refactor 2020-09-03 09:44:47 -04:00
senhuang42
202b2954a2 Remove comma-separated decls 2020-09-03 09:28:40 -04:00
senhuang42
a480b02044 Adjust comments and struct to better reflect FIO_ctx_t 2020-09-03 09:26:30 -04:00
senhuang42
7842f43197 Fix 1 file multifile logic, remove unnecessary variable assignments 2020-09-03 09:22:07 -04:00
senhuang42
dbe5305250 Adjusted extra explanation warning to always pop up if applicable 2020-09-02 08:44:42 -04:00
senhuang42
99039988a5 Fixed newline issue and adjusted wording in comment 2020-09-01 13:18:30 -04:00
senhuang42
3a7d625d6b Cleanup comments, add function to set FIO_ctx_t.nbFilesProcessed 2020-09-01 12:54:21 -04:00
senhuang42
a03917c751 Fix the logic in printing final status updates 2020-09-01 12:46:06 -04:00
senhuang42
a6414f1247 Integrate refactor into status print for multifiles, adjust logic for printing as needed 2020-09-01 12:34:43 -04:00
senhuang42
d54566f334 First pass at refactoring and creating new FIO_ctx_t */ 2020-08-28 11:01:04 -04:00
senhuang42
01828b27e2 Improve documentation, update man 2020-08-27 17:57:20 -04:00
senhuang42
7e867ad61f Fix potential memory leak 2020-08-26 18:52:32 -04:00
senhuang42
7991c55181 Move logic into new function FIO_removeMultiFilesWarning, add support for decompression 2020-08-26 16:50:20 -04:00
senhuang42
3aec385a10 Fix merge conflicts 2020-08-26 15:43:38 -04:00
Yann Collet
a8c66881e5
Merge pull request #2283 from senhuang42/progress_bars_for_multiple_files
Refreshing progress bar for processing multiple files
2020-08-26 11:54:50 -07:00
senhuang42
fed7e7850b Fix bug in user prompt where line was not flushed on negative response 2020-08-26 11:57:38 -04:00
senhuang42
b6abbc3ce5 Fix single file decompression summary, remove escape code dependency, add currFileIdx field to FIO_prefs, general cleanups/bugfixes 2020-08-26 11:35:07 -04:00
senhuang42
96222cd195 Small formatting changes 2020-08-25 18:03:08 -04:00
senhuang42
47a9ad2ae3 Remove decompression status update 2020-08-25 17:54:00 -04:00
senhuang42
acee48d4c1 Small cleanups 2020-08-25 17:34:00 -04:00
senhuang42
a3401ca7d0 Make compatible with displaylevel = 2 and decompression 2020-08-25 17:23:47 -04:00
senhuang42
da38891a87 Add initial live update with displayLevel = 2, add new field to FIO_prefs_t 2020-08-25 16:46:47 -04:00
senhuang42
7a7cd8861a Add initial functionality to support -q 2020-08-25 13:50:44 -04:00
senhuang42
aab11ce3db Unified warning prompts into new function UTIL_requireUserConfirmationToProceed() 2020-08-25 11:25:49 -04:00
senhuang42
dde97de6c4 Only ask to proceed if using --rm, otherwise just display warning. -f bypasses it all. More robust tests 2020-08-24 20:20:39 -04:00
senhuang42
1acf243540 Add a warning whenever (de)compressing multiple files into one source, or into stdout 2020-08-24 19:10:03 -04:00
senhuang42
a030560d62 Add new DCtx param: validateChecksum and update unit tests 2020-08-24 17:28:00 -04:00
senhuang42
44c54a3e31 Addressing comments: more comments, cleanup, remove extra function, checksum logic 2020-08-24 16:14:19 -04:00
senhuang42
e3f5f9658a Added CLI tests for --no-check, fixed ignore checksum logic 2020-08-22 16:05:40 -04:00
senhuang42
47685ac856 Move enum into zstd.h, and fix pesky switch() logic 2020-08-21 18:18:53 -04:00
senhuang42
1b34b15e6b Adding CLI capability to invoke decompression with no checksum 2020-08-21 17:49:30 -04:00
W. Felix Handte
953f0a072a Fix MS Build 2020-08-10 17:28:34 -04:00
W. Felix Handte
b02cdf63b0 Clean Up Redundant Checks, Rename FIO_remove() -> FIO_removeFile() 2020-08-10 15:43:17 -04:00
W. Felix Handte
51ac0207af Remove UTIL_statFile() and UTIL_statDir(); Decompose Former Call-Sites 2020-08-10 15:28:02 -04:00
W. Felix Handte
c1449143c5 Share stat() Calls in Uses of UTIL_chmod() 2020-08-05 12:10:42 -04:00
W. Felix Handte
0a8aacb4db Use stat() to Check that File Should be chmod()-ed
Rather than special-casing a check for `/dev/null`, this uses `stat()` to
avoid `chmod()`-ing any non-regular file. I believe this is the desirable
behavior. `UTIL_chmod()` is never called on directories at the moment, only
output files.
2020-08-05 12:00:12 -04:00
W. Felix Handte
7238cca1a1 Deduplicate Some Low-Hanging Fruit of Redundant Stat Calls 2020-08-05 01:08:34 -04:00
W. Felix Handte
b6e24bc4dc Rename UTIL_getFileStat() -> UTIL_statFile() and UTIL_getDirectoryStat() -> UTIL_statDir()
I want to introduce versions of many of these functions that take pre-
populated `stat_t` objects and use those rather than doing their own redundant
`stat()` internally. These functions will have `...Stat()` suffixes. So this
commit renames these existing functions into the active voice, to avoid
confusion.
2020-08-05 00:40:16 -04:00
Xin Xie
9a8ccd4ba3 Add output-dir-mirror option 2020-06-24 22:12:11 -07:00
Bimba Shrestha
b06710889f error when not using --stream-size on pipe 2020-06-18 09:31:06 -07:00
Bimba Shrestha
6653321ba3 passing streamSrcSize when set 2020-06-18 09:28:18 -07:00
Bimba Shrestha
e2838d9eb9 Spelling mistakes 2020-06-05 05:11:26 -05:00
Bimba Shrestha
f847909cf2 Using ZSTD_MAX_WINDOWLOG instead of INT_MAX 2020-05-26 09:23:26 -07:00
Bimba Shrestha
01b38ef162 Change --patch-from limit from 4gb to 2gb 2020-05-26 06:52:30 -07:00
W. Felix Handte
7dcca6bc64 Also Move programs/ Directory to Relative Includes 2020-05-04 15:20:26 -04:00
Bimba Shrestha
659ff85cf5 changing to cParams.hashLog 2020-04-21 21:12:50 -05:00
Bimba Shrestha
b640802ff8 updating chainlog bound 2020-04-20 11:34:24 -07:00
Bimba Shrestha
587a20a12a adding newline 2020-04-20 10:25:58 -07:00
Bimba Shrestha
5b0a452cac
Adding --long support for --patch-from (#1959)
* adding long support for patch-from

* adding refPrefix to dictionary_decompress

* adding refPrefix to dictionary_loader

* conversion nit

* triggering log mode on chainLog < fileLog and removing old threshold

* adding refPrefix to dictionary_round_trip

* adding docs

* adding enableldm + forceWindow test for dict

* separate patch-from logic into FIO_adjustParamsForPatchFromMode

* moving memLimit adjustment to outside ifdefs (need for decomp)

* removing refPrefix gate on dictionary_round_trip

* rebase on top of dev refPrefix change

* making sure refPrefx + ldm is < 1% of srcSize

* combining notes for patch-from

* moving memlimit logic inside fileio.c

* adding display for optimal parser and long mode trigger

* conversion nit

* fuzzer found heap-overflow fix

* another conversion nit

* moving FIO_adjustMemLimitForPatchFromMode outside ifndef

* making params immutable

* moving memLimit update before createDictBuffer call

* making maxSrcSize unsigned long long

* making dictSize and maxSrcSize params unsigned long long

* error on files larger than 4gb

* extend refPrefix test to include round trip

* conversion to size_t

* making sure ldm is at least 10x better

* removing break

* including zstd_compress_internal and removing redundant macros

* exposing ZSTD_cycleLog()

* using cycleLog instead of chainLog

* add some more docs about user optimizations

* formatting
2020-04-17 15:58:53 -05:00
Yann Collet
458a1a1723 minor refactor
- fix a few comments
- reorder some parameters, to enforce "mutable references first"
- simplified fwriteSparse()
2020-04-13 14:09:57 -07:00
Bimba Shrestha
d0412f3aba no percentage on readsize == 0 2020-04-03 12:10:02 -07:00
Bimba Shrestha
bf1856c26f removing max(1, ..) 2020-04-03 12:07:17 -07:00
Nick Terrell
ac58c8d720 Fix copyright and license lines
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
2020-03-26 17:02:06 -07:00
Bimba Shrestha
9388dac2c0 Setting ZSTD_c_contentSizeFlag instead of UNKNOWN filesize 2020-03-09 15:40:18 -05:00
Bimba Shrestha
6d8e761ba3 Inverting content size prefs bool 2020-03-09 14:16:10 -05:00
Bimba Shrestha
167244a64f --no-content-size cli 2020-03-09 13:07:29 -05:00
Bimba Shrestha
f25a6e9f8f Adding new cli endpoint --patch-from= (#1940)
* Adding new cli endpoint --diff-from=

* Appveyor conversion nit

* Using bool set trick instead of direct set

* Removing --diff-from and only leaving --diff-from=#

* Throwing error when both dictFileName vars are set

* Clean up syntax

* Renaming diff-from to patch-from

* Revering comma separated syntax clean up

* Updating playtests with patch-from

* Uncommenting accidentally commented

* Updating remaining docs and var names to be patch-from instead of diff-from

* Constifying

* Using existing log2 function and removing newly created one

* Argument order (moving prefs to end)

* Using comma separated syntax

* Moving to outside #ifndef
2020-01-10 14:25:24 -08:00
Peter Lesslie
8eb499d354 Check for fread failure
On failure fread may return either a short read or 0. Need to use
ferror to detect error versus eof.
2019-11-26 20:47:58 -06:00
Yann Collet
9a22140ef4 created UTIL_chmod()
protecting "/dev/null" from having its permissions changed.

also : minor : improved consistency of util.h API
2019-11-25 13:45:22 -08:00
Yann Collet
7aaac3f69c fix #1904
/dev/null permissions were modified when using sudo rights.
This fixes this bug during decompression.

More importantly, this patch  adds a test, triggered in TravisCI,
ensuring unaltered /dev/null permissions.
2019-11-25 10:35:36 -08:00
Yann Collet
47034cd6c3
Merge pull request #1850 from bimbashrestha/named_pipes
Gating named pipe support on windows
2019-10-30 12:18:58 -07:00
Shashank Tavildar
9ab6a747d4 Created utility function to extract extension from filename, fixed tests 2019-10-29 12:27:54 -07:00
Bimba Shrestha
da73b11b14
Merge branch 'dev' into named_pipes 2019-10-29 01:17:13 -04:00
Shashank Tavildar
0f2bff2faf Addressing comments, removing cyclic dependency with header file, updating tests 2019-10-28 18:21:47 -07:00
Shashank Tavildar
02433e0b15 Addressing comments: -Created a list of extensions defined in fileio.h, -Updated test 2019-10-28 14:54:54 -07:00
Shashank Tavildar
48f856640e Added --exclude-compressed flag feature that skips compression of precompressed files 2019-10-28 13:57:40 -07:00
Bimba Shrestha
5e59d51991 Merge branch 'master' of https://github.com/bimbashrestha/zstd into named_pipes 2019-10-25 15:43:41 -07:00
Bimba Shrestha
0b52d878b2 Cleaning up gate and adding comment to flag 2019-10-25 14:06:50 -07:00
Bimba Shrestha
5b45ff4f1c Gating named file support on windows 2019-10-25 11:32:38 -07:00
W. Felix Handte
4eccc82875 Minor Fixes 2019-10-25 13:59:19 -04:00
W. Felix Handte
91c3f545cc Fix Build; Refactor 2019-10-24 20:18:57 -04:00
Felix Handte
506e1a1176
Merge branch 'tzst-ext-tmp' into tzst-ext 2019-10-24 17:49:34 -04:00