Elliot Gorokhovsky
71c0c07c19
Allow user to specify memory limit for dictionary training
2021-12-14 14:29:01 -05:00
Felix Handte
0c26d98c0d
Merge pull request #2910 from felixhandte/reject-irregular-dicts
...
Reject Irregular Dictionary Files
2021-12-09 11:44:37 -05:00
W. Felix Handte
9985e10fda
Reject Irregular Dictionary Files
...
I hadn't seen #2890 , so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.
Addresses #2874 .
2021-12-08 16:17:04 -05:00
binhdvo
38dfc4699e
Imply -q when stderr is not a tty ( #2884 )
...
* Imply -q when stderr is not a tty
2021-12-07 16:56:19 -05:00
Nick Terrell
e7b0ae385e
Merge pull request #2890 from 15596858998/dec_1201
...
fixbug CLI's -D fails when the argument is not a regular file
2021-12-06 13:18:15 -05:00
Yann Collet
a134737ad4
Merge pull request #2687 from sapiippo/dev
...
Makefile: fix build for mingw
2021-12-02 10:53:02 -08:00
Alexander Kanavin
1e514feec6
Makefile: sort all wildcard file list expansions
...
Otherwise the order is non-deterministic and breaks
reproducible builds.
2021-12-02 12:04:11 +01:00
Yann Collet
3133d1e86e
Merge pull request #2876 from 15596858998/dev
...
Solve the bug of extra output newline character
2021-12-01 15:10:08 -08:00
15596858998
a8adfa7f67
fixbug CLI's -D fails when the argument is not a regular file
2021-12-01 21:47:35 +08:00
binhdvo
d7e1736375
Fix build for cygwin/bsd ( #2882 )
2021-11-29 14:11:39 -05:00
binhdvo
7abebc847b
Clarify documentation for -c ( #2883 )
2021-11-29 14:10:43 -05:00
15596858998
e315a047c7
更新 fileio.c
...
Solve the logic problem of wrong output of newline characters.
2021-11-23 22:08:23 +08:00
Dimitris Apostolou
ebbd675998
Fix typos
2021-11-13 10:04:04 +02:00
Kevin Svetlitski
7fbd126e08
Suppress spurious unused parameter warning
2021-11-12 10:44:26 -08:00
Kevin Svetlitski
375e3aad6c
Ensure formatting directives for displaying size_t are portable
2021-11-11 13:17:32 -08:00
Kevin Svetlitski
365c91194c
Ensure print*CParams functions are only defined when used
2021-11-11 12:14:56 -08:00
Kevin Svetlitski
df9b7755cb
Fix const-ness of FIO_displayCompressionParameters
2021-11-10 17:25:27 -08:00
Kevin Svetlitski
63fe6198ed
Display --zstd= subparameters in command-line ready form in verbose mode
2021-11-10 17:20:43 -08:00
Kevin Svetlitski
0665d4c1c2
Display command line parameters with concrete values in verbose mode
2021-11-05 12:01:20 -07:00
Kevin Svetlitski
b3888193d9
Report memory required to decompress while compressing in verbose mode
2021-11-04 16:25:40 -07:00
Yann Collet
23dd28df67
minor improvements to benchmark display
2021-10-26 23:23:30 -07:00
stanjo74
52598d54e9
Limit train samples ( #2809 )
...
* Limit training samples size to 2GB
* simplified DISPLAYLEVEL() macro to use global vqriable instead of local.
* refactored training samples loading
* fixed compiler warning
* addressed comments from the pull request
* addressed @terrelln comments
* missed some fixes
* fixed type mismatch
* Fixed bug passing estimated number of samples rather insted of the loaded number of samples.
Changed unit conversion not to use bit-shifts.
* fixed a declaration after code
* fixed type conversion compile errors
* fixed more type castting
* fixed more type mismatching
* changed sizes type to size_t
* move type casting
* more type cast fixes
2021-10-04 17:47:52 -07:00
Yann Collet
7868f38019
Merge pull request #2747 from Helflym/dev
...
Add AIX support in Makefile
2021-10-01 08:13:39 -07:00
Yann Collet
8150891939
regenerated zstdless.1
2021-09-22 14:48:51 -07:00
Yann Collet
3addf2f277
updated zstdgrep man page
2021-09-22 14:30:59 -07:00
Yann Collet
70b36c2308
update zstdgrep doc
...
to mention ripgrep alternative which transparently supports zstd-compressed files.
2021-09-22 14:30:06 -07:00
Yann Collet
999f8778af
updated man pages
...
using ronn-ng
2021-09-22 14:18:24 -07:00
Yann Collet
2832bbbbdc
emphasize usage of -r in documentation
...
notably as a way to overcome shell expansion limitations,
notably in a scenario of dictionary training.
2021-09-22 14:00:20 -07:00
senhuang42
b5c35d7ea3
Use new paramSwitch enum for LCM, row matchfinder, and block splitter
2021-09-21 14:22:02 -04:00
Nick Terrell
8bf699aa59
[build] Add support for ASM files in Make + CMake
...
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
`make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.
The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.
The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
2021-09-17 14:13:53 -07:00
Yann Collet
f58e63bee7
Merge branch 'dev' into opt_investigation
2021-09-12 01:42:49 -07:00
Yann Collet
640c5b1f77
fix automated_benchmarking
...
make it able to process text output sent into either stdout or stderr
2021-09-12 01:36:18 -07:00
Felix Handte
d68aa19a2f
Merge pull request #2749 from felixhandte/zstd-fast-pipelined
...
Pipelined Implementation of ZSTD_fast (~+5% Speed)
2021-09-09 17:05:30 -04:00
senhuang42
30fe49af4e
Fix patch-from help msg typo
2021-09-07 10:08:35 -04:00
Yann Collet
08ceda3dfc
new statistics update policy
...
small general compression ratio improvement for btopt+ strategies/
2021-09-04 00:52:44 -07:00
Yann Collet
eab692211e
removed pretty-print of sizes in benchmark
...
This is less appropriate for this mode :
benchmark is about accuracy,
it's important to read the exact values.
2021-09-03 12:51:02 -07:00
W. Felix Handte
ab8aa49b8d
Fix Benchmark Corruption Display
2021-09-01 14:15:03 -04:00
Yann Collet
f21977c5e6
fix playTests.sh when EXE_PREFIX not null
2021-08-29 17:20:12 -07:00
senhuang42
dce48f53df
Fix benchzstd error message
2021-08-23 19:10:16 -04:00
Clément Chigot
399849e236
Makefile: add AIX support
...
For lib, AIX linker doesn't allow --soname.
2021-08-13 10:25:14 +02:00
W. Felix Handte
9cd6c1ff4d
Update mtime and atime for Written Files
2021-08-04 14:49:56 -04:00
W. Felix Handte
a719edbbc3
Pull utime() Call into Helper
2021-08-04 14:49:00 -04:00
makise-homura
a5f518ae27
Change zstdcli's main() declaration due to -Wmain on some compilers
2021-07-14 19:55:47 +03:00
makise-homura
d4ad02c721
Add support for MCST LCC compiler
2021-07-10 03:57:06 +03:00
Binh Vo
6a46e38deb
Add option to use logical cores for default threads
2021-06-16 15:46:17 -04:00
binhdvo
89127e5ee2
Merge pull request #2705 from binhdvo/bootcamp
...
Add support for negative values in advanced flags
2021-06-11 14:08:23 -04:00
Binh Vo
6fad35c6a1
Add support for negative levels in --adapt=min and --adapt=max"
2021-06-11 12:13:09 -04:00
W. Felix Handte
8c00807bbc
Whitespace Fixes to Improve Cross-Line Alignment
2021-06-10 13:28:38 -04:00
W. Felix Handte
87e94e3e39
Convert Other Size Displays to Use Human-Readable Formatting
2021-06-10 12:57:46 -04:00
W. Felix Handte
2af3687c50
Switch to Binary Size Prefixes (e.g., "MB" -> "MiB")
...
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
2021-06-10 12:53:07 -04:00
W. Felix Handte
9c340ce014
Require -vv to Enable Full Precision
2021-06-10 12:53:07 -04:00
W. Felix Handte
bc46b6efe4
Apply to Other Print Statement as Well
2021-06-10 12:53:07 -04:00
W. Felix Handte
7e0058848c
Fix Whitespace
2021-06-10 12:53:07 -04:00
W. Felix Handte
93bb368d74
Change Suffix (e.g., "G" -> " GB")
2021-06-10 12:53:07 -04:00
W. Felix Handte
464bfb022e
In Verbose Mode, Preserve Full Precision Where Possible
2021-06-10 12:53:07 -04:00
W. Felix Handte
9b67219b1e
Fix Integer Constants; Fix Comparison
2021-06-10 12:53:07 -04:00
W. Felix Handte
bbb81c8801
Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting
...
This produces the following formatting:
Size | `zstd` | `ls -lh`
---------- | ------ | --------
1 | 1 | 1
12 | 12 | 12
123 | 123 | 123
1234 | 1.21K | 1.3K
12345 | 12.1K | 13K
123456 | 121K | 121K
1234567 | 1.18M | 1.2M
12345678 | 11.8M | 12M
123456789 | 118M | 118M
1234567890 | 1.15G | 1.2G
999 | 999 | 999
1000 | 1000 | 1000
1001 | 1001 | 1001
1023 | 1023 | 1023
1024 | 1.000K | 1.0K
1025 | 1.00K | 1.1K
999999 | 977K | 977K
1000000 | 977K | 977K
1000001 | 977K | 977K
1023999 | 1000K | 1000K
1024000 | 1000K | 1000K
1024001 | 1000K | 1001K
1048575 | 1024K | 1.0M
1048576 | 1.000M | 1.0M
1048577 | 1.00M | 1.1M
This was produced with the following invocation:
```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
2021-06-10 12:53:07 -04:00
Scott Baker
8e0a9695d7
Attempt to fix a failing test with help from @aqrit
2021-06-10 12:53:07 -04:00
Scott Baker
1eb852854b
Some fixes to address things @felixhandte found
2021-06-10 12:53:07 -04:00
Scott Baker
376a2730a8
Try enabling the BIG strings now the unsigned long long is in effect
2021-06-10 12:53:07 -04:00
Scott Baker
20b9b00b41
Try unsigned long long
2021-06-10 12:53:07 -04:00
Scott Baker
64385ef7cb
Update humanSize() to skip the big numbers (it requires 64 bit)
2021-06-10 12:53:07 -04:00
Scott Baker
1ef6f3d079
Use unsigned long instead to help with some tests
2021-06-10 12:53:07 -04:00
Scott Baker
e5fc830795
human_size() should use size_t
2021-06-10 12:53:07 -04:00
Scott Baker
35576e63ce
Convert tabs to spaces
2021-06-10 12:53:07 -04:00
Scott Baker
77001f00fb
Use human_size() on the "multiple files compressed" output also
2021-06-10 12:53:07 -04:00
Scott Baker
894698d3b6
Use human_size() in the benchmark output also
2021-06-10 12:53:07 -04:00
Scott Baker
4e0d9f1cc8
Move the variable declarations to the top
2021-06-10 12:53:07 -04:00
Scott Baker
eefdbcd93a
Make the variable types match
2021-06-10 12:53:07 -04:00
Scott Baker
b6b23dfe64
Convert names to CamelCase
2021-06-10 12:53:07 -04:00
Scott Baker
b70175e5ec
Put the human_size() function in util.c
2021-06-10 12:53:07 -04:00
Scott Baker
26fab1d963
Make the CLI output the file sizes in human readable format
2021-06-10 12:53:07 -04:00
binhdvo
325952f878
Revert "Add support for --long-param flag, fix #2104 "
2021-06-09 15:35:43 -04:00
Binh Vo
6583fa3f0a
Add support for --long-param flag
2021-06-09 14:07:52 -04:00
Binh Vo
d2f31b6627
Fix --progress flag to properly control progress display and default progress display on when using -v
2021-06-08 17:24:38 -04:00
Binh Vo
1e17184ad0
Add documentation for --patch-from
2021-06-03 11:12:27 -04:00
Samuli Piippo
fa2a6d4746
Makefile: fix build for mingw
...
Add ${EXT} to required places to make install succeed for mingw build.
2021-05-27 16:09:37 +03:00
Yann Collet
8fae35591e
Merge branch 'dev' of github.com:facebook/zstd into dev
2021-05-12 13:12:30 -07:00
Olivier Perret
d4548c96cb
fileio: clamp value of windowLog in patch-mode ( #2637 )
...
With small enough input files, the inferred value of fileWindowLog could
be smaller than ZSTD_WINDOWLOG_MIN.
This can be reproduced like so:
$ echo abc > small
$ echo abcdef > small2
$ zstd --patch-from small small2 -o patch
previously, this would fail with the error "zstd: error 11 : Parameter is out of bound"
2021-05-12 16:11:15 -04:00
Yann Collet
cb0cad9b79
reduce Max nb Workers to 64 in 32-bit mode
...
and restored limit to 256 when in 64-bit mode
(it was reduced to 200 to give more room for 32-bit).
This should fix test instability issues
using lot of threads in 32-bit environments.
2021-05-12 13:10:25 -07:00
Yann Collet
8a53a882f2
updated generated man pages for v1.5.0 ( #2635 )
2021-05-11 18:17:31 -04:00
sen
13449d7ce1
Add PHONY targets to makefiles ( #2629 )
2021-05-07 14:03:19 -04:00
sen
6030cdfede
Add --progress flag ( #2595 )
2021-05-06 14:50:28 -04:00
Yann Collet
2f7bbd6539
Merge pull request #2620 from facebook/winFilelist
...
fix --filelist compatibility with Windows cr+lf line ending
2021-05-06 11:35:16 -07:00
Yann Collet
df05b2ba7c
fix --filelist compatibility with Windows cr+lf line ending
2021-05-05 18:01:55 -07:00
Yann Collet
9750f3c87b
improved benchmark experience on Windows
...
benchmark results are not progressively displayed on Windows terminal.
For long benchmark sessions, nothing is displayed,
until the end, where everything is flushed.
Force display to be flushed after each update.
Updates happen roughtly every second, or even less,
so it's not a substantial workload.
2021-05-05 16:52:21 -07:00
W. Felix Handte
4f9c6fdb7f
Attempt to Fix Windows Build Error
2021-05-05 13:13:56 -04:00
W. Felix Handte
da61918c75
Also Pass Mode Bits in on Windows
...
I think in some unix emulation environments on Windows, (cygwin?) mode bits
are somehow respected. So we might as well pass them in. Can't hurt.
2021-05-05 13:10:34 -04:00
W. Felix Handte
45c4918ccf
Fix Build for Windows
2021-05-05 13:10:34 -04:00
W. Felix Handte
1fb10ba831
Don't Block Removing File on Being Able to Read It
...
`open()`'s mode bits are only applied to files that are created by the call.
If the output file already exists, but is not readable, the `fopen()` would
fail, preventing us from removing it, which would mean that the file would
not end up with the correct permission bits.
It's not clear to me why the `fopen()` is there at all. `UTIL_isRegularFile()`
should be sufficient, AFAICT.
2021-05-05 13:10:34 -04:00
W. Felix Handte
b87f97b3ea
Create Files with Desired Permissions; Avoid chmod(); Remove UTIL_chmod()
2021-05-05 13:10:34 -04:00
Felix Handte
2d10544b84
Merge pull request #2613 from felixhandte/allow-block-device
...
Allow Reading from Block Devices with `--force`
2021-05-05 13:06:32 -04:00
Yann Collet
455fd1a067
updated documentation regarding minimum job size
2021-05-05 09:03:11 -07:00
W. Felix Handte
33f3e293e8
Allow Reading from Block Devices with --force
2021-05-04 16:25:26 -04:00
Nick Terrell
a8ecf4ff88
Merge pull request #2597 from terrelln/public-headers
...
[1.5.0] Move `zstd_errors.h` and `zdict.h` to `lib/` root
2021-05-04 11:28:41 -07:00
W. Felix Handte
ee122baacf
Detect Presence of md5 on Darwin
...
This fixes #2568 .
2021-05-04 12:33:19 -04:00
Nick Terrell
09149beaf8
[1.5.0] Move zstd_errors.h and zdict.h to lib/ root
...
`zstd_errors.h` and `zdict.h` are public headers, so they deserve to be
in the root `lib/` directory with `zstd.h`, not mixed in with our private
headers.
2021-04-30 15:13:54 -07:00
Nick Terrell
4694423c4f
Add and integrate lazy row hash strategy
2021-04-07 09:53:34 -07:00
Niclas Rosenvik
e7647180cd
Stop complaining about hash tool not found
...
If build_dir is set the zstd build complains about md5sum not being found.
Fix this by checking if build_dir is set before checking and using the hash tool
just like in lib/Makefile .
2021-04-02 13:00:19 -07:00
Nick Terrell
a494308ae9
[copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
...
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
