Commit Graph

9629 Commits

Author SHA1 Message Date
Yann Collet
c0c5ffa973 streaming compression : lazy parameter adaptation with stable input
effectively makes ZSTD_c_stableInput compatible ZSTD_compressStream()
and zstd_e_continue operation mode.
2022-01-26 10:31:24 -08:00
Yann Collet
5684bae4f6 minor refactoring
on streaming compression implementation.
2022-01-26 10:31:23 -08:00
Yann Collet
fc2ea97442 refactored fuzzer tests for sequence compression api
add explicit delimiter mode to libfuzzer test
2022-01-26 00:19:35 -08:00
Yann Collet
87dcd3326a fix sequence compression API in Explicit Delimiter mode 2022-01-25 13:33:41 -08:00
brailovich
62583dc1ea
Merge pull request #1 from brailovich/brailovich-patch-1
fix for error message in recursive mode for an empty folder
2022-01-24 17:54:41 -08:00
brailovich
4021b78437
fix for error message in recursive mode for an empty folder
-r on empty directory resulted in zstd waiting input from stdin. now zstd exits without error and prints a warning message explaining why no processing happened (no files or directories to process).
2022-01-24 17:42:21 -08:00
Yann Collet
cc7d23bcec
Merge pull request #2965 from facebook/offbase
Converge sumtype (offset | repcode) numeric representation towards offBase
2022-01-24 15:47:42 -08:00
Yonatan Komornik
70df5de1b2
AsyncIO compression part 1 - refactor of existing asyncio code (#3021)
* Refactored fileio.c:
- Extracted asyncio code to fileio_asyncio.c/.h
- Moved type definitions to fileio_types.h
- Moved common macro definitions needed by both fileio.c and fileio_asyncio.c to fileio_common.h

* Bugfix - rename fileio_asycio to fileio_asyncio

* Added copyrights & license to new files

* CR fixes
2022-01-24 14:43:02 -08:00
Nick Terrell
87f81d0796
Merge pull request #3026 from trixirt/from-linux-fix
cleanup double word in comment.
2022-01-24 14:31:16 -08:00
Tom Rix
2b957afec7 cleanup double word in comment.
Remove the second 'a' and 'into'

Signed-off-by: Tom Rix <trix@redhat.com>
2022-01-24 12:43:39 -08:00
Yann Collet
feaaf7a6b1 slightly shortened status and summary lines in very verbose mode 2022-01-21 21:38:35 -08:00
binhdvo
17017ac8db
Change zstdless behavior to align with zless (#2909)
* Change zstdless behavior to align with zless
2022-01-21 19:57:19 -05:00
Yann Collet
b27356fd27
Merge pull request #3005 from cwoffenden/faster-amalgamate
Use faster Python script to amalgamate
2022-01-21 16:12:22 -08:00
Yann Collet
24318093cc slightly shortened compression status update line
to fit within 80 columns limit.
2022-01-21 14:08:46 -08:00
Yonatan Komornik
8ab95f24da
Merge pull request #2985 from yoniko/zstd-output-file-buffer
ZSTD CLI: Use buffered output
2022-01-21 13:57:05 -08:00
Yonatan Komornik
1598e6c634
Async write for decompression (#2975)
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
2022-01-21 13:55:41 -08:00
Nick Terrell
2f03c1996f
Merge pull request #3013 from WojciechMula/simplify-asm
Simplify HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop
2022-01-21 11:13:40 -08:00
Felix Handte
4a35912b3b
Merge pull request #3018 from felixhandte/gh-action-release-artifacts-trigger-fix
Trigger Release Artifact Generation on Publish
2022-01-20 21:08:59 -05:00
Yann Collet
71921e596f
Merge pull request #2983 from facebook/minLitPricev2
[opt] minor compression ratio improvement
2022-01-20 16:02:31 -08:00
W. Felix Handte
fa9cb4510a Trigger Release Artifact Generation on Publish
We previously triggered release artifact generation on release creation. We
sometimes observed that the action failed to run. I hypothesized that we were
hitting rate limiting or something. I just stumbled across [this documentat-
ion](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release), which says:

> Note: Workflows are not triggered for the `created`, `edited`, or `deleted`
> activity types for draft releases. When you create your release through the
> GitHub browser UI, your release may automatically be saved as a draft.

This must have been what was happening. This commit therefore changes the
trigger to the `published` activity. This should be more reliable.

This does have the unfortunate side effect that artifacts won't be generated
or attached until *after* the release has been published, which is what I was
trying to avoid by using the `created` activity. Oh well.
2022-01-20 17:36:28 -05:00
Felix Handte
330c97d2bf
Merge pull request #3015 from felixhandte/enable-cet-test-illegal-instruction
Add GitHub Action Checking that Zstd Runs Successfully Under CET
2022-01-20 16:18:42 -05:00
Elliot Gorokhovsky
a8f1aa2f6d
Merge pull request #3016 from embg/macro_lint
Minor lint fix
2022-01-20 13:05:02 -07:00
Felix Handte
c7e8315d88
Merge pull request #2994 from hjl-tools/hjl/cet-report/dev
x86: Append -z cet-report=error to LDFLAGS
2022-01-20 14:58:13 -05:00
Elliot Gorokhovsky
f936dd89cb Minor lint fix 2022-01-20 11:54:43 -07:00
H.J. Lu
d6fcdd123c x86: Append -z cet-report=error to LDFLAGS
Append -z cet-report=error to LDFLAGS if -fcf-protection is enabled by
default in compiler to catch the missing Intel CET marker:

compiling multi-threaded dynamic library 1.5.1
/usr/local/bin/ld: obj/conf_f408b4c825de923ffc88f7f21b6884b1/dynamic/huf_decompress_amd64.o: error: missing IBT and SHSTK properties
collect2: error: ld returned 1 exit status
...
LINK obj/conf_dbc0b41e36c44111bb0bb918e093d7c1/zstd
/usr/local/bin/ld: obj/conf_dbc0b41e36c44111bb0bb918e093d7c1/huf_decompress_amd64.o: error: missing IBT and SHSTK properties
collect2: error: ld returned 1 exit status
2022-01-20 08:32:01 -08:00
W. Felix Handte
7cf80cb94c Add GitHub Action Checking that Zstd Runs Successfully Under CET 2022-01-20 11:25:52 -05:00
Felix Handte
4dfc4eca9a
Merge pull request #2992 from hjl-tools/hjl/cet/dev
x86-64: Enable Intel CET
2022-01-20 11:24:47 -05:00
Carl Woffenden
3f181b6192 More descriptive exclusion error; updated docs and copyright 2022-01-20 14:50:31 +01:00
Wojciech Muła
e74ca7979e Simplify HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop
Get rid of three divisions. The original expression was:

    opmin := min((oend0 - op0) / 10, (oend1 - op1) / 10, (oend2 - op2) / 10, (oend3 - op3) / 10)
    r15   := min(r15, opmin)

The division by 10 can be moved outside the `min`:

    opmin := min(oend0 - op0, oend1 - op1, oend2 - op2, oend3 - op3)
    r15   := min(r15, opmin/10)
2022-01-19 18:38:46 +01:00
Carl Woffenden
dc983e7d68 Typo (and missing commit) 2022-01-19 18:05:35 +01:00
Elliot Gorokhovsky
40bc9ee6c4
Merge pull request #2999 from embg/ci_instructions_update
Update CI documentation
2022-01-19 10:03:07 -07:00
Elliot Gorokhovsky
8250faa01b Update CI documentation 2022-01-19 09:58:31 -07:00
Carl Woffenden
786263ea85 Suggestion from code review 2022-01-19 17:48:10 +01:00
Carl Woffenden
566ebce347
Python style change
Co-authored-by: Alexandre Bury <alexandre.bury@gmail.com>
2022-01-19 17:33:20 +01:00
Carl Woffenden
5fd6ddaf8b Fixed bugs found in other projects
When testing amalgamating other projects it was found: invalid Unicode errors were tripping Python's text IO, and the header search order appears differs from the shell version.
2022-01-19 16:56:03 +01:00
Carl Woffenden
dd7d29a19c Updated README 2022-01-19 15:57:33 +01:00
Carl Woffenden
7d90f0b520 Test and tidy
Made the Python more Python-like. Added notes and general tidy. Tested exclusions and building with various options. Tested all scripts.
2022-01-19 11:32:53 +01:00
Nick Terrell
ef1eafded3
Merge pull request #3009 from terrelln/asm-flags
[build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
2022-01-18 15:48:11 -08:00
Nick Terrell
3369a106eb
Merge pull request #2998 from floppym/issue2997
Avoid updating timestamps when the destination is stdout
2022-01-18 15:14:06 -08:00
Nick Terrell
8ea3d57de4 [build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
* Add `-Wa,--noexecstack` to both `ASFLAGS` and `CFLAGS`
* Pass `ASFLAGS` to `.S` compilation instead of `CFLAGS`

Fixes #3006.
2022-01-18 15:11:29 -08:00
Carl Woffenden
8f1e51f99f Feature parity with original shell script; needs further testing 2022-01-18 19:07:18 +01:00
Elliot Gorokhovsky
fe8fefb28d
Merge pull request #3000 from embg/mt_memory_readability
Documentation and minor refactor to clarify MT memory management
2022-01-18 10:18:01 -07:00
Elliot Gorokhovsky
9b6dfedf0c Documentation and minor refactor to clarify MT memory management. 2022-01-18 09:43:05 -07:00
Carl Woffenden
829ac2e9ce Work-in-progress; annotated types, added docs, parsed and resolved excluded files 2022-01-18 11:43:01 +01:00
Carl Woffenden
7e50d1e8c1 Using faster Python script to amalgamate 2022-01-17 18:59:54 +01:00
hjl-tools
ff92884a89
Merge branch 'facebook:dev' into hjl/cet/dev 2022-01-16 10:07:32 -08:00
Mike Gilbert
57a86d9ec6 Avoid updating timestamps when the destination is stdout
Fixes: 9cd6c1ff4d
Fixes: https://github.com/facebook/zstd/issues/2997
2022-01-13 16:47:18 -05:00
Felix Handte
f4a552a3fa
Merge pull request #2987 from felixhandte/prepare-v1.5.2
Prepare v1.5.2
2022-01-11 17:47:39 -05:00
H.J. Lu
51ab182bd4 x86-64: Enable Intel CET
Intel Control-flow Enforcement Technology (CET):

https://en.wikipedia.org/wiki/Control-flow_integrity#Intel_Control-flow_Enforcement_Technology

requires that on Linux, all linker input files are marked as CET enabled
in .note.gnu.property section.  For high-level language source codes,
.note.gnu.property section is added by compiler with the -fcf-protection
option.  For assembly sources, include <cet.h> to add .note.gnu.property
section.
2022-01-11 13:19:16 -08:00
Felix Handte
e113c0ca49
Merge pull request #2993 from hjl-tools/hjl/asm/dev
x86-64: Hide internal assembly functions
2022-01-11 14:31:51 -05:00