Commit Graph

185 Commits

Author SHA1 Message Date
Yann Collet
ec487d3265 faster CLI decompression speed for frames with -BD4 setting
lz4frame favors the faster prefix mode
when decompressing a frame with linked blocks.

This significantly improved CLI decompression on files compressed with -BD4 setting.
On my laptop, decompressing `enwik9` went from 0.89s to 0.52s.

This improvement is only for linked blocks.
It's more visible for small block sizes.
2022-08-11 17:48:35 -07:00
Yann Collet
f01b7b5209 can select validation of CRC during benchmark
on command line, using existing long command --no-frame-crc.
Note : it's effectively more than that, since _all_ checksums are disabled.
2022-07-29 22:13:38 +02:00
Yann Collet
e8f0baa3f9 added ability to skip checksum calculation when decoding LZ4 Frames 2022-07-29 21:47:34 +02:00
Yann Collet
ca6e522bff
Merge pull request #1115 from lz4/lz4f_customMem
Support for Custom Memory managers
2022-07-15 18:18:17 -07:00
Yann Collet
6784e78e00 support skippable frames within pipe
fix #977

fseek() doesn't work for pipe,
switch to "read and forget" mode in such case.
2022-07-15 19:30:53 +02:00
Yann Collet
e535d6424a implemented LZ4F_createCDict_advanced() 2022-07-14 01:36:18 +02:00
Yann Collet
a3c4f0d0a3 implemented LZ4F_createDecompressionContext_advanced() 2022-07-13 21:39:59 +02:00
Yann Collet
c76c9c53ac Merge branch 'dev' into lz4f_customMem 2022-07-13 20:43:37 +02:00
Yann Collet
270529e80e implemented first custom memory manager interface
for compression context only for the time being,
using LZ4F_createCompressionContext_advanced().

Added basic test in frametest.c
2022-07-13 20:23:13 +02:00
Yann Collet
832b444266 fix stricter enum type requirements for C++ 2022-07-13 16:18:22 +02:00
Yann Collet
7deae4bd22 minor : proper interface for LZ4F_getBlockSize()
and proper documentation.
Also : updated manual
2022-07-13 15:55:56 +02:00
Yann Collet
9de5b571d7 minor refactor : simplify LZ4F_makeBlock
one less argument
2022-07-11 03:46:15 -07:00
Alexander Mohr
0ac3c74de1
review: fix findings
* replace assert with test for LZ4F_uncompressedUpdate
* update documentation to incldue correct docstring
* remove unecessary entry point
* remove compress_linked_block_mode from fuzzing test

Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
2022-07-05 21:14:34 +02:00
Alexander Mohr
42eb47d42f
uncompressed-api: allow uncompressed_update only for independent blocks
Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
2022-07-05 11:56:23 +02:00
Alexander Mohr
e595150baf
lz4frame: correct start and size after flush
when the block mode changes a flush is executed, to prevent
mixing compressed and uncompressed data.
Prior to this commit dstStart, dstPtr, dstCapacity
where not updated to include the offset from bytesWritten.
For inputs > blockSize this meant the flushed data was
overwritten.

Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
2022-07-04 08:51:37 +02:00
Alexander Mohr
af447b22c8 meson: fix meson build
add static dependency to examples
2022-06-12 00:41:55 +02:00
Alexander Mohr
3c57d2f185 lz4frame: fix different linkage error
Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
2022-06-11 22:47:50 +02:00
Alexander Mohr
1738b50443 fuzz-test: add fuzz test for uncompressed api
add a fuzzing test for uncompressed frame api

Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
2022-06-11 22:47:44 +02:00
Alexander Mohr
62f6cef564
review: Fix review findings
This commit fixes the review findings

Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
2022-06-10 06:00:38 +00:00
Alexander Mohr
4aeb5020c3
frame-api: add method to insert uncomressed data
new method `uncompressed_update` allows to insert blocks without
compression into the lz4 stream.
The usage is documented in the frameCompress example

Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
2022-06-09 17:08:01 +02:00
Yann Collet
3bf71d35d4 introduced FORWARD_IF_ERROR()
and removed prefix from RETURN_ERROR(_IF)
to improve brevity
as it's just a local macro (no bleeding in user's namespace).
2022-05-09 21:45:07 -07:00
Yann Collet
5d0adabd28 introduced macro LZ4F_RETURN_ERROR_IF 2022-05-06 18:45:41 -07:00
Yann Collet
80ececaaed introduced macro LZ4F_RETURN_ERROR 2022-05-06 18:35:06 -07:00
Yann Collet
ebfad2da84 removed ERROR_GENERIC from lz4frame.h
created a new error code in the process : LZ4F_ERROR_compressionState_uninitialized
2022-05-06 18:19:58 -07:00
Yann Collet
da75a4096e updated one error code 2022-04-06 06:38:21 -07:00
Yann Collet
379c1a10ca Introduce MIN and MAX bounds to LZ4_MEMORY_USAGE
ensure that `frametest` works fine with these values,
notably with low LZ4_MEMORY_USAGE (dict test notably)

following suggestions from @t-mat at #1016
2022-01-30 23:02:55 -08:00
Dimitri Papadopoulos
4bdfb08b95
Fix typos found by codespell 2021-11-25 15:32:37 +01:00
Eddy Jansson
0a0922067f Don't reuse state memory that's too small.
Ensure that the memory block we're trying
to reuse is large enough for the new state.

Fixes #974
2021-07-31 19:05:29 +02:00
Yann Collet
59273b7127 fix UB lz4:988 and lz4:1178
ensure `dictBase` is only used
when there is an actual dictionary content.
2021-05-28 00:56:26 -07:00
Yann Collet
8e46846287 fix UB of lz4frame:907
now line 912
by ensuring pointer arithmetic is only performed
if there is a reason for an internal buffer to be used.
2021-05-27 22:59:22 -07:00
Yann Collet
86f5e9d0f4 minor : lz4frame include stdlib only if needed 2020-11-09 21:23:56 -08:00
Yann Collet
27959b43ec LZ4F_decompress requires a valid dctx state
This is now explicitly documented and asserted.
fix #927
2020-11-06 20:46:35 -08:00
Yann Collet
1e5ff76b36 add LZ4F_decompress() tests with (NULL,0) input and output
fix one (rare & complex) issue discovered by this test
2020-10-01 21:10:08 -07:00
Yann Collet
03f006a7ea make scan-build accept assert() 2020-10-01 10:48:22 -07:00
Yann Collet
fe74eb2031 added memcpy() related SA warning fixes
memcpy() on NULL is UB, even if length is 0.
2020-09-30 21:20:07 -07:00
Yann Collet
392809d66e fix minor static analyzer warnings
detected by scan-build and cppcheck
fix #786
2020-09-29 17:09:58 -07:00
Yann Collet
10d2e1c694 fixed lz4frame with blocks of size 1
properly track history
2020-09-17 14:43:02 -07:00
Yann Collet
e9cfa49d0d Clarifies and fix EndMark
EndMark, the 4-bytes value indicating the end of frame,
must be `0x00000000`.

Previously, it was just mentioned as a `0-size` block.
But such definition could encompass uncompressed blocks of size 0,
with a header of value `0x80000000`.

But the intention was to also support uncompressed empty blocks.
They could be used as a keep-alive signal.
Note that compressed empty blocks are already supported,
it's just that they have a size 1 instead of 0 (for the `0` token).

Unfortunately, the decoder implementation was also wrong,
and would also interpret a `0x80000000` block header as an endMark.

This issue evaded detection so far simply because
this situation never happens, as LZ4Frame always issues
a clean 0x00000000 value as a endMark.
It also does not flush empty blocks.

This is fixed in this PR.
The decoder can now deal with empty uncompressed blocks,
and do not confuse them with EndMark.
The specification is also clarified.
Finally, FrameTest is updated to randomly insert empty blocks during fuzzing.
2020-08-12 17:27:33 -07:00
Christopher Harvie
57c35511d4 fix LZ4F_compressBound when automatic flushing is enabled 2020-05-13 21:48:12 -04:00
Nick Terrell
87e52f7d5d [lz4frame] Fix unused variable warnings in fuzzing mode 2019-07-19 14:44:06 -07:00
Nick Terrell
b487660309 [lz4frame] Skip magic and checksums in fuzzing mode
When `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is defined we skip
magic and checksum checks. This makes it easier to fuzz decompression.
2019-07-18 18:45:32 -07:00
Niko Dzhus
2be2fe43a8 fix temporary buffer use when input size hint is respected 2019-05-24 22:08:44 +03:00
gstedman
98a86c8ef6 Add multiframe report to --list command 2019-05-15 21:13:19 +01:00
Brenden Eng
9e056bc032 Include block checksum in worst case scenario calculation of dstCapacity 2019-04-25 22:37:39 -04:00
Yann Collet
af35920c81 lz4frame: initializers compatibility with C++
fix #679, reported by @degski
2019-04-19 10:23:50 -07:00
Yann Collet
6fc763cd98 ensure consistent definition and usage of FREEMEM
as suggested by @sloutsky in #671
2019-04-16 11:26:03 -07:00
Yann Collet
3215069741 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev 2019-04-13 15:55:33 -07:00
Yann Collet
749c3400bd fixed incorrect assertion condition
output can use the full length of output buffer
2019-04-13 15:55:06 -07:00
Yann Collet
f8b7605034 fixed minor Visual warnings
since Visual 2017,
worries about potential overflow, which are actually impossible.
Replaced (c * a) by (c ? a : 0).
Will likely replaced a * by a cmov.
Probably harmless for performance.
2019-04-12 16:49:01 -07:00
Yann Collet
9e501702bd made LZ4F_getHeaderSize() public 2019-04-09 18:10:58 -07:00