Deduplicated files are reparse points with IO_REPARSE_TAG_DEDUP tag. Such
files are created by a dedup service running in the background, so a regular
file may be converted to a dedup reparse point at any time and without user's
intervention. For all intents and purposes dedup files should look like
normal, regular files, so it makes sense to report them as such in
Boost.Filesystem methods like status(), symlink_status() and everything
based on those. This commit implements this.
Closes https://github.com/boostorg/filesystem/issues/262.
Added ERROR_CALL_NOT_IMPLEMENTED to the list of error codes that are used
to permanently downgrade directory querying method. This error code is
returned by Wine, which up until version 7.21 did not support
FileIdExtdDirectoryRestartInfo and FileFullDirectoryRestartInfo.
Further, use non-permanent downgrade on ERROR_INVALID_PARAMETER. Apparently,
some mounted filesystems don't implement even the older info classes,
such as FileFullDirectoryRestartInfo and FileIdBothDirectoryRestartInfo.
These info classes are otherwise supported by the system and work on other
filesystems.
Lastly, if querying FileIdBothDirectoryRestartInfo fails, fall back to
NtQueryDirectoryFile API.
Fixes https://github.com/boostorg/filesystem/issues/255.
Fixes https://github.com/boostorg/filesystem/issues/266.
Closes https://github.com/boostorg/filesystem/pull/267.
Path traits and relevant path members were reworked to better support
wider range of types that are compatible with path constructors, assignment
and appending members. Added support for C++17 std::string_view,
boost::string_view and boost::container::string as the possible string
types accepted by path members.
Also extended support for types convertible to one of the string types.
Previously, user's type had to be convertible to a string with a character
type that matches the native path character type. Now all supported character
types are acceptable.
Additionally, restricted members accepting a pair of iterators to only accept
iterators whose value types are one of the supported path character types.
Lastly, path::compare and comparison operators now only accept path arguments,
relying on path conversion constructors to do the job of supporting various
source types. Also removed noexcept from compare as it is using lex_compare
and iterators internally and those can throw.
Closes https://github.com/boostorg/filesystem/issues/208.
Users are advised to use string types and iterators instead of containers
to construct/assign/append to paths.
In v4, the support for containers is removed.
During its operation, weakly_canonical would call status() on the path
consisting only from the root name of the input path. This would fail
with ERROR_INVALID_FUNCTION if the root name starts with the "\\?\" prefix,
as the root name path is not absolute.
To fix this, we don't check the status of the root name path (which is
not the correct check anyways as it tests the current directory on the
corresponding drive for existence, which is not what we want). Additionally,
avoid calling status() on the paths containing dot and dot-dot elements
during the weakly_canonical execution for the same reason - the "\\?\"
prefix disables most of the path processing in Windows APIs, including
dot and dot-dot elements resolution.
Fixes https://github.com/boostorg/filesystem/issues/247.
This follows up the previous update for POSIX.
The new implementation of remove_all on Windows Vista and later uses
NtCreateFile internal function in order to open files relative to
a previously opened directory handle, similar to POSIX openat.
Furthermore, querying file status and removing the file is now also
done through file handles to avoid performing path resolutions.
Closes https://github.com/boostorg/filesystem/issues/224.
The previous implementation could still allow for following symlinks
while remove_all is running if a directory was replaced with a symlink
higher in the tree than remove_all is currently processing. This was
reported here:
https://github.com/boostorg/filesystem/issues/224#issuecomment-1183738097
The solution is to use POSIX.1-2008 *at APIs to prevent symlink resolution
higher in the directory tree while iterating over the subtree in remove_all.
This required updating the directory iterator construction interface so that
it is possible to pass the base directory fd and return fd of the directory
used by the iterator. This is done via platform-specific params that are
currently defined only for POSIX. Additionally, status, symlink_status and
remove were extended to accept the base directory fd as well.
Other systems, including Windows, remain vulnerable.
Related to https://github.com/boostorg/filesystem/issues/224.
The copy_file_range implementation of copy_file used to set incorrect sendfile
fallback if copy_file_range failed with ENOSYS. The fallback would skip
checking the filesystem type for whether it is supported by sendfile.
Also, wrapped sendfile and copy_file_range implementations in structs to
silence clang warnings about using C++11 feature in C++03 mode: the functions
are in an anonymous namespace and therefore have internal linkage, and
pointers to such functions are not allowed to be used in non-type template
parameters in C++03.
Reproduce the workaround for GetFileInformationByHandleEx returning
ERROR_INVALID_PARAMETER when querying FILE_ATTRIBUTE_TAG_INFO on FAT/exFAT
filesystems in directory iterator construction.
Fixes https://github.com/boostorg/filesystem/issues/237.
Apparently, GetFileInformationByHandleEx(FileAttributeTagInfo) fails
with ERROR_INVALID_PARAMETER on FAT/exFAT filesystems, which used to
be interpreted as "file not found" result in (symlink_)status(). The
file is clearly present since it was successfully opened before,
and the error is presumably because the filesystem does not support
reparse points and cannot return a ReparseTag.
Check that error code and also ERROR_NOT_SUPPORTED for good measure
and fall back to the legacy code path that works for FAT/exFAT.
Fixes https://github.com/boostorg/filesystem/issues/236.
For some system files and folders like "System Volume Information"
CreateFileW fails with ERROR_ACCESS_DENIED while GetFileAttributesW
succeeds. GetFileAttributesW doesn't allow to discover whether the
file is a symlink or some other kind of a reparse point, so this
fallback will only work for files that are not reparse points,
symlinks or not. For reparse points continue to report error.
Closes https://github.com/boostorg/filesystem/issues/234.
Even though the syscall number is defined in kernel headers, the syscall
is blacklisted by seccomp in runtime.
Note that Android 11 also introduces the libc wrapper for statx, so
effectively the library will always use the libc wrapper on Android.
Reported in https://github.com/boostorg/filesystem/issues/229.
Windows 10 1709 and later support POSIX semantics for removing files,
which means the file name is removed from the filesystem namespace as
soon as the file is marked for deletion. This makes opening the file
afterwards impossible, and allows creating a new file with the same
name, even if the deleted file is still open and in use.
The implementation uses runtime detection of the feature in the OS.
We are also using two more implementations for file removal: one that
employs the more recent FILE_DISPOSITION_FLAG_IGNORE_READONLY_ATTRIBUTE
flag (available since Windows 10 1809), and FILE_DISPOSITION_INFO
structure (supported since Windows Vista). The former allows to optimize
removal of read-only files, and the latter allows to make file deletion
atomic (i.e. not prone to failure if the file is replaced on the filesystem
while the operation is executing). The implementation is chosen in
runtime, depending on which one succeeds removing a file.
Also, added support for deleting read-only directories, in addition
to non-directory files, and simplified code a little.
Closes https://github.com/boostorg/filesystem/issues/216.
We have changed v4 path::extension() to not return the filename if
it starts with a dot, but we have not made path::replace_extension
version-dependent, which made v3 behave like v4. Fixing this,
and optimizing v4 path::replace_extension in the process.
directory_iterator implementation now explicitly opens a directory
handle and relies on GetFileInformationByHandleEx or NtQueryDirectoryFile
to query information of the files in the directory. With
GetFileInformationByHandleEx, there are at least three ways to request
the information, each supported on different Windows versions and
different filesystems and providing different sets of information.
We support all three (FILE_ID_BOTH_DIR_INFO, FILE_FULL_DIR_INFO and
FILE_ID_EXTD_DIR_INFO), and we fall back to older ones if the newer
ones are not supported. GetFileInformationByHandleEx is available
since Windows Vista.
NtQueryDirectoryFile is an NT API that provides similar information to
GetFileInformationByHandleEx, but exists since at least Windows XP,
but some sources state it has existed since NT 4.0. This API is now
used when GetFileInformationByHandleEx is not available and it replaces
FindFirstFileW/FindNextFileW/FindClose based implementation (which
supposedly uses NtQueryDirectoryFile internally). Being an NT API,
we have to handle NTSTATUS error codes returned from it, so we had to
refactor unique path code that also used NTSTATUS previously.
Opening a directory handle explicitly allows to support O_NOFOLLOW
semantics on Windows, as we are able to prevent CreateFile from
following to the reparse point target. This is needed for remove_all
implementation to fix CVE-2022-21658. Similar to POSIX, remove_all_impl
on Windows now requests the directory iterator to not follow symlinks
when it is used to dive into a directory.
FindFirstFileW/FindNextFileW/FindClose based implementation is still
preserved for now for Windows CE, as it is unlikely to support neither
GetFileInformationByHandleEx nor NtQueryDirectoryFile. Given that
Windows CE has been untested for many years and is probably broken
anyway, its support is now declared deprecated. The related code,
including Find*-based directory iterator, will be removed in a
future release.
Closes https://github.com/boostorg/filesystem/issues/224.
And a couple cleanup changes:
- Since we now may use GetFileInformationByHandleEx, we are now using
it to query FILE_ATTRIBUTE_TAG_INFO for testing whether a reparse
point is a symlink.
- In resize_file_impl, we now specify all sharing flags to allow the
operation to succeed if the file is already opened by someone.
Turns out string_file.hpp was tested in operations_unit_test, though not
included directly. Since string_file.hpp contents are disabled when the
test is compiled (as any other deprecated functionality), the test failed
to compile.
Moved the string_file.hpp test to deprecated_test, which explicitly enables
deprecated functionality. Also, include string_file.hpp in filesystem.hpp
if explicitly requested by defining BOOST_FILESYSTEM_DEPRECATED.
Another process could replace the directory being processed by remove_all
with a symlink after remove_all called symlink_status but before
it creates a directory iterator. As a result, remove_all would remove
the linked directory contents instead of removing the symlink.
On POSIX systems that support fdopendir and O_NOFOLLOW flag for open(2),
this can be prevented by opening the directory with O_NOFOLLOW before
iterating. This will fail if the directory was replaced with a symlink.
No protection on other systems.
Reported in https://github.com/boostorg/filesystem/issues/224.
MSVC and possibly some other compilers that don't support __attribute__((used))
may remove the global p_init_path_globals pointer in a special data section
because it is not referenced anywhere. Add a dummy global object that
references the pointer in its constructor as a workaround.
Fixes https://github.com/boostorg/filesystem/issues/217.
Reworked remove() operation to separate POSIX and Windows implementations.
On Windows, if the file to be removed is read-only, try to reset the read-only
attribute before deleting the file. If deleting fails (other than because the
file is already deleted), try to restore the read-only attribute.
As a side effect, we were able to remove an implementation detail value from
the file_type enum that was used by the old remove() implementation.
Added a test for remove() on a read-only file on Windows. Also added tests
for remove_all(), including for cases with symlinks, hardlinks and read-only
files.
Also, corrected mklink /J argument in tests. The command accepts /j (lowercase)
to the same effect, but the formal help lists /J (uppercase) to create junctions.
Reported in https://github.com/boostorg/filesystem/issues/216.
In v4 path::lexically_normal, don't generate a trailing dot element if the
original path ends with a directory separator or dot. Also omit the trailing
directory separator the normalized path ends with a dot-dot element.
Additionally, convert directory separators to preferred separators in
root name on Windows (in v3 and v4). This may be significant for UNC paths.
Because of the changed semantics of appending operations in v4, path
composition in absolute() would produce incorrect results because at some
point it would append root directory and therefore discard root name
that was potentially added before. The updated implementation fixes that,
and also fixes the case when the input path is already absolute and
starts with a root directory, and the base path has a root name.
Previously, the returned path would contain the root name from the
base path, while the correct thing to do is to return the input path
as is.
Appending an absolute path now results in assigning the path, as
specified in C++17. This change is made for consistency with C++
and other languages that implement path manipulation (e.g. Python).
In Boost.Filesystem v3 path appending mostly worked as a slight upgrade
of concatenation, where appending would only add directory separators
when necessary, but not consider semantics of the root name and root
directory of the appended paths. This would work well for relative paths,
but produce unexpected results for paths with root names.
In v4, we now implement appending that is aware of root name and directory
of the appendedn paths. This means that appending a path with a root name
and/or directory no longer concatenates the paths, but rather rebases the
appended path on top of the source path. In particular, if the appended path
has a root name different from the source path, the append operation will
act as assignment.
This is closer to C++20 std::filesystem but not exactly the same. The
difference is for the case when the appended path is absolute. The C++20
spec requires assignment in this case, Boost.Filesystem v4 deliberately
omits this check. This is to ensure the correct result for UNC paths on
POSIX systems, where "//net/foo" / "/bar" is expected to produce "//net/bar",
not "/bar".
As part of this work, refactored path constructors and operators for more
optimal implementation and reducing the number of overloads.
Closes https://github.com/boostorg/filesystem/issues/214.
When the path ends with a non-root directory separator, no longer
produce a trailing dot element (filename). Instead, return an empty
path.
This affects not only path iterators and path::filename, but also any
other APIs that rely on them.
Closes https://github.com/boostorg/filesystem/issues/193.
The new implementation is prepared for the removal of the implicit
trailing dots in v4 path. It also no longer uses recursion
internally and therefore is better protected against stack overflows.
As a side effect of this rewrite, create_directories no longer reports
error if the input path consists entirely of dot and dot-dot elements.
This is in line with C++20 std::filesystem behavior.
Windows APIs such as GetFileAttributesW perform lexical path normalization
internally, which means e.g. "C:\a\.." resolves to an existing path
even if "C:\a" doesn't. This breaks depection of the longest sequence
of existing path elements in weakly_canonical and results in an error
in canonical that is called on that sequence.
As a workaround, perform forward iteration on Windows, so that we
stop on the first path element that doesn't exist.
Also, while at it, corrected error code reported from weakly_canonical
when status fails with an error.
Closes https://github.com/boostorg/filesystem/issues/201.
This works around recurringissues when Boost.Filesystem is used during
program termination (for example, in Boost.Log, when it performs the final
log file rotation). At that point, the path locale as well as dot and dot-dot
paths may no longer be available.
Also, MSVC 14.2 has a bug[1] that results in a deadlock whet dot or
dot-dot path is being created during program termination, while atexit
callbacks are being run in the main thread. This change works around it
as the new code does not call atexit on initialization of these paths.
This is only supported on MSVC, GCC, Clang and compatible compilers that
support MSVC-specific or GCC-specific means to customize global initialization
order.
[1]: https://github.com/boostorg/log/issues/153
Fall back to reading /dev/(u)random if getrandom fails with ENOSYS.
Also, extracted the portability macros for atomics to a separate header
to be able to use them in unique_path.cpp. Rearranged function pointers
initialization to decouple the initializer object from the particular
system calls.
For getrandom, the ENOSYS failure is only cached if the compiler supports
specifying global object initialization priority, which is needed to ensure
that the function pointer is initialized before the syscall initializer
in a different TU. If the compiler does not support this feature, just
always attempt getrandom first.
This can be useful if the syscall is present at compile time but fails with
ENOSYS at run time (for example, in Docker containers that restrict the syscall,
even if available on the host).
Additionally, marked statx syscall wrappers with attributes to disable MSAN
for them. It was reported that MSAN on clang 10 is showing errors
accessing uninitialized data in stx_mask, which must be initialized by the
syscall.
Related to https://github.com/boostorg/filesystem/issues/172
Related to https://github.com/boostorg/filesystem/issues/185
Boost.Filesystem v4 will contain breaking changes from v3 that are required
for better compatibility with C++17 std::filesystem. It will also remove
the deprecated features of v3.
Updated docs to reflect the differences between v3 and v4. Updated tests
to verify both v3 and v4 where the differences are present.
This is a breaking change.
path::filename accessor now only returns the actual filename or the implied
trailing dot element of the path, if it ends with a separator other than
root directory. This makes boost::filesystem::path behavior closer to that
of std::filesystem::path.
Updated tests and docs accordingly.
Closes https://github.com/boostorg/filesystem/issues/194.
If Windows is running in Developer mode, it is possible to specify
SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE flag to CreateSymbolicLinkW
so that the call doesn't require elevated privileges.
While at it, explicitly separated implementation of create_symlink and
create_directory_symlink for POSIX and Windows.
This doesn't compile with gcc 8 on MinGW-w64, and fails in runtime with
gcc 10.2 and clang 8.0.1 on Cygwin64 because character code conversion errors,
so basically std::filesystem never works with wide paths on Windows.
We still use wide paths as `const wchar_t*` with libc++ though.
Also, changed BOOST_FILESYSTEM_C_STR definition to accept the path as
an argument and use that definition in the tests rather than duplicating it.
Related to https://github.com/boostorg/filesystem/issues/181.
libstdc++ starting 8.1.0 in C++17 mode support std::filesystem and is able to
open the file streams with std::filesystem::path constructible from wide
strings.
libc++ starting 7.0 also supports std::filesystem::path in C++17 mode and may
also support passing `const wchar_t*` strings to open files.
Closes https://github.com/boostorg/filesystem/issues/181.
