This is an overall cleanup of everything that b3sum is doing, especially
file opening and memory mapping, which makes it easier for the regular
hashing mode to share code with the checking mode.
Rather than breaking the check parse with more output, we'll have a rule
that a Unicode replacement character (�) in a path name automatically
fails the check.
As proposed in
https://github.com/BLAKE3-team/BLAKE3/issues/33#issuecomment-623153164
This brings b3sum behavior close to md5sum. All occurrences of backslash
are replaced with "\\", and all occurrences of (Unix) newline are
replaced with "\n". In addition, any line containing these escapes has a
single "\" prepended to the front.
Filepaths were already being converted to UTF-8 with to_string_lossy(),
but this commit adds an extra warning when that conversion is in fact
lossy (because the path is not valid Unicode). This new warning is
printed to stdout, with the goal of deliberately breaking --check (which
is not yet implemented) in this case.
I've tested manually and found that 1.0.4 is the oldest version of `cc`
that builds successfully for us. (Version 1.0.3 is missing the
`is_flag_supported` method.)
This change might help with
https://github.com/BLAKE3-team/BLAKE3/issues/83. That said, the
underlying issue there is related to "minimum supported Rust versions",
and `blake3` does not yet have an MSRV other than latest stable.
These configs and code are adapted from the CI workflow in
https://github.com/oconnor663/blake3-py, especially the
upload_github_release_asset.py script, which is copied verbatim.
Two main changes:
- In https://github.com/BLAKE3-team/BLAKE3/issues/79 we learned that
older versions of Clang require AVX-512 flags even to compile AVX-512
assembly.
- If the C compiler doesn't support AVX-512, we'd still prefer to build
the SSE4.1 and AVX2 assembly implementations, on x86_64.
Whereas vinserti64x4 is present on AVX512F, vinserti32x8 requires
AVX512DQ, which we do not test for. At this point there is not
much risk of incompatibility, since Skylake-X chips have all the
requires instruction sets, but let's be precise about this.
A helper function was incorrectly restricted to x86-only. CI doesn't
currently cover this, because benchmarks are nightly-only, and it's kind
of inconvenient to set RUSTC_BOOTSTRAP=1 through `cross` (which normally
doesn't propagate env vars). But maybe we should start...
Changes since 0.3.0:
- The x86 build now automatically falls back to "pure" Rust intrinsics,
under either of two possible conditions:
1. The `cc` crate fails to invoke a C compiler at all, indicating that
nothing of the right name (e.g. "cc" or "$CC" on Unix) is installed.
2. The `cc` crate detects that the compiler doesn't support AVX-512
flags, usually because it's too old.
The end result should be that most callers successfully build the
assembly implementations, and that callers who can't build those see a
warning but not an error. (And note that Cargo suppresses warnings for
non-path depencies.)
There are two scenarios where compiling AVX-512 C or assembly code might
not work:
1. There might not be a C compiler installed at all. Most commonly this
is either in cross-compiling situations, or with the Windows GNU
target.
2. The installed C compiler might not support e.g. -mavx512f, because
it's too old.
In both of these cases, print a relevant warning, and then automatically
fall back to using the pure Rust intrinsics build.
Note that this only affects x86 targets. Other targets always use pure
Rust, unless the "neon" feature is enabled.
Changes since version 0.2.3:
- The optimized assembly implementations are now built by default. They
perform better than the intrinsics implementations, and they compile
much more quickly. Bringing the default behavior in line with reported
benchmark figures should also simplify things for people running their
own benchmarks. Previously this crate only built Rust intrinsics
implementations by default, and the assembly implementations were
gated by the (slightly confusingly named) "c" feature. Now the "c"
feature is gone, and applications that need the old behavior can use
the new "pure" feature. Mainly this will be applications that don't
want to require a C compiler. Note that the `b3sum` crate previously
activated the "c" feature by default, so its behavior hasn't changed.
The biggest change here is that assembly implementations are enabled by
default.
Added features:
- "pure" (Pure Rust, with no C or assembly implementations.)
Removed features:
- "c" (Now basically the default.)
Renamed features;
- "c_prefer_intrinsics" -> "prefer_intrinsics"
- "c_neon" -> "neon"
Unchanged:
- "rayon"
- "std" (Still the only feature on by default.)
Changes since version 0.2.2:
- Bug fix: Commit 13556be fixes a crash on Windows when using the SSE4.1
assembly implementation (--features=c, set by default for b3sum). This
is undefined behavior and therefore a potential security issue.
- b3sum now supports the --num-threads flag.
- The C API now includes a blake3_hasher_finalize_seek() function, which
returns output from any position in the extended output stream.
- Build fix: Commit 5fad419 fixes a compiler error in the AVX-512 C
intrinsics implementation targeting the Windows GNU ABI.
As part of this change, make the rayon and memmap dependencies
mandatory. This simplifies the code a lot, and I'm not aware of any
callers who build b3sum without the default dependencies.
If --num-threads is not given, or if its value is 0, b3sum will still
respect the RAYON_NUM_THREADS environment variable.
Changes since 0.2.1 (and since c-0.2.0):
- Fix a performance issue when the caller makes multiple calls to
update() with uneven lengths. (#69, reported by @willbryant.)
If the total number of chunks hashed so far is e.g. 1, and update() is
called with e.g. 8 more chunks, we can't compress all 8 together. We
have to break the input up, to make sure that that 1 lone chunk CV gets
merged with its proper sibling, and that in general the correct layout
of the tree is preserved. What we should do is hash 1-2-4-1 chunks of
input, using increasing powers of 2 (with some cleanup at the end). What
we were doing was 2-2-2-2 chunks. This was the result of a mistaken
optimization that got us stuck with an always-odd number of chunks so
far.
Fixes https://github.com/BLAKE3-team/BLAKE3/issues/69.
This release is motivated by a fix for a potential security
vulnerability. 421a21abd8 fixes a bug
introduced in a1c4c4efb5. A truncated
pointer register led to a segfault on x86-64 under Clang 7 and 8.
Clang 9 happens to be unaffected, but the behavior is undefined in
general. See also:
https://github.com/BLAKE3-team/BLAKE3/issues/60#issuecomment-585838317
The C implementation of BLAKE3 hasn't been formally packaged anywhere,
and most callers vendor code from master. This release tag is intended
to make the fix above more visible, to encourage callers to update their
vendored copies. We will continue to publish tags like this whenever
bugs in the C implementation are fixed, or if there are any incompatible
API changes.
Note that the issue above does not impact callers of the Rust `blake3`
crate. The affected file, `blake3_dispatch.c`, is not compiled by that
crate in any configuration. It does impact callers of the internal
`blake3_c_rust_bindings` crate, but that crate is not published on
crates.io and not intended for production use.
Jack O'Connor
Samuel Neves