Changes since 0.3.3:
- `b3sum` now supports the `--check` flag. This is intended to be a
drop-in replacement for e.g. `md5sum --check` from Coreutils. The
behavior is somewhat stricter than Coreutils with respect to invalid
Unicode in filenames. For a complete description of how `--check`
works, see the file `b3sum/what_does_check_do.md`.
- To support the `--check` feature, backslashes and newlines that appear
in filenames are now escaped in the output of `b3sum`. This is done
the same way as in Coreutils.
- To support `--check` interoperability between Unix and Windows,
backslashes in filepaths on Windows are now replaced with forward
slashes in the output of `b3sum`. Note that this is different from
Coreutils.
This is an overall cleanup of everything that b3sum is doing, especially
file opening and memory mapping, which makes it easier for the regular
hashing mode to share code with the checking mode.
Rather than breaking the check parse with more output, we'll have a rule
that a Unicode replacement character (�) in a path name automatically
fails the check.
As proposed in
https://github.com/BLAKE3-team/BLAKE3/issues/33#issuecomment-623153164
This brings b3sum behavior close to md5sum. All occurrences of backslash
are replaced with "\\", and all occurrences of (Unix) newline are
replaced with "\n". In addition, any line containing these escapes has a
single "\" prepended to the front.
Filepaths were already being converted to UTF-8 with to_string_lossy(),
but this commit adds an extra warning when that conversion is in fact
lossy (because the path is not valid Unicode). This new warning is
printed to stdout, with the goal of deliberately breaking --check (which
is not yet implemented) in this case.
I've tested manually and found that 1.0.4 is the oldest version of `cc`
that builds successfully for us. (Version 1.0.3 is missing the
`is_flag_supported` method.)
This change might help with
https://github.com/BLAKE3-team/BLAKE3/issues/83. That said, the
underlying issue there is related to "minimum supported Rust versions",
and `blake3` does not yet have an MSRV other than latest stable.
These configs and code are adapted from the CI workflow in
https://github.com/oconnor663/blake3-py, especially the
upload_github_release_asset.py script, which is copied verbatim.
Two main changes:
- In https://github.com/BLAKE3-team/BLAKE3/issues/79 we learned that
older versions of Clang require AVX-512 flags even to compile AVX-512
assembly.
- If the C compiler doesn't support AVX-512, we'd still prefer to build
the SSE4.1 and AVX2 assembly implementations, on x86_64.
Whereas vinserti64x4 is present on AVX512F, vinserti32x8 requires
AVX512DQ, which we do not test for. At this point there is not
much risk of incompatibility, since Skylake-X chips have all the
requires instruction sets, but let's be precise about this.
A helper function was incorrectly restricted to x86-only. CI doesn't
currently cover this, because benchmarks are nightly-only, and it's kind
of inconvenient to set RUSTC_BOOTSTRAP=1 through `cross` (which normally
doesn't propagate env vars). But maybe we should start...
Changes since 0.3.0:
- The x86 build now automatically falls back to "pure" Rust intrinsics,
under either of two possible conditions:
1. The `cc` crate fails to invoke a C compiler at all, indicating that
nothing of the right name (e.g. "cc" or "$CC" on Unix) is installed.
2. The `cc` crate detects that the compiler doesn't support AVX-512
flags, usually because it's too old.
The end result should be that most callers successfully build the
assembly implementations, and that callers who can't build those see a
warning but not an error. (And note that Cargo suppresses warnings for
non-path depencies.)
There are two scenarios where compiling AVX-512 C or assembly code might
not work:
1. There might not be a C compiler installed at all. Most commonly this
is either in cross-compiling situations, or with the Windows GNU
target.
2. The installed C compiler might not support e.g. -mavx512f, because
it's too old.
In both of these cases, print a relevant warning, and then automatically
fall back to using the pure Rust intrinsics build.
Note that this only affects x86 targets. Other targets always use pure
Rust, unless the "neon" feature is enabled.
Changes since version 0.2.3:
- The optimized assembly implementations are now built by default. They
perform better than the intrinsics implementations, and they compile
much more quickly. Bringing the default behavior in line with reported
benchmark figures should also simplify things for people running their
own benchmarks. Previously this crate only built Rust intrinsics
implementations by default, and the assembly implementations were
gated by the (slightly confusingly named) "c" feature. Now the "c"
feature is gone, and applications that need the old behavior can use
the new "pure" feature. Mainly this will be applications that don't
want to require a C compiler. Note that the `b3sum` crate previously
activated the "c" feature by default, so its behavior hasn't changed.
The biggest change here is that assembly implementations are enabled by
default.
Added features:
- "pure" (Pure Rust, with no C or assembly implementations.)
Removed features:
- "c" (Now basically the default.)
Renamed features;
- "c_prefer_intrinsics" -> "prefer_intrinsics"
- "c_neon" -> "neon"
Unchanged:
- "rayon"
- "std" (Still the only feature on by default.)
Changes since version 0.2.2:
- Bug fix: Commit 13556be fixes a crash on Windows when using the SSE4.1
assembly implementation (--features=c, set by default for b3sum). This
is undefined behavior and therefore a potential security issue.
- b3sum now supports the --num-threads flag.
- The C API now includes a blake3_hasher_finalize_seek() function, which
returns output from any position in the extended output stream.
- Build fix: Commit 5fad419 fixes a compiler error in the AVX-512 C
intrinsics implementation targeting the Windows GNU ABI.
Samuel Neves
Jack O'Connor
Justus K
Jack O'Connor