Change "-f <file>" to not support patterns with a NUL-byte in them
under --fixed-strings. We'll now only support these under
"--perl-regexp" with PCRE v2.
A previous change to grep's documentation changed the description of
"-f <file>" to be vague enough as to not promise that this would work.
By dropping support for this we make it a whole lot easier to move
away from the kwset backend, which we'll do in a subsequent change.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The behavior of "grep" when patterns contained a NUL-byte has always
been haphazard, and has served the vagaries of the implementation more
than anything else. A pattern containing a NUL-byte can only be
provided via "-f <file>". Since pickaxe (log search) has no such flag
the NUL-byte in patterns has only ever been supported by "grep" (and
not "log --grep").
Since 9eceddeec6 ("Use kwset in grep", 2011-08-21) patterns containing
"\0" were considered fixed. In 966be95549 ("grep: add tests to fix
blind spots with \0 patterns", 2017-05-20) I added tests for this
behavior.
Change the behavior to do the obvious thing, i.e. don't silently
discard a regex pattern and make it implicitly fixed just because they
contain a NUL-byte. Instead die if the backend in question can't
handle them, e.g. --basic-regexp is combined with such a pattern.
This is desired because from a user's point of view it's the obvious
thing to do. Whether we support BRE/ERE/Perl syntax is different from
whether our implementation is limited by C-strings. These patterns are
obscure enough that I think this behavior change is OK, especially
since we never documented the old behavior.
Doing this also makes it easier to replace the kwset backend with
something else, since we'll no longer strictly need it for anything we
can't easily use another fixed-string backend for.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move the tests for "-f <file>" where "<file>" contains a NUL byte
pattern into their own file. I added most of these tests in
966be95549 ("grep: add tests to fix blind spots with \0 patterns",
2017-05-20).
Whether a regex engine supports matching binary content is very
different from whether it matches binary patterns. Since
2f8952250a ("regex: add regexec_buf() that can work on a non
NUL-terminated string", 2016-09-21) we've required REG_STARTEND of our
regex engines so we can match binary content, but only the PCRE v2
engine can sensibly match binary patterns.
Since 9eceddeec6 ("Use kwset in grep", 2011-08-21) we've been punting
patterns containing NUL-byte and considering them fixed, except in
cases where "--ignore-case" is provided and they're non-ASCII, see
5c1ebcca4d ("grep/icase: avoid kwsset on literal non-ascii strings",
2016-06-25). Subsequent commits will change this behavior.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move the "grep binary" test case added in aca20dd558 ("grep: add test
script for binary file handling", 2010-05-22) so that it lives
alongside the rest of the "grep" tests in t781*. This would have left
a gap in the t/700* namespace, so move a "filter-branch" test down,
leaving the "t7010-setup.sh" test as the next one after that.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since e944d9d932 ("grep: rewrite an if/else condition to avoid
duplicate expression", 2016-06-25) the "ascii_only" variable has only
been used once in compile_regexp(), let's just inline it there.
This makes the code easier to read, and might make it marginally
faster depending on compiler optimizations.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In 5212f91deb ("t4210: skip command-line encoding tests on mingw",
2014-07-17) the positive tests in this file were skipped. That left
the negative tests that don't produce a match.
An upcoming change to migrate the "fixed" backend of grep to PCRE v2
will cause these "log" commands to produce an error instead on
MinGW. This is because the command-line on that platform implicitly
has its encoding changed before being passed to git. See [1].
1. https://public-inbox.org/git/nycvar.QRO.7.76.6.1907011515150.44@tvgsbejvaqbjf.bet/
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Fix a bug introduced in 18547aacf5 ("grep/pcre: support utf-8",
2016-06-25) that was missed due to a blindspot in our tests, as
discussed in the previous commit. I then blindly copied the same bug
in 94da9193a6 ("grep: add support for PCRE v2", 2017-06-01) when
adding the PCRE v2 code.
We should not tell PCRE that we're processing UTF-8 just because we're
dealing with non-ASCII. In the case of e.g. "log --encoding=<...>"
under is_utf8_locale() the haystack might be in ISO-8859-1, and the
needle might be in a non-UTF-8 encoding.
Maybe we should be more strict here and die earlier? Should we also be
converting the needle to the encoding in question, and failing if it's
not a string that's valid in that encoding? Maybe.
But for now matching this as non-UTF8 at least has some hope of
producing sensible results, since we know that our default heuristic
of assuming the text to be matched is in the user locale encoding
isn't true when we've explicitly encoded it to be in a different
encoding.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Improve the tests added in 04deccda11 ("log: re-encode commit messages
before grepping", 2013-02-11) to test the regex backends. Those tests
never worked as advertised, due to the is_fixed() optimization in
grep.c (which was in place at the time), and the needle in the tests
being a fixed string.
We'd thus always use the "fixed" backend during the tests, which would
use the kwset() backend. This backend liberally accepts any garbage
input, so invalid encodings would be silently accepted.
In a follow-up commit we'll fix this bug, this test just demonstrates
the existing issue.
In practice this issue happened on Windows, see [1], but due to the
structure of the existing tests & how liberal the kwset code is about
garbage we missed this.
Cover this blind spot by testing all our regex engines. The PCRE
backend will spot these invalid encodings. It's possible that this
test breaks the "basic" and "extended" backends on some systems that
are more anal than glibc about the encoding of locale issues with
POSIX functions that I can remember, but PCRE is more careful about
the validation.
1. https://public-inbox.org/git/nycvar.QRO.7.76.6.1906271113090.44@tvgsbejvaqbjf.bet/
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The list of for-each like macros used by clang-format has been
updated.
* mo/clang-format-for-each-update:
clang-format: use git grep to generate the ForEachMacros list
The URL decoding code has been updated to avoid going past the end
of the string while parsing %-<hex>-<hex> sequence.
* md/url-parse-harden:
url: do not allow %00 to represent NUL in URLs
url: do not read past end of buffer
The description about slashes in gitignore patterns (used to
indicate things like "anchored to this level only" and "only
matches directories") has been revamped.
* an/ignore-doc-update:
gitignore.txt: make slash-rules more readable
The filter_data used in the list-objects-filter (which manages a
lazily sparse clone repository) did not use the dynamic array API
correctly---'nr' is supposed to point at one past the last element
of the array in use. This has been corrected.
* md/list-objects-filter-memfix:
list-objects-filter: correct usage of ALLOC_GROW
"git fetch" into a lazy clone forgot to fetch base objects that are
necessary to complete delta in a thin packfile, which has been
corrected.
* jt/partial-clone-missing-ref-delta-base:
t5616: cover case of client having delta base
t5616: use correct flag to check object is missing
index-pack: prefetch missing REF_DELTA bases
t5616: refactor packfile replacement
The pattern "git diff/grep" use to extract funcname and words
boundary for Rust has been added.
* ml/userdiff-rust:
userdiff: two simplifications of patterns for rust
userdiff: add built-in pattern for rust
When creating a partial clone, the object filtering criteria is
recorded for the origin of the clone, but this incorrectly used a
hardcoded name "origin" to name that remote; it has been corrected
to honor the "--origin <name>" option.
* xl/record-partial-clone-origin:
clone: respect user supplied origin name when setting up partial clone
"git request-pull" learned to warn when the ref we ask them to pull
from in the local repository and in the published repository are
different.
* pb/request-pull-verify-remote-ref:
request-pull: warn if the remote object is not the same as the local one
request-pull: quote regex metacharacters in local ref
The command line to invoke a "git cat-file" command from inside
"git p4" was not properly quoted to protect a caret and running a
broken command on Windows, which has been corrected.
* mm/p4-unshelve-windows-fix:
p4 unshelve: fix "Not a valid object name HEAD0" on Windows
"git help git" was hard to discover (well, at least for some
people).
* po/git-help-on-git-itself:
Doc: git.txt: remove backticks from link and add git-scm.com/docs
git.c: show usage for accessing the git(1) help page
A new tutorial targetting specifically aspiring git-core
developers.
* es/first-contrib-tutorial:
doc: add some nit fixes to MyFirstContribution
documentation: add anchors to MyFirstContribution
documentation: add tutorial for first contribution
The data collected by fsmonitor was not properly written back to
the on-disk index file, breaking t7519 tests occasionally, which
has been corrected.
* js/fsmonitor-unflake:
mark_fsmonitor_valid(): mark the index as changed if needed
fill_stat_cache_info(): prepare for an fsmonitor fix
Prepare use of reachability index in topological walker that works
on a range (A..B).
* ds/topo-traversal-using-commit-graph:
revision: keep topo-walk free of unintersting commits
revision: use generation for A..B --topo-order queries
The pattern "git diff/grep" use to extract funcname and words
boundary for Matlab has been extend to cover Octave, which is more
or less equivalent.
* bl/userdiff-octave:
userdiff: fix grammar and style issues
userdiff: add Octave
"git clone --recurse-submodules" learned to set up the submodules
to ignore commit object names recorded in the superproject gitlink
and instead use the commits that happen to be at the tip of the
remote-tracking branches from the get-go, by passing the new
"--remote-submodules" option.
* ba/clone-remote-submodules:
clone: add `--remote-submodules` flag
"git merge --squash" is designed to update the working tree and the
index without creating the commit, and this cannot be countermanded
by adding the "--commit" option; the command now refuses to work
when both options are given.
* vv/merge-squash-with-explicit-commit:
merge: refuse --commit with --squash
"git bundle verify" needs to see if prerequisite objects exist in
the receiving repository, but the command did not check if we are
in a repository upfront, which has been corrected.
* js/bundle-verify-require-object-store:
bundle verify: error out if called without an object database
"git am -i --resolved" segfaulted after trying to see a commit as
if it were a tree, which has been corrected.
* jk/am-i-resolved-fix:
am: fix --interactive HEAD tree resolution
am: drop tty requirement for --interactive
am: read interactive input from stdin
am: simplify prompt response handling
The server side support for "git fetch" used to show incorrect
value for the HEAD symbolic ref when the namespace feature is in
use, which has been corrected.
* jk/HEAD-symref-in-xfer-namespaces:
upload-pack: strip namespace from symref data
"git update-server-info" used to leave stale packfiles in its
output, which has been corrected.
* ew/server-info-remove-crufts:
server-info: do not list unlinked packs
A "merge -c" instruction during "git rebase --rebase-merges" should
give the user a chance to edit the log message, even when there is
otherwise no need to create a new merge and replace the existing
one (i.e. fast-forward instead), but did not. Which has been
corrected.
* pw/rebase-edit-message-for-replayed-merge:
rebase -r: always reword merge -c
The way of specifying the path to find dynamic libraries at runtime
has been simplified. The old default to pass -R/path/to/dir has been
replaced with the new default to pass -Wl,-rpath,/path/to/dir,
which is the more recent GCC uses. Those who need to build with an
old GCC can still use "CC_LD_DYNPATH=-R"
* ab/deprecate-R-for-dynpath:
Makefile: remove the NO_R_TO_GCC_LINKER flag
The ownership rule for the file descriptor to fast-import remote
backend was mixed up, leading to unrelated file descriptor getting
closed, which has been fixed.
* mh/import-transport-fd-fix:
Use xmmap_gently instead of xmmap in use_pack
dup() the input fd for fast-import used for remote helpers
"git update-server-info" learned not to rewrite the file with the
same contents.
* ew/update-server-info:
update-server-info: avoid needless overwrites
Improve the code to show args with potential typo that cannot be
interpreted as a commit-ish.
* jk/help-unknown-ref-fix:
help_unknown_ref(): check for refname ambiguity
help_unknown_ref(): duplicate collected refnames
"git format-patch" learns a configuration to set the default for
its --notes=<ref> option.
* dl/format-patch-notes-config:
format-patch: teach format.notes config option
git-format-patch.txt: document --no-notes option
"git merge" learned "--quit" option that cleans up the in-progress
merge while leaving the working tree and the index still in a mess.
* nd/merge-quit:
merge: add --quit
merge: remove drop_save() in favor of remove_merge_branch_state()
Developer support to emulate unsatisfied prerequisites in tests to
ensure that the remainer of the tests still succeeds when tests
with prerequisites are skipped.
* ab/fail-prereqs-in-test:
tests: add a special setup where prerequisites fail
"git worktree add" used to fail when another worktree connected to
the same repository was corrupt, which has been corrected.
* nd/corrupt-worktrees:
worktree add: be tolerant of corrupt worktrees
