2005-05-14 07:50:32 +02:00
|
|
|
# Run tests
|
|
|
|
|
#
|
|
|
|
|
# Copyright (c) 2005 Junio C Hamano
|
|
|
|
|
#
|
2005-05-14 17:58:22 +02:00
|
|
|
|
2010-05-14 11:31:38 +02:00
|
|
|
-include ../config.mak.autogen
|
2009-08-09 10:50:37 +02:00
|
|
|
-include ../config.mak
|
|
|
|
|
|
2012-12-09 11:36:17 +01:00
|
|
|
#GIT_TEST_OPTS = --verbose --debug
|
2005-09-24 21:50:29 +02:00
|
|
|
SHELL_PATH ?= $(SHELL)
|
t/Makefile: introduce TEST_SHELL_PATH
You may want to run the test suite with a different shell
than you use to build Git. For instance, you may build with
SHELL_PATH=/bin/sh (because it's faster, or it's what you
expect to exist on systems where the build will be used) but
want to run the test suite with bash (e.g., since that
allows using "-x" reliably across the whole test suite).
There's currently no good way to do this.
You might think that doing two separate make invocations,
like:
make &&
make -C t SHELL_PATH=/bin/bash
would work. And it _almost_ does. The second make will see
our bash SHELL_PATH, and we'll use that to run the
individual test scripts (or tell prove to use it to do so).
So far so good.
But this breaks down when "--tee" or "--verbose-log" is
used. Those options cause the test script to actually
re-exec itself using $SHELL_PATH. But wait, wouldn't our
second make invocation have set SHELL_PATH correctly in the
environment?
Yes, but test-lib.sh sources GIT-BUILD-OPTIONS, which we
built during the first "make". And that overrides the
environment, giving us the original SHELL_PATH again.
Let's introduce a new variable that lets you specify a
specific shell to be run for the test scripts. Note that we
have to touch both the main and t/ Makefiles, since we have
to record it in GIT-BUILD-OPTIONS in one, and use it in the
latter.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-12-08 11:47:22 +01:00
|
|
|
TEST_SHELL_PATH ?= $(SHELL_PATH)
|
2010-08-08 16:49:26 +02:00
|
|
|
PERL_PATH ?= /usr/bin/perl
|
2005-09-30 22:31:16 +02:00
|
|
|
TAR ?= $(TAR)
|
2007-07-14 19:51:44 +02:00
|
|
|
RM ?= rm -f
|
2010-10-14 10:53:36 +02:00
|
|
|
PROVE ?= prove
|
|
|
|
|
DEFAULT_TEST_TARGET ?= test
|
2014-07-09 21:34:42 +02:00
|
|
|
TEST_LINT ?= test-lint
|
2005-05-14 07:50:32 +02:00
|
|
|
|
2013-04-26 20:55:52 +02:00
|
|
|
ifdef TEST_OUTPUT_DIRECTORY
|
2013-05-06 14:35:46 +02:00
|
|
|
TEST_RESULTS_DIRECTORY = $(TEST_OUTPUT_DIRECTORY)/test-results
|
2013-04-26 20:55:52 +02:00
|
|
|
else
|
|
|
|
|
TEST_RESULTS_DIRECTORY = test-results
|
|
|
|
|
endif
|
|
|
|
|
|
2005-10-10 22:50:01 +02:00
|
|
|
# Shell quote;
|
2006-02-18 12:40:22 +01:00
|
|
|
SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
|
t/Makefile: introduce TEST_SHELL_PATH
You may want to run the test suite with a different shell
than you use to build Git. For instance, you may build with
SHELL_PATH=/bin/sh (because it's faster, or it's what you
expect to exist on systems where the build will be used) but
want to run the test suite with bash (e.g., since that
allows using "-x" reliably across the whole test suite).
There's currently no good way to do this.
You might think that doing two separate make invocations,
like:
make &&
make -C t SHELL_PATH=/bin/bash
would work. And it _almost_ does. The second make will see
our bash SHELL_PATH, and we'll use that to run the
individual test scripts (or tell prove to use it to do so).
So far so good.
But this breaks down when "--tee" or "--verbose-log" is
used. Those options cause the test script to actually
re-exec itself using $SHELL_PATH. But wait, wouldn't our
second make invocation have set SHELL_PATH correctly in the
environment?
Yes, but test-lib.sh sources GIT-BUILD-OPTIONS, which we
built during the first "make". And that overrides the
environment, giving us the original SHELL_PATH again.
Let's introduce a new variable that lets you specify a
specific shell to be run for the test scripts. Note that we
have to touch both the main and t/ Makefiles, since we have
to record it in GIT-BUILD-OPTIONS in one, and use it in the
latter.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-12-08 11:47:22 +01:00
|
|
|
TEST_SHELL_PATH_SQ = $(subst ','\'',$(TEST_SHELL_PATH))
|
2013-01-03 00:20:19 +01:00
|
|
|
PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH))
|
2013-04-26 20:55:52 +02:00
|
|
|
TEST_RESULTS_DIRECTORY_SQ = $(subst ','\'',$(TEST_RESULTS_DIRECTORY))
|
2005-10-10 22:50:01 +02:00
|
|
|
|
t/Makefile: Use $(sort ...) explicitly where needed
Starting from GNU Make 3.82 $(wildcard ...) no longer sorts the result
(from NEWS):
* WARNING: Backward-incompatibility!
Wildcards were not documented as returning sorted values, but the results
have been sorted up until this release.. If your makefiles require sorted
results from wildcard expansions, use the $(sort ...) function to request
it explicitly.
http://repo.or.cz/w/make.git/commitdiff/2a59dc32aaf0681dec569f32a9d7ab88a379d34f
I usually watch test progress visually, and if tests are sorted, even
with make -j4 they go more or less incrementally by their t number. On
the other side, without sorting, tests are executed in seemingly random
order even for -j1. Let's please maintain sane tests order for perceived
prettyness.
Another note is that in GNU Make sort also works as uniq, so after sort
being removed, we might expect e.g. $(wildcard *.sh a.*) to produce
duplicates for e.g. "a.sh". From this point of view, adding sort could
be seen as hardening t/Makefile from accidentally introduced dups.
It turned out that prevous releases of GNU Make did not perform full
sort in $(wildcard), only sorting results for each pattern, that's why
explicit sort-as-uniq is relevant even for older makes.
Signed-off-by: Kirill Smelkov <kirr@navytux.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-09-03 22:41:21 +02:00
|
|
|
T = $(sort $(wildcard t[0-9][0-9][0-9][0-9]-*.sh))
|
|
|
|
|
TGITWEB = $(sort $(wildcard t95[0-9][0-9]-*.sh))
|
2014-07-09 21:34:12 +02:00
|
|
|
THELPERS = $(sort $(filter-out $(T),$(wildcard *.sh)))
|
2005-05-14 07:50:32 +02:00
|
|
|
|
2010-10-14 10:53:36 +02:00
|
|
|
all: $(DEFAULT_TEST_TARGET)
|
|
|
|
|
|
2010-12-13 18:22:38 +01:00
|
|
|
test: pre-clean $(TEST_LINT)
|
tests: Clarify dependencies between tests, 'aggregate-results' and 'clean'
The Makefile targets 'aggregate-results' and 'clean' pretended to be
independent. This is not true, of course, since aggregate-results
needs the results _before_ they are removed.
Likewise, the tests should have been run already when the results are
to be aggregated.
However, as it is legitimate to run only a few tests, and then aggregate
just those results, so another target is introduced, that depends on all
tests, then aggregates the results, and only then removes the results.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-08-08 07:59:18 +02:00
|
|
|
$(MAKE) aggregate-results-and-cleanup
|
2005-11-08 10:51:10 +01:00
|
|
|
|
t/Makefile: add a rule to re-run previously-failed tests
This patch automates the process of determining which tests failed
previously and re-running them.
While developing patch series, it is a good practice to run the test
suite from time to time, just to make sure that obvious bugs are caught
early. With complex patch series, it is common to run `make -j15 -k
test`, i.e. run the tests in parallel and *not* stop at the first
failing test but continue. This has the advantage of identifying
possibly multiple problems in one big test run.
It is particularly important to reduce the turn-around time thusly on
Windows, where the test suite spends 45 minutes on the computer on which
this patch was developed.
It is the most convenient way to determine which tests failed after
running the entire test suite, in parallel, to look for left-over "trash
directory.t*" subdirectories in the t/ subdirectory. However, those
directories might live outside t/ when overridden using the
--root=<directory> option, to which the Makefile has no access. The next
best method is to grep explicitly for failed tests in the test-results/
directory, which the Makefile *can* access.
Please note that the often-recommended `prove` tool requires Perl, and
that opens a whole new can of worms on Windows. As no native Windows Perl
comes with Subversion bindings, we have to use a Perl in Git for Windows
that uses the POSIX emulation layer named MSYS2 (which is a portable
version of Cygwin). When using this emulation layer under stress, e.g.
when running massively-parallel tests, unexplicable crashes occur quite
frequently, and instead of having a solution to the original problem, the
developer now has an additional, quite huge problem. For that reason, this
developer rejected `prove` as a solution and went with this patch instead.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-27 18:21:30 +01:00
|
|
|
failed:
|
|
|
|
|
@failed=$$(cd '$(TEST_RESULTS_DIRECTORY_SQ)' && \
|
|
|
|
|
grep -l '^failed [1-9]' *.counts | \
|
|
|
|
|
sed -n 's/\.counts$$/.sh/p') && \
|
|
|
|
|
test -z "$$failed" || $(MAKE) $$failed
|
|
|
|
|
|
2010-12-13 18:22:38 +01:00
|
|
|
prove: pre-clean $(TEST_LINT)
|
t/Makefile: introduce TEST_SHELL_PATH
You may want to run the test suite with a different shell
than you use to build Git. For instance, you may build with
SHELL_PATH=/bin/sh (because it's faster, or it's what you
expect to exist on systems where the build will be used) but
want to run the test suite with bash (e.g., since that
allows using "-x" reliably across the whole test suite).
There's currently no good way to do this.
You might think that doing two separate make invocations,
like:
make &&
make -C t SHELL_PATH=/bin/bash
would work. And it _almost_ does. The second make will see
our bash SHELL_PATH, and we'll use that to run the
individual test scripts (or tell prove to use it to do so).
So far so good.
But this breaks down when "--tee" or "--verbose-log" is
used. Those options cause the test script to actually
re-exec itself using $SHELL_PATH. But wait, wouldn't our
second make invocation have set SHELL_PATH correctly in the
environment?
Yes, but test-lib.sh sources GIT-BUILD-OPTIONS, which we
built during the first "make". And that overrides the
environment, giving us the original SHELL_PATH again.
Let's introduce a new variable that lets you specify a
specific shell to be run for the test scripts. Note that we
have to touch both the main and t/ Makefiles, since we have
to record it in GIT-BUILD-OPTIONS in one, and use it in the
latter.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-12-08 11:47:22 +01:00
|
|
|
@echo "*** prove ***"; $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)
|
2012-05-02 17:31:52 +02:00
|
|
|
$(MAKE) clean-except-prove-cache
|
2010-10-14 10:53:36 +02:00
|
|
|
|
2005-11-08 10:51:10 +01:00
|
|
|
$(T):
|
t/Makefile: introduce TEST_SHELL_PATH
You may want to run the test suite with a different shell
than you use to build Git. For instance, you may build with
SHELL_PATH=/bin/sh (because it's faster, or it's what you
expect to exist on systems where the build will be used) but
want to run the test suite with bash (e.g., since that
allows using "-x" reliably across the whole test suite).
There's currently no good way to do this.
You might think that doing two separate make invocations,
like:
make &&
make -C t SHELL_PATH=/bin/bash
would work. And it _almost_ does. The second make will see
our bash SHELL_PATH, and we'll use that to run the
individual test scripts (or tell prove to use it to do so).
So far so good.
But this breaks down when "--tee" or "--verbose-log" is
used. Those options cause the test script to actually
re-exec itself using $SHELL_PATH. But wait, wouldn't our
second make invocation have set SHELL_PATH correctly in the
environment?
Yes, but test-lib.sh sources GIT-BUILD-OPTIONS, which we
built during the first "make". And that overrides the
environment, giving us the original SHELL_PATH again.
Let's introduce a new variable that lets you specify a
specific shell to be run for the test scripts. Note that we
have to touch both the main and t/ Makefiles, since we have
to record it in GIT-BUILD-OPTIONS in one, and use it in the
latter.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-12-08 11:47:22 +01:00
|
|
|
@echo "*** $@ ***"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)
|
2005-05-14 07:50:32 +02:00
|
|
|
|
2008-06-08 16:04:35 +02:00
|
|
|
pre-clean:
|
2013-04-26 20:55:52 +02:00
|
|
|
$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'
|
2008-06-08 16:04:35 +02:00
|
|
|
|
2012-05-02 17:31:52 +02:00
|
|
|
clean-except-prove-cache:
|
2013-04-26 20:55:52 +02:00
|
|
|
$(RM) -r 'trash directory'.* '$(TEST_RESULTS_DIRECTORY_SQ)'
|
2010-03-13 21:41:20 +01:00
|
|
|
$(RM) -r valgrind/bin
|
2012-05-02 17:31:52 +02:00
|
|
|
|
|
|
|
|
clean: clean-except-prove-cache
|
2010-07-24 00:58:44 +02:00
|
|
|
$(RM) .prove
|
2005-11-08 10:51:10 +01:00
|
|
|
|
t/Makefile: ensure that paths are valid on platforms we care
Some pathnames that are okay on ext4 and on HFS+ cannot be checked
out on Windows. Tests that want to see operations on such paths on
filesystems that support them must do so behind appropriate test
prerequisites, and must not include them in the source tree (instead
they should create them when they run). Otherwise, the source tree
cannot even be checked out.
Make sure that double-quotes, asterisk, colon, greater/less-than,
question-mark, backslash, tab, vertical-bar, as well as any non-ASCII
characters never appear in the pathnames with a new test-lint-* target
as part of a `make test`. To that end, we call `git ls-files` (ensuring
that the paths are quoted properly), relying on the fact that paths
containing non-ASCII characters are quoted within double-quotes.
In case that the source code does not actually live in a Git
repository (e.g. when extracted from a .zip file), or that the `git`
executable cannot be executed, we simply ignore the error for now; In
that case, our trusty Continuous Integration will be the last line of
defense and catch any problematic file name.
Noticed when a topic wanted to add a pathname with '>' in it. A
check like this will prevent a similar problems from happening in the
future.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-16 17:13:25 +02:00
|
|
|
test-lint: test-lint-duplicates test-lint-executable test-lint-shell-syntax \
|
|
|
|
|
test-lint-filenames
|
2010-12-13 18:22:38 +01:00
|
|
|
|
|
|
|
|
test-lint-duplicates:
|
|
|
|
|
@dups=`echo $(T) | tr ' ' '\n' | sed 's/-.*//' | sort | uniq -d` && \
|
|
|
|
|
test -z "$$dups" || { \
|
|
|
|
|
echo >&2 "duplicate test numbers:" $$dups; exit 1; }
|
|
|
|
|
|
|
|
|
|
test-lint-executable:
|
|
|
|
|
@bad=`for i in $(T); do test -x "$$i" || echo $$i; done` && \
|
|
|
|
|
test -z "$$bad" || { \
|
|
|
|
|
echo >&2 "non-executable tests:" $$bad; exit 1; }
|
|
|
|
|
|
2013-01-03 00:20:19 +01:00
|
|
|
test-lint-shell-syntax:
|
2014-07-09 21:34:12 +02:00
|
|
|
@'$(PERL_PATH_SQ)' check-non-portable-shell.pl $(T) $(THELPERS)
|
2013-01-03 00:20:19 +01:00
|
|
|
|
t/Makefile: ensure that paths are valid on platforms we care
Some pathnames that are okay on ext4 and on HFS+ cannot be checked
out on Windows. Tests that want to see operations on such paths on
filesystems that support them must do so behind appropriate test
prerequisites, and must not include them in the source tree (instead
they should create them when they run). Otherwise, the source tree
cannot even be checked out.
Make sure that double-quotes, asterisk, colon, greater/less-than,
question-mark, backslash, tab, vertical-bar, as well as any non-ASCII
characters never appear in the pathnames with a new test-lint-* target
as part of a `make test`. To that end, we call `git ls-files` (ensuring
that the paths are quoted properly), relying on the fact that paths
containing non-ASCII characters are quoted within double-quotes.
In case that the source code does not actually live in a Git
repository (e.g. when extracted from a .zip file), or that the `git`
executable cannot be executed, we simply ignore the error for now; In
that case, our trusty Continuous Integration will be the last line of
defense and catch any problematic file name.
Noticed when a topic wanted to add a pathname with '>' in it. A
check like this will prevent a similar problems from happening in the
future.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-16 17:13:25 +02:00
|
|
|
test-lint-filenames:
|
|
|
|
|
@# We do *not* pass a glob to ls-files but use grep instead, to catch
|
|
|
|
|
@# non-ASCII characters (which are quoted within double-quotes)
|
|
|
|
|
@bad="$$(git -c core.quotepath=true ls-files 2>/dev/null | \
|
|
|
|
|
grep '["*:<>?\\|]')"; \
|
|
|
|
|
test -z "$$bad" || { \
|
|
|
|
|
echo >&2 "non-portable file name(s): $$bad"; exit 1; }
|
|
|
|
|
|
tests: Clarify dependencies between tests, 'aggregate-results' and 'clean'
The Makefile targets 'aggregate-results' and 'clean' pretended to be
independent. This is not true, of course, since aggregate-results
needs the results _before_ they are removed.
Likewise, the tests should have been run already when the results are
to be aggregated.
However, as it is legitimate to run only a few tests, and then aggregate
just those results, so another target is introduced, that depends on all
tests, then aggregates the results, and only then removes the results.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-08-08 07:59:18 +02:00
|
|
|
aggregate-results-and-cleanup: $(T)
|
|
|
|
|
$(MAKE) aggregate-results
|
|
|
|
|
$(MAKE) clean
|
|
|
|
|
|
2008-06-08 16:04:35 +02:00
|
|
|
aggregate-results:
|
2013-04-26 20:55:52 +02:00
|
|
|
for f in '$(TEST_RESULTS_DIRECTORY_SQ)'/t*-*.counts; do \
|
2010-06-02 02:13:44 +02:00
|
|
|
echo "$$f"; \
|
|
|
|
|
done | '$(SHELL_PATH_SQ)' ./aggregate-results.sh
|
2008-06-08 16:04:35 +02:00
|
|
|
|
2010-09-26 15:02:26 +02:00
|
|
|
gitweb-test:
|
|
|
|
|
$(MAKE) $(TGITWEB)
|
|
|
|
|
|
2009-02-04 00:26:18 +01:00
|
|
|
valgrind:
|
2011-06-17 10:29:57 +02:00
|
|
|
$(MAKE) GIT_TEST_OPTS="$(GIT_TEST_OPTS) --valgrind"
|
2009-02-04 00:26:18 +01:00
|
|
|
|
2012-02-17 11:25:09 +01:00
|
|
|
perf:
|
|
|
|
|
$(MAKE) -C perf/ all
|
|
|
|
|
|
|
|
|
|
.PHONY: pre-clean $(T) aggregate-results clean valgrind perf
|
