1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-05-21 02:16:12 +02:00
git/t/test-lib.sh

1681 lines
40 KiB
Bash
Raw Normal View History

# Test framework for git. See t/README for usage.
#
# Copyright (c) 2005 Junio C Hamano
#
test-lib.sh: Add explicit license detail, with change from GPLv2 to GPLv2+. Dear Junio, this is a resend of relicensing patch for test suite library, which was initially sent by Carl Worth. Since the time you sent me acks for this patch collected by you, I collected 8 additional acks as is documented at https://git.wiki.kernel.org/index.php/Test-lib_reclicensing. There are still three contributors missing: Bert Wesarg, Stephan Beyer and Bryan Donlan. The contributions of first two are clearly not copyrightable. I'm not sure about the copyrightability of Bryan Donlan's contributions (git log -p --author='Bryan Donlan' t/test-lib.sh). Carl told me that in your ack collection process you missed only three acks. So I wonder whether you already did some analysis of which contributions are copyrightable. If so, are the missing acks in the list bellow? Thanks Michal 8<--------8<--------8<-------- This file has had no explicit license information noted in it, but has clearly been created and modified according to the terms of GPLv2 as with the rest of the git code base. The purpose of relicensing is to allow other GPLv3+ projects (in particular, the notmuch project: http://notmuchmail.org) to use this same test-suite structure and to contribute changes back as well. Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz> Acked-by: Alex Riesen <raa.lkml@gmail.com> Acked-by: Brandon Casey <drafnel@gmail.com> Acked-by: Clemens Buchacher <drizzd@aon.at> Acked-by: David Reiss <dreiss@facebook.com> Acked-by: Emil Sit <sit@emilsit.net> Acked-by: Eric Wong <normalperson@yhbt.net> Acked-by: Fredrik Kuivinen <frekui@gmail.com> Acked-by: Gerrit Pape <pape@smarden.org> Acked-by: Christian Couder <chriscool@tuxfamily.org> Acked-by: Jakub Narebski <jnareb@gmail.com> Acked-by: Jeff King <peff@peff.net> Acked-by: Johan Herland <johan@herland.net> Acked-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Acked-by: Johannes Sixt <j6t@kdbg.org> Acked-by: Jonathan Nieder <jrnieder@gmail.com> Acked-by: Josh Triplett <josh@joshtriplett.org> Acked-by: Junio C Hamano <gitster@pobox.com> Acked-by: Lea Wiemann <lewiemann@gmail.com> Acked-by: Markus Heidelberg <markus.heidelberg@web.de> Acked-by: Martin Waitz <tali@admingilde.org> Acked-by: Matthew Ogilvie <mmogilvi_git@miniinfo.net> Acked-by: Matthias Lederhofer <matled@gmx.net> Acked-by: Michael J Gruber <git@drmicha.warpmail.net> Acked-by: Michele Ballabio <barra_cuda@katamail.com> Acked-by: Miklos Vajna <vmiklos@frugalware.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Acked-by: Pavel Roskin <proski@gnu.org> Acked-by: Petr Baudis <pasky@ucw.cz> Acked-by: Pierre Habouzit <madcoder@debian.org> Acked-by: Robin Rosenberg <robin.rosenberg@dewire.com> Acked-by: Shawn O. Pearce <spearce@spearce.org> Acked-by: Stephen Boyd <bebarino@gmail.com> Acked-by: Sverre Rabbelier <srabbelier@gmail.com> Acked-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-16 15:53:59 +02:00
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see http://www.gnu.org/licenses/ .
# Test the binaries we have just built. The tests are kept in
# t/ subdirectory and are run in 'trash directory' subdirectory.
if test -z "$TEST_DIRECTORY"
then
# We allow tests to override this, in case they want to run tests
# outside of t/, e.g. for running tests on the test library
# itself.
TEST_DIRECTORY=$(pwd)
else
# ensure that TEST_DIRECTORY is an absolute path so that it
# is valid even if the current working directory is changed
TEST_DIRECTORY=$(cd "$TEST_DIRECTORY" && pwd) || exit 1
fi
if test -z "$TEST_OUTPUT_DIRECTORY"
then
# Similarly, override this to store the test-results subdir
# elsewhere
TEST_OUTPUT_DIRECTORY=$TEST_DIRECTORY
fi
GIT_BUILD_DIR="$TEST_DIRECTORY"/..
test-lib: set ASAN_OPTIONS variable before we run git We turn off ASan's leak detection by default in the test suite because it's too noisy. But we don't do so until part-way through test-lib. This is before we've run any tests, but after we do our initial "./git" to see if the binary has even been built. When built with clang, this seems to work fine. However, using "gcc -fsanitize=address", the leak checker seems to complain more aggressively: $ ./git ... ==5352==ERROR: LeakSanitizer: detected memory leaks Direct leak of 2 byte(s) in 1 object(s) allocated from: #0 0x7f120e7afcf8 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1cf8) #1 0x559fc2a3ce41 in do_xmalloc /home/peff/compile/git/wrapper.c:60 #2 0x559fc2a3cf1a in do_xmallocz /home/peff/compile/git/wrapper.c:100 #3 0x559fc2a3d0ad in xmallocz /home/peff/compile/git/wrapper.c:108 #4 0x559fc2a3d0ad in xmemdupz /home/peff/compile/git/wrapper.c:124 #5 0x559fc2a3d0ad in xstrndup /home/peff/compile/git/wrapper.c:130 #6 0x559fc274535a in main /home/peff/compile/git/common-main.c:39 #7 0x7f120dabd2b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0) This is a leak in the sense that we never free it, but it's in a global that is meant to last the whole program. So it's not really interesting or in need of fixing. And at any rate, mentioning leaks outside of the test_expect blocks is certainly unwelcome, as it pollutes stderr. Let's bump the setting of ASAN_OPTIONS higher in test-lib.sh to catch our initial "can we even run git?" test. While we're at it, we can add a comment to make it a bit less inscrutable. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-07-10 15:24:35 +02:00
# If we were built with ASAN, it may complain about leaks
# of program-lifetime variables. Disable it by default to lower
# the noise level. This needs to happen at the start of the script,
# before we even do our "did we build git yet" check (since we don't
# want that one to complain to stderr).
: ${ASAN_OPTIONS=detect_leaks=0:abort_on_error=1}
test-lib: set ASAN_OPTIONS variable before we run git We turn off ASan's leak detection by default in the test suite because it's too noisy. But we don't do so until part-way through test-lib. This is before we've run any tests, but after we do our initial "./git" to see if the binary has even been built. When built with clang, this seems to work fine. However, using "gcc -fsanitize=address", the leak checker seems to complain more aggressively: $ ./git ... ==5352==ERROR: LeakSanitizer: detected memory leaks Direct leak of 2 byte(s) in 1 object(s) allocated from: #0 0x7f120e7afcf8 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1cf8) #1 0x559fc2a3ce41 in do_xmalloc /home/peff/compile/git/wrapper.c:60 #2 0x559fc2a3cf1a in do_xmallocz /home/peff/compile/git/wrapper.c:100 #3 0x559fc2a3d0ad in xmallocz /home/peff/compile/git/wrapper.c:108 #4 0x559fc2a3d0ad in xmemdupz /home/peff/compile/git/wrapper.c:124 #5 0x559fc2a3d0ad in xstrndup /home/peff/compile/git/wrapper.c:130 #6 0x559fc274535a in main /home/peff/compile/git/common-main.c:39 #7 0x7f120dabd2b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0) This is a leak in the sense that we never free it, but it's in a global that is meant to last the whole program. So it's not really interesting or in need of fixing. And at any rate, mentioning leaks outside of the test_expect blocks is certainly unwelcome, as it pollutes stderr. Let's bump the setting of ASAN_OPTIONS higher in test-lib.sh to catch our initial "can we even run git?" test. While we're at it, we can add a comment to make it a bit less inscrutable. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-07-10 15:24:35 +02:00
export ASAN_OPTIONS
# If LSAN is in effect we _do_ want leak checking, but we still
# want to abort so that we notice the problems.
: ${LSAN_OPTIONS=abort_on_error=1}
export LSAN_OPTIONS
tests: explicitly use `git.exe` on Windows On Windows, when we refer to `/an/absolute/path/to/git`, it magically resolves `git.exe` at that location. Except if something of the name `git` exists next to that `git.exe`. So if we call `$BUILD_DIR/git`, it will find `$BUILD_DIR/git.exe` *only* if there is not, say, a directory called `$BUILD_DIR/git`. Such a directory, however, exists in Git for Windows when building with Visual Studio (our Visual Studio project generator defaults to putting the build files into a directory whose name is the base name of the corresponding `.exe`). In the bin-wrappers/* scripts, we already take pains to use `git.exe` rather than `git`, as this could pick up the wrong thing on Windows (i.e. if there exists a `git` file or directory in the build directory). Now we do the same in the tests' start-up code. This also helps when testing an installed Git, as there might be even more likely some stray file or directory in the way. Note: the only way we can record whether the `.exe` suffix is by writing it to the `GIT-BUILD-OPTIONS` file and sourcing it at the beginning of `t/test-lib.sh`. This is not a requirement introduced by this patch, but we move the call to be able to use the `$X` variable that holds the file extension, if any. Note also: the many, many calls to `git this` and `git that` are unaffected, as the regular PATH search will find the `.exe` files on Windows (and not be confused by a directory of the name `git` that is in one of the directories listed in the `PATH` variable), while `/path/to/git` would not, per se, know that it is looking for an executable and happily prefer such a directory. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-14 17:32:11 +01:00
if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
then
echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'
exit 1
fi
. "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
export PERL_PATH SHELL_PATH
# Disallow the use of abbreviated options in the test suite by default
if test -z "${GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS}"
then
GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS=true
export GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS
fi
################################################################
# It appears that people try to run tests without building...
tests: explicitly use `git.exe` on Windows On Windows, when we refer to `/an/absolute/path/to/git`, it magically resolves `git.exe` at that location. Except if something of the name `git` exists next to that `git.exe`. So if we call `$BUILD_DIR/git`, it will find `$BUILD_DIR/git.exe` *only* if there is not, say, a directory called `$BUILD_DIR/git`. Such a directory, however, exists in Git for Windows when building with Visual Studio (our Visual Studio project generator defaults to putting the build files into a directory whose name is the base name of the corresponding `.exe`). In the bin-wrappers/* scripts, we already take pains to use `git.exe` rather than `git`, as this could pick up the wrong thing on Windows (i.e. if there exists a `git` file or directory in the build directory). Now we do the same in the tests' start-up code. This also helps when testing an installed Git, as there might be even more likely some stray file or directory in the way. Note: the only way we can record whether the `.exe` suffix is by writing it to the `GIT-BUILD-OPTIONS` file and sourcing it at the beginning of `t/test-lib.sh`. This is not a requirement introduced by this patch, but we move the call to be able to use the `$X` variable that holds the file extension, if any. Note also: the many, many calls to `git this` and `git that` are unaffected, as the regular PATH search will find the `.exe` files on Windows (and not be confused by a directory of the name `git` that is in one of the directories listed in the `PATH` variable), while `/path/to/git` would not, per se, know that it is looking for an executable and happily prefer such a directory. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-14 17:32:11 +01:00
"${GIT_TEST_INSTALLED:-$GIT_BUILD_DIR}/git$X" >/dev/null
if test $? != 1
then
if test -n "$GIT_TEST_INSTALLED"
then
echo >&2 "error: there is no working Git at '$GIT_TEST_INSTALLED'"
else
echo >&2 'error: you do not seem to have built git yet.'
fi
exit 1
fi
store_arg_to=
opt_required_arg=
# $1: option string
# $2: name of the var where the arg will be stored
mark_option_requires_arg () {
if test -n "$opt_required_arg"
then
echo "error: options that require args cannot be bundled" \
"together: '$opt_required_arg' and '$1'" >&2
exit 1
fi
opt_required_arg=$1
store_arg_to=$2
}
parse_option () {
local opt="$1"
case "$opt" in
-d|--d|--de|--deb|--debu|--debug)
debug=t ;;
-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)
immediate=t ;;
-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)
GIT_TEST_LONG=t; export GIT_TEST_LONG ;;
-r)
mark_option_requires_arg "$opt" run_list
;;
--run=*)
run_list=${opt#--*=} ;;
-h|--h|--he|--hel|--help)
help=t ;;
-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)
verbose=t ;;
--verbose-only=*)
verbose_only=${opt#--*=}
;;
-q|--q|--qu|--qui|--quie|--quiet)
# Ignore --quiet under a TAP::Harness. Saying how many tests
# passed without the ok/not ok details is always an error.
test -z "$HARNESS_ACTIVE" && quiet=t ;;
--with-dashes)
with_dashes=t ;;
--no-bin-wrappers)
no_bin_wrappers=t ;;
--no-color)
color= ;;
--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)
valgrind=memcheck
tee=t
;;
--valgrind=*)
valgrind=${opt#--*=}
tee=t
;;
--valgrind-only=*)
valgrind_only=${opt#--*=}
tee=t
;;
--tee)
tee=t ;;
--root=*)
root=${opt#--*=} ;;
--chain-lint)
GIT_TEST_CHAIN_LINT=1 ;;
--no-chain-lint)
GIT_TEST_CHAIN_LINT=0 ;;
-x)
trace=t ;;
-V|--verbose-log)
verbose_log=t
tee=t
;;
--write-junit-xml)
write_junit_xml=t
;;
test-lib: add the '--stress' option to run a test repeatedly under load Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
--stress)
stress=t ;;
--stress=*)
echo "error: --stress does not accept an argument: '$opt'" >&2
echo "did you mean --stress-jobs=${opt#*=} or --stress-limit=${opt#*=}?" >&2
exit 1
;;
--stress-jobs=*)
stress=t;
stress_jobs=${opt#--*=}
case "$stress_jobs" in
*[!0-9]*|0*|"")
echo "error: --stress-jobs=<N> requires the number of jobs to run" >&2
test-lib: add the '--stress' option to run a test repeatedly under load Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
exit 1
;;
*) # Good.
;;
esac
;;
--stress-limit=*)
stress=t;
stress_limit=${opt#--*=}
case "$stress_limit" in
*[!0-9]*|0*|"")
echo "error: --stress-limit=<N> requires the number of repetitions" >&2
exit 1
;;
*) # Good.
;;
esac
;;
*)
echo "error: unknown test option '$opt'" >&2; exit 1 ;;
esac
}
# Parse options while taking care to leave $@ intact, so we will still
# have all the original command line options when executing the test
# script again for '--tee' and '--verbose-log' later.
for opt
do
if test -n "$store_arg_to"
then
eval $store_arg_to=\$opt
store_arg_to=
opt_required_arg=
continue
fi
case "$opt" in
--*|-?)
parse_option "$opt" ;;
-?*)
# bundled short options must be fed separately to parse_option
opt=${opt#-}
while test -n "$opt"
do
extra=${opt#?}
this=${opt%$extra}
opt=$extra
parse_option "-$this"
done
;;
*)
echo "error: unknown test option '$opt'" >&2; exit 1 ;;
esac
done
if test -n "$store_arg_to"
then
echo "error: $opt_required_arg requires an argument" >&2
exit 1
fi
if test -n "$valgrind_only"
then
test -z "$valgrind" && valgrind=memcheck
test -z "$verbose" && verbose_only="$valgrind_only"
elif test -n "$valgrind"
then
test -z "$verbose_log" && verbose=t
fi
test-lib: add the '--stress' option to run a test repeatedly under load Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
if test -n "$stress"
then
verbose=t
trace=t
immediate=t
fi
TEST_STRESS_JOB_SFX="${GIT_TEST_STRESS_JOB_NR:+.stress-$GIT_TEST_STRESS_JOB_NR}"
TEST_NAME="$(basename "$0" .sh)"
tests: show the test name and number at the start of verbose output The verbose output of every test looks something like this: expecting success: echo content >file && git add file && git commit -m "add file" [master (root-commit) d1fbfbd] add file Author: A U Thor <author@example.com> 1 file changed, 1 insertion(+) create mode 100644 file ok 1 - commit works i.e. first an "expecting success" (or "checking known breakage") line followed by the commands to be executed, then the output of those comamnds, and finally an "ok"/"not ok" line containing the test name. Note that the test's name is only shown at the very end. With '-x' tracing enabled and/or in longer tests the verbose output might be several screenfulls long, making it harder than necessary to find where the output of the test with a given name starts (especially when the outputs to different file descriptors are racing, and the "expecting success"/command block arrives earlier than the "ok" line of the previous test). Print the test name at the start of the test's verbose output, i.e. at the end of the "expecting success" and "checking known breakage" lines, to make the start of a particular test a bit easier to recognize. Also print the test script and test case numbers, to help those poor souls who regularly have to scan through the combined verbose output of several test scripts. So the dummy test above would start like this: expecting success of 9999.1 'commit works': echo content >file && [...] Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-08-05 23:04:47 +02:00
TEST_NUMBER="${TEST_NAME%%-*}"
TEST_NUMBER="${TEST_NUMBER#t}"
TEST_RESULTS_DIR="$TEST_OUTPUT_DIRECTORY/test-results"
test-lib: add the '--stress' option to run a test repeatedly under load Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
TEST_RESULTS_BASE="$TEST_RESULTS_DIR/$TEST_NAME$TEST_STRESS_JOB_SFX"
TRASH_DIRECTORY="trash directory.$TEST_NAME$TEST_STRESS_JOB_SFX"
test -n "$root" && TRASH_DIRECTORY="$root/$TRASH_DIRECTORY"
case "$TRASH_DIRECTORY" in
/*) ;; # absolute path is good
*) TRASH_DIRECTORY="$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY" ;;
esac
test-lib: add the '--stress' option to run a test repeatedly under load Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
# If --stress was passed, run this test repeatedly in several parallel loops.
if test "$GIT_TEST_STRESS_STARTED" = "done"
then
: # Don't stress test again.
elif test -n "$stress"
then
if test -n "$stress_jobs"
test-lib: add the '--stress' option to run a test repeatedly under load Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
then
job_count=$stress_jobs
test-lib: add the '--stress' option to run a test repeatedly under load Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
elif test -n "$GIT_TEST_STRESS_LOAD"
then
job_count="$GIT_TEST_STRESS_LOAD"
elif job_count=$(getconf _NPROCESSORS_ONLN 2>/dev/null) &&
test -n "$job_count"
then
job_count=$((2 * $job_count))
else
job_count=8
fi
mkdir -p "$TEST_RESULTS_DIR"
stressfail="$TEST_RESULTS_BASE.stress-failed"
rm -f "$stressfail"
stress_exit=0
trap '
kill $job_pids 2>/dev/null
wait
stress_exit=1
' TERM INT HUP
job_pids=
job_nr=0
while test $job_nr -lt "$job_count"
do
(
GIT_TEST_STRESS_STARTED=done
GIT_TEST_STRESS_JOB_NR=$job_nr
export GIT_TEST_STRESS_STARTED GIT_TEST_STRESS_JOB_NR
trap '
kill $test_pid 2>/dev/null
wait
exit 1
' TERM INT
cnt=1
while ! test -e "$stressfail" &&
{ test -z "$stress_limit" ||
test $cnt -le $stress_limit ; }
test-lib: add the '--stress' option to run a test repeatedly under load Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
do
$TEST_SHELL_PATH "$0" "$@" >"$TEST_RESULTS_BASE.stress-$job_nr.out" 2>&1 &
test_pid=$!
if wait $test_pid
then
printf "OK %2d.%d\n" $GIT_TEST_STRESS_JOB_NR $cnt
else
echo $GIT_TEST_STRESS_JOB_NR >>"$stressfail"
printf "FAIL %2d.%d\n" $GIT_TEST_STRESS_JOB_NR $cnt
fi
cnt=$(($cnt + 1))
done
) &
job_pids="$job_pids $!"
job_nr=$(($job_nr + 1))
done
wait
if test -f "$stressfail"
then
stress_exit=1
test-lib: add the '--stress' option to run a test repeatedly under load Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
echo "Log(s) of failed test run(s):"
for failed_job_nr in $(sort -n "$stressfail")
do
echo "Contents of '$TEST_RESULTS_BASE.stress-$failed_job_nr.out':"
cat "$TEST_RESULTS_BASE.stress-$failed_job_nr.out"
done
rm -rf "$TRASH_DIRECTORY.stress-failed"
# Move the last one.
mv "$TRASH_DIRECTORY.stress-$failed_job_nr" "$TRASH_DIRECTORY.stress-failed"
fi
exit $stress_exit
fi
# if --tee was passed, write the output not only to the terminal, but
# additionally to the file test-results/$BASENAME.out, too.
if test "$GIT_TEST_TEE_STARTED" = "done"
then
: # do not redirect again
elif test -n "$tee"
then
mkdir -p "$TEST_RESULTS_DIR"
test-lib: add --verbose-log option The "--verbose" option redirects output from arbitrary test commands to stdout. This is useful for examining the output manually, like: ./t5547-push-quarantine.sh -v | less But it also means that the output is intermingled with the TAP directives, which can confuse a TAP parser like "prove". This has always been a potential problem, but became an issue recently when one test happened to output the word "ok" on a line by itself, which prove interprets as a test success: $ prove t5547-push-quarantine.sh :: -v t5547-push-quarantine.sh .. 1/? To dest.git * [new branch] HEAD -> master To dest.git ! [remote rejected] reject -> reject (pre-receive hook declined) error: failed to push some refs to 'dest.git' fatal: git cat-file d08c8eba97f4e683ece08654c7c8d2ba0c03b129: bad file t5547-push-quarantine.sh .. Failed -1/4 subtests Test Summary Report ------------------- t5547-push-quarantine.sh (Wstat: 0 Tests: 5 Failed: 0) Parse errors: Tests out of sequence. Found (2) but expected (3) Tests out of sequence. Found (3) but expected (4) Tests out of sequence. Found (4) but expected (5) Bad plan. You planned 4 tests but ran 5. Files=1, Tests=5, 0 wallclock secs ( 0.01 usr + 0.01 sys = 0.02 CPU) Result: FAIL One answer is "if it hurts, don't do it", but that's not quite the whole story. The Travis tests use "--verbose --tee" so that they can get the benefit of prove's parallel options, along with a verbose log in case there is a failure. We just need the verbose output to go to the log, but keep stdout clean. Getting this right turns out to be surprisingly difficult. Here's the progression of alternatives I considered: 1. Add an option to write verbose output to stderr. This is hard to capture, though, because we want each test to have its own log (because they're all run in parallel and the jumbled output would be useless). 2. Add an option to write verbose output to a file in test-results. This works, but the log is missing all of the non-verbose output, which gives context. 3. Like (2), but teach say_color() to additionally output to the log. This mostly works, but misses any output that happens outside of the say() functions (which isn't a lot, but is a potential maintenance headache). 4. Like (2), but make the log file the same as the "--tee" file. That almost works, but now we have two processes opening the same file. That gives us two separate descriptors, each with their own idea of the current position. They'll each start writing at offset 0, and overwrite each other's data. 5. Like (4), but in each case open the file for appending. That atomically positions each write at the end of the file. It's possible we may still get sheared writes between the two processes, but this is already the case when writing to stdout. It's not a problem in practice because the test harness generally waits for snippets to finish before writing the TAP output. We can ignore buffering issues with tee, because POSIX mandates that it does not buffer. Likewise, POSIX specifies "tee -a", so it should be available everywhere. This patch implements option (5), which seems to work well in practice. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-21 12:48:00 +02:00
# Make this filename available to the sub-process in case it is using
# --verbose-log.
GIT_TEST_TEE_OUTPUT_FILE=$TEST_RESULTS_BASE.out
test-lib: add --verbose-log option The "--verbose" option redirects output from arbitrary test commands to stdout. This is useful for examining the output manually, like: ./t5547-push-quarantine.sh -v | less But it also means that the output is intermingled with the TAP directives, which can confuse a TAP parser like "prove". This has always been a potential problem, but became an issue recently when one test happened to output the word "ok" on a line by itself, which prove interprets as a test success: $ prove t5547-push-quarantine.sh :: -v t5547-push-quarantine.sh .. 1/? To dest.git * [new branch] HEAD -> master To dest.git ! [remote rejected] reject -> reject (pre-receive hook declined) error: failed to push some refs to 'dest.git' fatal: git cat-file d08c8eba97f4e683ece08654c7c8d2ba0c03b129: bad file t5547-push-quarantine.sh .. Failed -1/4 subtests Test Summary Report ------------------- t5547-push-quarantine.sh (Wstat: 0 Tests: 5 Failed: 0) Parse errors: Tests out of sequence. Found (2) but expected (3) Tests out of sequence. Found (3) but expected (4) Tests out of sequence. Found (4) but expected (5) Bad plan. You planned 4 tests but ran 5. Files=1, Tests=5, 0 wallclock secs ( 0.01 usr + 0.01 sys = 0.02 CPU) Result: FAIL One answer is "if it hurts, don't do it", but that's not quite the whole story. The Travis tests use "--verbose --tee" so that they can get the benefit of prove's parallel options, along with a verbose log in case there is a failure. We just need the verbose output to go to the log, but keep stdout clean. Getting this right turns out to be surprisingly difficult. Here's the progression of alternatives I considered: 1. Add an option to write verbose output to stderr. This is hard to capture, though, because we want each test to have its own log (because they're all run in parallel and the jumbled output would be useless). 2. Add an option to write verbose output to a file in test-results. This works, but the log is missing all of the non-verbose output, which gives context. 3. Like (2), but teach say_color() to additionally output to the log. This mostly works, but misses any output that happens outside of the say() functions (which isn't a lot, but is a potential maintenance headache). 4. Like (2), but make the log file the same as the "--tee" file. That almost works, but now we have two processes opening the same file. That gives us two separate descriptors, each with their own idea of the current position. They'll each start writing at offset 0, and overwrite each other's data. 5. Like (4), but in each case open the file for appending. That atomically positions each write at the end of the file. It's possible we may still get sheared writes between the two processes, but this is already the case when writing to stdout. It's not a problem in practice because the test harness generally waits for snippets to finish before writing the TAP output. We can ignore buffering issues with tee, because POSIX mandates that it does not buffer. Likewise, POSIX specifies "tee -a", so it should be available everywhere. This patch implements option (5), which seems to work well in practice. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-21 12:48:00 +02:00
export GIT_TEST_TEE_OUTPUT_FILE
# Truncate before calling "tee -a" to get rid of the results
# from any previous runs.
>"$GIT_TEST_TEE_OUTPUT_FILE"
(GIT_TEST_TEE_STARTED=done ${TEST_SHELL_PATH} "$0" "$@" 2>&1;
echo $? >"$TEST_RESULTS_BASE.exit") | tee -a "$GIT_TEST_TEE_OUTPUT_FILE"
test "$(cat "$TEST_RESULTS_BASE.exit")" = 0
exit
fi
if test -n "$trace" && test -n "$test_untraceable"
then
# '-x' tracing requested, but this test script can't be reliably
# traced, unless it is run with a Bash version supporting
# BASH_XTRACEFD (introduced in Bash v4.1).
#
# Perform this version check _after_ the test script was
# potentially re-executed with $TEST_SHELL_PATH for '--tee' or
# '--verbose-log', so the right shell is checked and the
# warning is issued only once.
if test -n "$BASH_VERSION" && eval '
test ${BASH_VERSINFO[0]} -gt 4 || {
test ${BASH_VERSINFO[0]} -eq 4 &&
test ${BASH_VERSINFO[1]} -ge 1
}
'
then
: Executed by a Bash version supporting BASH_XTRACEFD. Good.
else
echo >&2 "warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD"
trace=
fi
fi
if test -n "$trace" && test -z "$verbose_log"
then
verbose=t
fi
# For repeatability, reset the environment to known value.
# TERM is sanitized below, after saving color control sequences.
LANG=C
LC_ALL=C
PAGER=cat
TZ=UTC
export LANG LC_ALL PAGER TZ
EDITOR=:
i18n: make GETTEXT_POISON a runtime option Change the GETTEXT_POISON compile-time + runtime GIT_GETTEXT_POISON test parameter to only be a GIT_TEST_GETTEXT_POISON=<non-empty?> runtime parameter, to be consistent with other parameters documented in "Running tests with special setups" in t/README. When I added GETTEXT_POISON in bb946bba76 ("i18n: add GETTEXT_POISON to simulate unfriendly translator", 2011-02-22) I was concerned with ensuring that the _() function would get constant folded if NO_GETTEXT was defined, and likewise that GETTEXT_POISON would be compiled out unless it was defined. But as the benchmark in my [1] shows doing a one-off runtime getenv("GIT_TEST_[...]") is trivial, and since GETTEXT_POISON was originally added the GIT_TEST_* env variables have become the common idiom for turning on special test setups. So change GETTEXT_POISON to work the same way. Now the GETTEXT_POISON=YesPlease compile-time option is gone, and running the tests with GIT_TEST_GETTEXT_POISON=[YesPlease|] can be toggled on/off without recompiling. This allows for conditionally amending tests to test with/without poison, similar to what 859fdc0c3c ("commit-graph: define GIT_TEST_COMMIT_GRAPH", 2018-08-29) did for GIT_TEST_COMMIT_GRAPH. Do some of that, now we e.g. always run the t0205-gettext-poison.sh test. I did enough there to remove the GETTEXT_POISON prerequisite, but its inverse C_LOCALE_OUTPUT is still around, and surely some tests using it can be converted to e.g. always set GIT_TEST_GETTEXT_POISON=. Notes on the implementation: * We still compile a dedicated GETTEXT_POISON build in Travis CI. Perhaps this should be revisited and integrated into the "linux-gcc" build, see ae59a4e44f ("travis: run tests with GIT_TEST_SPLIT_INDEX", 2018-01-07) for prior art in that area. Then again maybe not, see [2]. * We now skip a test in t0000-basic.sh under GIT_TEST_GETTEXT_POISON=YesPlease that wasn't skipped before. This test relies on C locale output, but due to an edge case in how the previous implementation of GETTEXT_POISON worked (reading it from GIT-BUILD-OPTIONS) wasn't enabling poison correctly. Now it does, and needs to be skipped. * The getenv() function is not reentrant, so out of paranoia about code of the form: printf(_("%s"), getenv("some-env")); call use_gettext_poison() in our early setup in git_setup_gettext() so we populate the "poison_requested" variable in a codepath that's won't suffer from that race condition. * We error out in the Makefile if you're still saying GETTEXT_POISON=YesPlease to prompt users to change their invocation. * We should not print out poisoned messages during the test initialization itself to keep it more readable, so the test library hides the variable if set in $GIT_TEST_GETTEXT_POISON_ORIG during setup. See [3]. See also [4] for more on the motivation behind this patch, and the history of the GETTEXT_POISON facility. 1. https://public-inbox.org/git/871s8gd32p.fsf@evledraar.gmail.com/ 2. https://public-inbox.org/git/20181102163725.GY30222@szeder.dev/ 3. https://public-inbox.org/git/20181022202241.18629-2-szeder.dev@gmail.com/ 4. https://public-inbox.org/git/878t2pd6yu.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-08 22:15:29 +01:00
# A call to "unset" with no arguments causes at least Solaris 10
# /usr/xpg4/bin/sh and /bin/ksh to bail out. So keep the unsets
# deriving from the command substitution clustered with the other
# ones.
unset VISUAL EMAIL LANGUAGE COLUMNS $("$PERL_PATH" -e '
my @env = keys %ENV;
my $ok = join("|", qw(
TRACE
DEBUG
TEST
.*_TEST
PROVE
VALGRIND
UNZIP
PERF_
CURL_VERBOSE
TRACE_CURL
));
my @vars = grep(/^GIT_/ && !/^GIT_($ok)/o, @env);
print join("\n", @vars);
')
unset XDG_CACHE_HOME
unset XDG_CONFIG_HOME
unset GITPERLLIB
TEST_AUTHOR_LOCALNAME=author
TEST_AUTHOR_DOMAIN=example.com
GIT_AUTHOR_EMAIL=${TEST_AUTHOR_LOCALNAME}@${TEST_AUTHOR_DOMAIN}
GIT_AUTHOR_NAME='A U Thor'
test-lib: set deterministic default author/committer date We always set the name and email for committer and author idents to make the test suite more deterministic, but not timestamps. Many scripts use test_tick to get consistent and sensibly incrementing timestamps as they create commits. But other scripts don't particularly care about the timestamp, and are happy to use whatever the current system time is. This non-determinism can be annoying: - when debugging a test, comparing results between two runs can be difficult, because the commit ids change - this can sometimes cause tests to be racy. E.g., traversal order depends on timestamp order. Even in a well-ordered set of commands, because our timestamp granularity is one second, two commits might sometimes have the same timestamp and sometimes differ. Let's set a default timestamp for all scripts to use. Any that use test_tick already will be unaffected (because their first test_tick call will overwrite our default), but it will make things a bit more deterministic for those that don't. We should be able to choose any time we want here. I picked this one because: - it differs from the initial test_tick default, which may make it easier to distinguish when debugging tests. I picked "April 1st 13:14:15" in the hope that it might stand out. - it's slightly before the test_tick default. Some tests create some commits before the first call to test_tick, so using an older timestamps for those makes sense chronologically. Note that this isn't how things currently work (where system times are usually more recent than test_tick), but that also allows us to flush out a few hidden timestamp dependencies (like the one recently fixed in t5539). - we could likewise pick any timezone we want. Choosing +0000 would have required fixing up fewer tests, but we're more likely to turn up interesting cases by not matching $TZ exactly. And since test_tick already checks "-0700", let's try something in the "+" zone range for variety. It's possible that the non-deterministic times could help flush out bugs (e.g., if something broke when the clock flipped over to 2021, our test suite would let us know). But historically that hasn't been the case; all time-dependent outcomes we've seen turned out to be accidentally flaky tests (which we fixed by using test_tick). If we do want to cover handling the current time, we should dedicate one script to doing so, and have it unset GIT_COMMITTER_DATE explicitly. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-09 22:42:04 +02:00
GIT_AUTHOR_DATE='1112354055 +0200'
TEST_COMMITTER_LOCALNAME=committer
TEST_COMMITTER_DOMAIN=example.com
GIT_COMMITTER_EMAIL=${TEST_COMMITTER_LOCALNAME}@${TEST_COMMITTER_DOMAIN}
GIT_COMMITTER_NAME='C O Mitter'
test-lib: set deterministic default author/committer date We always set the name and email for committer and author idents to make the test suite more deterministic, but not timestamps. Many scripts use test_tick to get consistent and sensibly incrementing timestamps as they create commits. But other scripts don't particularly care about the timestamp, and are happy to use whatever the current system time is. This non-determinism can be annoying: - when debugging a test, comparing results between two runs can be difficult, because the commit ids change - this can sometimes cause tests to be racy. E.g., traversal order depends on timestamp order. Even in a well-ordered set of commands, because our timestamp granularity is one second, two commits might sometimes have the same timestamp and sometimes differ. Let's set a default timestamp for all scripts to use. Any that use test_tick already will be unaffected (because their first test_tick call will overwrite our default), but it will make things a bit more deterministic for those that don't. We should be able to choose any time we want here. I picked this one because: - it differs from the initial test_tick default, which may make it easier to distinguish when debugging tests. I picked "April 1st 13:14:15" in the hope that it might stand out. - it's slightly before the test_tick default. Some tests create some commits before the first call to test_tick, so using an older timestamps for those makes sense chronologically. Note that this isn't how things currently work (where system times are usually more recent than test_tick), but that also allows us to flush out a few hidden timestamp dependencies (like the one recently fixed in t5539). - we could likewise pick any timezone we want. Choosing +0000 would have required fixing up fewer tests, but we're more likely to turn up interesting cases by not matching $TZ exactly. And since test_tick already checks "-0700", let's try something in the "+" zone range for variety. It's possible that the non-deterministic times could help flush out bugs (e.g., if something broke when the clock flipped over to 2021, our test suite would let us know). But historically that hasn't been the case; all time-dependent outcomes we've seen turned out to be accidentally flaky tests (which we fixed by using test_tick). If we do want to cover handling the current time, we should dedicate one script to doing so, and have it unset GIT_COMMITTER_DATE explicitly. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-09 22:42:04 +02:00
GIT_COMMITTER_DATE='1112354055 +0200'
GIT_MERGE_VERBOSITY=5
merge: use editor by default in interactive sessions Traditionally, a cleanly resolved merge was committed by "git merge" using the auto-generated merge commit log message without invoking the editor. After 5 years of use in the field, it turns out that people perform too many unjustified merges of the upstream history into their topic branches. These merges are not just useless, but they are often not explained well, and making the end result unreadable when it gets time for merging their history back to their upstream. Earlier we added the "--edit" option to the command, so that people can edit the log message to explain and justify their merge commits. Let's take it one step further and spawn the editor by default when we are in an interactive session (i.e. the standard input and the standard output are pointing at the same tty device). There may be existing scripts that leave the standard input and the standard output of the "git merge" connected to whatever environment the scripts were started, and such invocation might trigger the above "interactive session" heuristics. GIT_MERGE_AUTOEDIT environment variable can be set to "no" at the beginning of such scripts to use the historical behaviour while the script runs. Note that this backward compatibility is meant only for scripts, and we deliberately do *not* support "merge.edit = yes/no/auto" configuration option to allow people to keep the historical behaviour. Suggested-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-01-11 07:44:45 +01:00
GIT_MERGE_AUTOEDIT=no
export GIT_MERGE_VERBOSITY GIT_MERGE_AUTOEDIT
export GIT_AUTHOR_EMAIL GIT_AUTHOR_NAME
export GIT_COMMITTER_EMAIL GIT_COMMITTER_NAME
test-lib: set deterministic default author/committer date We always set the name and email for committer and author idents to make the test suite more deterministic, but not timestamps. Many scripts use test_tick to get consistent and sensibly incrementing timestamps as they create commits. But other scripts don't particularly care about the timestamp, and are happy to use whatever the current system time is. This non-determinism can be annoying: - when debugging a test, comparing results between two runs can be difficult, because the commit ids change - this can sometimes cause tests to be racy. E.g., traversal order depends on timestamp order. Even in a well-ordered set of commands, because our timestamp granularity is one second, two commits might sometimes have the same timestamp and sometimes differ. Let's set a default timestamp for all scripts to use. Any that use test_tick already will be unaffected (because their first test_tick call will overwrite our default), but it will make things a bit more deterministic for those that don't. We should be able to choose any time we want here. I picked this one because: - it differs from the initial test_tick default, which may make it easier to distinguish when debugging tests. I picked "April 1st 13:14:15" in the hope that it might stand out. - it's slightly before the test_tick default. Some tests create some commits before the first call to test_tick, so using an older timestamps for those makes sense chronologically. Note that this isn't how things currently work (where system times are usually more recent than test_tick), but that also allows us to flush out a few hidden timestamp dependencies (like the one recently fixed in t5539). - we could likewise pick any timezone we want. Choosing +0000 would have required fixing up fewer tests, but we're more likely to turn up interesting cases by not matching $TZ exactly. And since test_tick already checks "-0700", let's try something in the "+" zone range for variety. It's possible that the non-deterministic times could help flush out bugs (e.g., if something broke when the clock flipped over to 2021, our test suite would let us know). But historically that hasn't been the case; all time-dependent outcomes we've seen turned out to be accidentally flaky tests (which we fixed by using test_tick). If we do want to cover handling the current time, we should dedicate one script to doing so, and have it unset GIT_COMMITTER_DATE explicitly. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-09 22:42:04 +02:00
export GIT_COMMITTER_DATE GIT_AUTHOR_DATE
export EDITOR
GIT_DEFAULT_HASH="${GIT_TEST_DEFAULT_HASH:-sha1}"
export GIT_DEFAULT_HASH
GIT_TEST_MERGE_ALGORITHM="${GIT_TEST_MERGE_ALGORITHM:-ort}"
export GIT_TEST_MERGE_ALGORITHM
# Tests using GIT_TRACE typically don't want <timestamp> <file>:<line> output
GIT_TRACE_BARE=1
export GIT_TRACE_BARE
# Use specific version of the index file format
if test -n "${GIT_TEST_INDEX_VERSION:+isset}"
then
GIT_INDEX_VERSION="$GIT_TEST_INDEX_VERSION"
export GIT_INDEX_VERSION
fi
perl: check for perl warnings while running tests We set "use warnings" in most of our perl code to catch problems. But as the name implies, warnings just emit a message to stderr and don't otherwise affect the program. So our tests are quite likely to miss that warnings are being spewed, as most of them do not look at stderr. We could ask perl to make all warnings fatal, but this is likely annoying for non-developers, who would rather have a running program with a warning than something that refuses to work at all. So instead, let's teach the perl code to respect an environment variable (GIT_PERL_FATAL_WARNINGS) to increase the severity of the warnings. This can be set for day-to-day running if people want to be really pedantic, but the primary use is to trigger it within the test suite. We could also trigger that for every test run, but likewise even the tests failing may be annoying to distro builders, etc (just as -Werror would be for compiling C code). So we'll tie it to a special test-mode variable (GIT_TEST_PERL_FATAL_WARNINGS) that can be set in the environment or as a Makefile knob, and we'll automatically turn the knob when DEVELOPER=1 is set. That should give developers and CI the more careful view without disrupting normal users or packagers. Note that the mapping from the GIT_TEST_* form to the GIT_* form in test-lib.sh is necessary even if they had the same name: the perl scripts need it to be normalized to a perl truth value, and we also have to make sure it's exported (we might have gotten it from the environment, but we might also have gotten it from GIT-BUILD-OPTIONS directly). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-10-22 05:24:00 +02:00
if test -n "$GIT_TEST_PERL_FATAL_WARNINGS"
then
GIT_PERL_FATAL_WARNINGS=1
export GIT_PERL_FATAL_WARNINGS
fi
Add MALLOC_CHECK_ and MALLOC_PERTURB_ libc env to the test suite for detecting heap corruption Recent versions of Linux libc (later than 5.4.23) and glibc (2.x) include a malloc() implementation which is tunable via environment variables. When MALLOC_CHECK_ is set, a special (less efficient) implementation is used which is designed to be tolerant against simple errors, such as double calls of free() with the same argument, or overruns of a single byte (off-by-one bugs). When MALLOC_CHECK_ is set to 3, a diagnostic message is printed on stderr and the program is aborted. Setting the MALLOC_PERTURB_ environment variable causes the malloc functions in libc to return memory which has been wiped and clear memory when it is returned. Of course this does not affect calloc which always does clear the memory. The reason for this exercise is, of course, to find code which uses memory returned by malloc without initializing it and code which uses code after it is freed. valgrind can do this but it's costly to run. The MALLOC_PERTURB_ exchanges the ability to detect problems in 100% of the cases with speed. The byte value used to initialize values returned by malloc is the byte value of the environment value. The value used to clear memory is the bitwise inverse. Setting MALLOC_PERTURB_ to zero disables the feature. This technique can find hard to detect bugs. It is therefore suggested to always use this flag (at least temporarily) when testing out code or a new distribution. But the test suite can use also valgrind(memcheck) via 'make valgrind' or 'make GIT_TEST_OPTS="--valgrind"'. Memcheck wraps client calls to malloc(), and puts a "red zone" on each end of each block in order to detect access overruns. Memcheck already detects double free() (up to the limit of the buffer which remembers pending free()). Thus memcheck subsumes all the documented coverage of MALLOC_CHECK_. If MALLOC_CHECK_ is set non-zero when running memcheck, then the overruns that might be detected by MALLOC_CHECK_ would be overruns on the wrapped blocks which include the red zones. Thus MALLOC_CHECK_ would be checking memcheck, and not the client. This is not useful, and actually is wasteful. The only possible [documented] advantage of using MALLOC_CHECK_ and memcheck together, would be if MALLOC_CHECK_ detected duplicate free() in more cases than memcheck because memcheck's buffer is too small. Therefore we don't use MALLOC_CHECK_ and valgrind(memcheck) at the same time. Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-09-14 18:54:22 +02:00
# Add libc MALLOC and MALLOC_PERTURB test
# only if we are not executing the test with valgrind
if test -n "$valgrind" ||
test -n "$TEST_NO_MALLOC_CHECK"
then
setup_malloc_check () {
: nothing
}
teardown_malloc_check () {
: nothing
}
else
setup_malloc_check () {
MALLOC_CHECK_=3 MALLOC_PERTURB_=165
export MALLOC_CHECK_ MALLOC_PERTURB_
}
teardown_malloc_check () {
unset MALLOC_CHECK_ MALLOC_PERTURB_
}
fi
Add MALLOC_CHECK_ and MALLOC_PERTURB_ libc env to the test suite for detecting heap corruption Recent versions of Linux libc (later than 5.4.23) and glibc (2.x) include a malloc() implementation which is tunable via environment variables. When MALLOC_CHECK_ is set, a special (less efficient) implementation is used which is designed to be tolerant against simple errors, such as double calls of free() with the same argument, or overruns of a single byte (off-by-one bugs). When MALLOC_CHECK_ is set to 3, a diagnostic message is printed on stderr and the program is aborted. Setting the MALLOC_PERTURB_ environment variable causes the malloc functions in libc to return memory which has been wiped and clear memory when it is returned. Of course this does not affect calloc which always does clear the memory. The reason for this exercise is, of course, to find code which uses memory returned by malloc without initializing it and code which uses code after it is freed. valgrind can do this but it's costly to run. The MALLOC_PERTURB_ exchanges the ability to detect problems in 100% of the cases with speed. The byte value used to initialize values returned by malloc is the byte value of the environment value. The value used to clear memory is the bitwise inverse. Setting MALLOC_PERTURB_ to zero disables the feature. This technique can find hard to detect bugs. It is therefore suggested to always use this flag (at least temporarily) when testing out code or a new distribution. But the test suite can use also valgrind(memcheck) via 'make valgrind' or 'make GIT_TEST_OPTS="--valgrind"'. Memcheck wraps client calls to malloc(), and puts a "red zone" on each end of each block in order to detect access overruns. Memcheck already detects double free() (up to the limit of the buffer which remembers pending free()). Thus memcheck subsumes all the documented coverage of MALLOC_CHECK_. If MALLOC_CHECK_ is set non-zero when running memcheck, then the overruns that might be detected by MALLOC_CHECK_ would be overruns on the wrapped blocks which include the red zones. Thus MALLOC_CHECK_ would be checking memcheck, and not the client. This is not useful, and actually is wasteful. The only possible [documented] advantage of using MALLOC_CHECK_ and memcheck together, would be if MALLOC_CHECK_ detected duplicate free() in more cases than memcheck because memcheck's buffer is too small. Therefore we don't use MALLOC_CHECK_ and valgrind(memcheck) at the same time. Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-09-14 18:54:22 +02:00
# Protect ourselves from common misconfiguration to export
# CDPATH into the environment
unset CDPATH
unset GREP_OPTIONS
unset UNZIP
case $(echo $GIT_TRACE |tr "[A-Z]" "[a-z]") in
1|2|true)
GIT_TRACE=4
;;
esac
# Line feed
LF='
'
# Single quote
SQ=\'
# UTF-8 ZERO WIDTH NON-JOINER, which HFS+ ignores
# when case-folding filenames
u200c=$(printf '\342\200\214')
export _x05 _x35 _x40 _z40 LF u200c EMPTY_TREE EMPTY_BLOB ZERO_OID OID_REGEX
Introduce a performance testing framework This introduces a performance testing framework under t/perf/. It tries to be as close to the test-lib.sh infrastructure as possible, and thus should be easy to get used to for git developers. The following points were considered for the implementation: 1. You usually want to compare arbitrary revisions/build trees against each other. They may not have the performance test under consideration, or even the perf-lib.sh infrastructure. To cope with this, the 'run' script lets you specify arbitrary build dirs and revisions. It even automatically builds the revisions if it doesn't have them at hand yet. 2. Usually you would not want to run all tests. It would take too long anyway. The 'run' script lets you specify which tests to run; or you can also do it manually. There is a Makefile for discoverability and 'make clean', but it is not meant for real-world use. 3. Creating test repos from scratch in every test is extremely time-consuming, and shipping or downloading such large/weird repos is out of the question. We leave this decision to the user. Two different sizes of test repos can be configured, and the scripts just copy one or more of those (using hardlinks for the object store). By default it tries to use the build tree's git.git repository. This is fairly fast and versatile. Using a copy instead of a clone preserves many properties that the user may want to test for, such as lots of loose objects, unpacked refs, etc. Signed-off-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-17 11:25:09 +01:00
# Each test should start with something like this, after copyright notices:
#
# test_description='Description of this test...
# This test checks if command xyzzy does the right thing...
# '
# . ./test-lib.sh
test "x$TERM" != "xdumb" && (
test -t 1 &&
tput bold >/dev/null 2>&1 &&
tput setaf 1 >/dev/null 2>&1 &&
tput sgr0 >/dev/null 2>&1
) &&
color=t
if test -n "$color"
then
# Save the color control sequences now rather than run tput
# each time say_color() is called. This is done for two
# reasons:
# * TERM will be changed to dumb
# * HOME will be changed to a temporary directory and tput
# might need to read ~/.terminfo from the original HOME
# directory to get the control sequences
# Note: This approach assumes the control sequences don't end
# in a newline for any terminal of interest (command
# substitutions strip trailing newlines). Given that most
# (all?) terminals in common use are related to ECMA-48, this
# shouldn't be a problem.
say_color_error=$(tput bold; tput setaf 1) # bold red
say_color_skip=$(tput setaf 4) # blue
say_color_warn=$(tput setaf 3) # brown/yellow
say_color_pass=$(tput setaf 2) # green
say_color_info=$(tput setaf 6) # cyan
say_color_reset=$(tput sgr0)
say_color_="" # no formatting for normal text
say_color () {
test -z "$1" && test -n "$quiet" && return
eval "say_color_color=\$say_color_$1"
shift
printf "%s\\n" "$say_color_color$*$say_color_reset"
}
else
say_color() {
test -z "$1" && test -n "$quiet" && return
shift
printf "%s\n" "$*"
}
fi
TERM=dumb
export TERM
error () {
say_color error "error: $*"
finalize_junit_xml
GIT_EXIT_OK=t
exit 1
}
tests: send "bug in the test script" errors to the script's stderr Some of the functions in our test library check that they were invoked properly with conditions like this: test "$#" = 2 || error "bug in the test script: not 2 parameters to test-expect-success" If this particular condition is triggered, then 'error' will abort the whole test script with a bold red error message [1] right away. However, under certain circumstances the test script will be aborted completely silently, namely if: - a similar condition in a test helper function like 'test_line_count' is triggered, - which is invoked from the test script's "main" shell [2], - and the test script is run manually (i.e. './t1234-foo.sh' as opposed to 'make t1234-foo.sh' or 'make test') [3] - and without the '--verbose' option, because the error message is printed from within 'test_eval_', where standard output is redirected either to /dev/null or to a log file. The only indication that something is wrong is that not all tests in the script are executed and at the end of the test script's output there is no "# passed all N tests" message, which are subtle and can easily go unnoticed, as I had to experience myself. Send these "bug in the test script" error messages directly to the test scripts standard error and thus to the terminal, so those bugs will be much harder to overlook. Instead of updating all ~20 such 'error' calls with a redirection, let's add a BUG() function to 'test-lib.sh', wrapping an 'error' call with the proper redirection and also including the common prefix of those error messages, and convert all those call sites [4] to use this new BUG() function instead. [1] That particular error message from 'test_expect_success' is printed in color only when running with or without '--verbose'; with '--tee' or '--verbose-log' the error is printed without color, but it is printed to the terminal nonetheless. [2] If such a condition is triggered in a subshell of a test, then 'error' won't be able to abort the whole test script, but only the subshell, which in turn causes the test to fail in the usual way, indicating loudly and clearly that something is wrong. [3] Well, 'error' aborts the test script the same way when run manually or by 'make' or 'prove', but both 'make' and 'prove' pay attention to the test script's exit status, and even a silently aborted test script would then trigger those tools' usual noticable error messages. [4] Strictly speaking, not all those 'error' calls need that redirection to send their output to the terminal, see e.g. 'test_expect_success' in the opening example, but I think it's better to be consistent. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-19 14:13:26 +01:00
BUG () {
error >&7 "bug in the test script: $*"
}
say () {
say_color info "$*"
}
if test -n "$HARNESS_ACTIVE"
then
if test "$verbose" = t || test -n "$verbose_only"
then
printf 'Bail out! %s\n' \
'verbose mode forbidden under TAP harness; try --verbose-log'
exit 1
fi
fi
test "${test_description}" != "" ||
error "Test script did not set test_description."
if test "$help" = "t"
then
printf '%s\n' "$test_description"
exit 0
fi
exec 5>&1
exec 6<&0
exec 7>&2
test-lib: add --verbose-log option The "--verbose" option redirects output from arbitrary test commands to stdout. This is useful for examining the output manually, like: ./t5547-push-quarantine.sh -v | less But it also means that the output is intermingled with the TAP directives, which can confuse a TAP parser like "prove". This has always been a potential problem, but became an issue recently when one test happened to output the word "ok" on a line by itself, which prove interprets as a test success: $ prove t5547-push-quarantine.sh :: -v t5547-push-quarantine.sh .. 1/? To dest.git * [new branch] HEAD -> master To dest.git ! [remote rejected] reject -> reject (pre-receive hook declined) error: failed to push some refs to 'dest.git' fatal: git cat-file d08c8eba97f4e683ece08654c7c8d2ba0c03b129: bad file t5547-push-quarantine.sh .. Failed -1/4 subtests Test Summary Report ------------------- t5547-push-quarantine.sh (Wstat: 0 Tests: 5 Failed: 0) Parse errors: Tests out of sequence. Found (2) but expected (3) Tests out of sequence. Found (3) but expected (4) Tests out of sequence. Found (4) but expected (5) Bad plan. You planned 4 tests but ran 5. Files=1, Tests=5, 0 wallclock secs ( 0.01 usr + 0.01 sys = 0.02 CPU) Result: FAIL One answer is "if it hurts, don't do it", but that's not quite the whole story. The Travis tests use "--verbose --tee" so that they can get the benefit of prove's parallel options, along with a verbose log in case there is a failure. We just need the verbose output to go to the log, but keep stdout clean. Getting this right turns out to be surprisingly difficult. Here's the progression of alternatives I considered: 1. Add an option to write verbose output to stderr. This is hard to capture, though, because we want each test to have its own log (because they're all run in parallel and the jumbled output would be useless). 2. Add an option to write verbose output to a file in test-results. This works, but the log is missing all of the non-verbose output, which gives context. 3. Like (2), but teach say_color() to additionally output to the log. This mostly works, but misses any output that happens outside of the say() functions (which isn't a lot, but is a potential maintenance headache). 4. Like (2), but make the log file the same as the "--tee" file. That almost works, but now we have two processes opening the same file. That gives us two separate descriptors, each with their own idea of the current position. They'll each start writing at offset 0, and overwrite each other's data. 5. Like (4), but in each case open the file for appending. That atomically positions each write at the end of the file. It's possible we may still get sheared writes between the two processes, but this is already the case when writing to stdout. It's not a problem in practice because the test harness generally waits for snippets to finish before writing the TAP output. We can ignore buffering issues with tee, because POSIX mandates that it does not buffer. Likewise, POSIX specifies "tee -a", so it should be available everywhere. This patch implements option (5), which seems to work well in practice. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-21 12:48:00 +02:00
if test "$verbose_log" = "t"
then
exec 3>>"$GIT_TEST_TEE_OUTPUT_FILE" 4>&3
elif test "$verbose" = "t"
then
exec 4>&2 3>&1
else
exec 4>/dev/null 3>/dev/null
fi
test-lib: set BASH_XTRACEFD automatically Passing "-x" to a test script enables the shell's "set -x" tracing, which can help with tracking down the command that is causing a failure. Unfortunately, it can also _cause_ failures in some tests that redirect the stderr of a shell function. Inside the function the shell continues to respect "set -x", and the trace output is collected along with whatever stderr is generated normally by the function. You can see an example of this by running: ./t0040-parse-options.sh -x -i which will fail immediately in the first test, as it expects: test_must_fail some-cmd 2>output.err to leave output.err empty (but with "-x" it has our trace output). Unfortunately there isn't a portable or scalable solution to this. We could teach test_must_fail to disable "set -x", but that doesn't help any of the other functions or subshells. However, we can work around it by pointing the "set -x" output to our descriptor 4, which always points to the original stderr of the test script. Unfortunately this only works for bash, but it's better than nothing (and other shells will just ignore the BASH_XTRACEFD variable). The patch itself is a simple one-liner, but note the caveats in the accompanying comments. Automatic tests for our "-x" option may be a bit too meta (and a pain, because they are bash-specific), but I did confirm that it works correctly both with regular "-x" and with "--verbose-only=1". This works because the latter flips "set -x" off and on for particular tests (if it didn't, we would get tracing for all tests, as going to descriptor 4 effectively circumvents the verbose flag). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-11 15:44:04 +02:00
# Send any "-x" output directly to stderr to avoid polluting tests
# which capture stderr. We can do this unconditionally since it
# has no effect if tracing isn't turned on.
#
# Note that this sets up the trace fd as soon as we assign the variable, so it
# must come after the creation of descriptor 4 above. Likewise, we must never
# unset this, as it has the side effect of closing descriptor 4, which we
# use to show verbose tests to the user.
#
# Note also that we don't need or want to export it. The tracing is local to
# this shell, and we would not want to influence any shells we exec.
BASH_XTRACEFD=4
test_failure=0
test_count=0
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 10:50:53 +01:00
test_fixed=0
test_broken=0
test_success=0
test_external_has_tap=0
die () {
code=$?
test-lib: introduce 'test_atexit' When running Apache, 'git daemon', or p4d, we want to kill them at the end of the test script, otherwise a leftover daemon process will keep its port open indefinitely, and thus will interfere with subsequent executions of the same test script. So far, we stop these daemon processes "manually", i.e.: - by registering functions or commands in the trap on EXIT to stop the daemon while preserving the last seen exit code before the trap (to deal with a failure when run with '--immediate' or with interrupts by ctrl-C), - and by invoking these functions/commands last thing before 'test_done' (and sometimes restoring the test framework's default trap on EXIT, to prevent the daemons from being killed twice). On one hand, we do this inconsistently, e.g. 'git p4' tests invoke different functions in the trap on EXIT and in the last test before 'test_done', and they neither restore the test framework's default trap on EXIT nor preserve the last seen exit code. On the other hand, this is error prone, because, as shown in a previous patch in this series, any output from the cleanup commands in the trap on EXIT can prevent a proper cleanup when a test script run with '--verbose-log' and certain shells, notably 'dash', is interrupted. Let's introduce 'test_atexit', which is loosely modeled after 'test_when_finished', but has a broader scope: rather than running the commands after the current test case, run them when the test script finishes, and also run them when the test is interrupted, or exits early in case of a failure while the '--immediate' option is in effect. When running the cleanup commands at the end of a successful test, then they will be run in 'test_done' before it removes the trash directory, i.e. the cleanup commands will still be able to access any pidfiles or socket files in there. When running the cleanup commands after an interrupt or failure with '--immediate', then they will be run in the trap on EXIT. In both cases they will be run in 'test_eval_', i.e. both standard error and output of all cleanup commands will go where they should according to the '-v' or '--verbose-log' options, and thus won't cause any troubles when interrupting a test script run with '--verbose-log'. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-13 13:24:11 +01:00
# This is responsible for running the atexit commands even when a
# test script run with '--immediate' fails, or when the user hits
# ctrl-C, i.e. when 'test_done' is not invoked at all.
test_atexit_handler || code=$?
if test -n "$GIT_EXIT_OK"
then
exit $code
else
echo >&5 "FATAL: Unexpected exit with code $code"
exit 1
fi
}
GIT_EXIT_OK=
trap 'die' EXIT
test-lib: fix interrupt handling with 'dash' and '--verbose-log -x' When a test script run with 'dash' and '--verbose-log -x' is interrupted by ctrl-C, SIGTERM, or closing the terminal window, then most of the time the registered EXIT trap actions are not executed. This is an annoying issue with tests involving daemons, because they should run cleanup commands to kill those daemon processes in the trap on EXIT, but since these cleanup commands are not executed, the daemons are left alive and keep their port open, thus interfering with subsequent execution of the same test script. The cause of this issue is the subtle combination of several factors (bear with me, or skip over the indented part): - Even when the test script is interrupted, the cleanup commands are not run in the trap on INT, TERM, or HUP, but in the trap on EXIT after the trap on the signals invokes 'exit' [1]. - According to POSIX [2]: "The environment in which the shell executes a trap on EXIT shall be identical to the environment immediately after the last command executed before the trap on EXIT was taken." Pertinent to the issue at hand is that all open file descriptors and the state of '-x' tracing should be preserved. All shells I've tried [3] preserve '-x'. Unfortunately, however: - 'dash' doesn't conform to this when it comes to open file descriptors: even when standard output and/or error are redirected somewhere when 'exit' is invoked, anything written to them in the trap on EXIT goes to the script's original stdout and stderr [4]. We can't dismiss this with a simple "it doesn't conform to POSIX, so we don't care", because 'dash' is the default /bin/sh in some of the more popular Linux distros. - As far as I can tell, POSIX doesn't explicitly say anything about the environment of trap actions for various signals. In practice it seems that most shells behave sensibly and preserve both open file descriptors and the state of '-x' tracing for the traps on INT, TERM, and HUP, including even 'dash'. The exceptions are 'mksh' and 'lksh': they do preserve '-x', but not the open file descriptors. - When a test script run with '-x' tracing enabled is interrupted, then it's very likely that the signal arrives mid-test, i.e.: - while '-x' tracing is enabled, and, consequently, our trap actions on INT, TERM, HUP, and EXIT will produce trace output as well. - while standard output and error are redirected to a log file, to the test script's original standard output and error, or to /dev/null, depending on whether the test script was run with '--verbose-log', '-v', or neither. According to the above, we can't rely on these redirections still be in effect when running the traps on INT, TERM, HUP, and/or EXIT. - When a test script is run with '--verbose-log', then the test script is re-executed with its standard output and error piped into 'tee', in order to send the "regular" non-verbose test's output both to the terminal and to the log file. When the test is interrupted, then the signal interrupts the downstream 'tee' as well. Putting these together, when a test script run with 'dash' and '--verbose-log -x' is interrupted, then 'dash' tries to write the trace output from the EXIT trap to the script's original standard error, but it very likely can't, because the 'tee' downstream of the pipe is interrupted as well. This causes the shell running the test script to die because of SIGPIPE, without running any of the commands in the EXIT trap. Disable '-x' tracing in the trap on INT, TERM, and HUP to avoid this issue, as it disables tracing in the chained trap on EXIT as well. Wrap it in a '{ ... } 2>/dev/null' block, so the trace of the command disabling the tracing doesn't go to standard error either [5]. Note that it's not only '-x' tracing that can be problematic, but any shell builtin, e.g. 'echo', that writes to standard output or error in the trap on EXIT, while a test running with 'dash' and '--verbose-log' (even without '-x') is interrupted. As far as I can tell, this is not an issue at the moment: - The cleanup commands to stop the credential-helper, Apache, or 'p4d' don't use any such shell builtins. - stop_git_daemon() does use 'say' and 'error', both wrappers around 'echo', but it redirects 'say' to fd 3, i.e. to the log file, and while 'error' does write to standard output, it comes only after the daemon was killed. - The non-builtin commands that actually stop the daemons ('kill', 'apache2 -k stop', 'git credential-cache exit') are silent, so they won't get SIGPIPE before finishing their job. [1] The trap on EXIT must run cleanup commands, because we want to stop any daemons when a test script run with '--immediate' fails and exits early with error. By chaining up the trap on signals to the trap on EXIT we can deal with cleanup commands a bit simpler, because the tests involving daemons only have to set a single trap. [2] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#trap [3] The shells I tried: dash, Bash, ksh, ksh93, mksh, lksh, yash, BusyBox sh, FreeBSD /bin/sh, NetBSD /bin/sh. [4] $ cat trap-output.sh #!/bin/sh trap "echo output; echo error >&2" EXIT { exit; } >OUT 2>ERR $ dash ./trap-output.sh output error $ wc -c OUT ERR 0 OUT 0 ERR On a related note, 'ksh', 'ksh93', and BusyBox sh don't conform to the specs in this respect, either. [5] This '{ set +x; } 2>/dev/null' trick won't help those shells that show trace output for any redirections and don't preserve open file descriptors for the trap on INT, TERM and HUP. The only such shells I'm aware of are 'mksh' and 'lksh'. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-13 13:24:09 +01:00
# Disable '-x' tracing, because with some shells, notably dash, it
# prevents running the cleanup commands when a test script run with
# '--verbose-log -x' is interrupted.
trap '{ code=$?; set +x; } 2>/dev/null; exit $code' INT TERM HUP
# The user-facing functions are loaded from a separate file so that
# test_perf subshells can have them too
. "$TEST_DIRECTORY/test-lib-functions.sh"
# You are not expected to call test_ok_ and test_failure_ directly, use
# the test_expect_* functions instead.
test_ok_ () {
if test -n "$write_junit_xml"
then
write_junit_xml_testcase "$*"
fi
test_success=$(($test_success + 1))
say_color "" "ok $test_count - $@"
}
test_failure_ () {
if test -n "$write_junit_xml"
then
junit_insert="<failure message=\"not ok $test_count -"
junit_insert="$junit_insert $(xml_attr_encode "$1")\">"
junit_insert="$junit_insert $(xml_attr_encode \
tests: include detailed trace logs with --write-junit-xml upon failure The JUnit XML format lends itself to be presented in a powerful UI, where you can drill down to the information you are interested in very quickly. For test failures, this usually means that you want to see the detailed trace of the failing tests. With Travis CI, we passed the `--verbose-log` option to get those traces. However, that seems excessive, as we do not need/use the logs in almost all of those cases: only when a test fails do we have a way to include the trace. So let's do something different when using Azure DevOps: let's run all the tests with `--quiet` first, and only if a failure is encountered, try to trace the commands as they are executed. Of course, we cannot turn on `--verbose-log` after the fact. So let's just re-run the test with all the same options, adding `--verbose-log`. And then munging the output file into the JUnit XML on the fly. Note: there is an off chance that re-running the test in verbose mode "fixes" the failures (and this does happen from time to time!). That is a possibility we should be able to live with. Ideally, we would label this as "Passed upon rerun", and Azure Pipelines even know about that outcome, but it is not available when using the JUnit XML format for now: https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:34 +01:00
"$(if test -n "$GIT_TEST_TEE_OUTPUT_FILE"
then
test-tool path-utils skip-n-bytes \
"$GIT_TEST_TEE_OUTPUT_FILE" $GIT_TEST_TEE_OFFSET
else
printf '%s\n' "$@" | sed 1d
fi)")"
junit_insert="$junit_insert</failure>"
tests: include detailed trace logs with --write-junit-xml upon failure The JUnit XML format lends itself to be presented in a powerful UI, where you can drill down to the information you are interested in very quickly. For test failures, this usually means that you want to see the detailed trace of the failing tests. With Travis CI, we passed the `--verbose-log` option to get those traces. However, that seems excessive, as we do not need/use the logs in almost all of those cases: only when a test fails do we have a way to include the trace. So let's do something different when using Azure DevOps: let's run all the tests with `--quiet` first, and only if a failure is encountered, try to trace the commands as they are executed. Of course, we cannot turn on `--verbose-log` after the fact. So let's just re-run the test with all the same options, adding `--verbose-log`. And then munging the output file into the JUnit XML on the fly. Note: there is an off chance that re-running the test in verbose mode "fixes" the failures (and this does happen from time to time!). That is a possibility we should be able to live with. Ideally, we would label this as "Passed upon rerun", and Azure Pipelines even know about that outcome, but it is not available when using the JUnit XML format for now: https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:34 +01:00
if test -n "$GIT_TEST_TEE_OUTPUT_FILE"
then
junit_insert="$junit_insert<system-err>$(xml_attr_encode \
"$(cat "$GIT_TEST_TEE_OUTPUT_FILE")")</system-err>"
fi
write_junit_xml_testcase "$1" " $junit_insert"
fi
test_failure=$(($test_failure + 1))
say_color error "not ok $test_count - $1"
shift
printf '%s\n' "$*" | sed -e 's/^/# /'
test "$immediate" = "" || { finalize_junit_xml; GIT_EXIT_OK=t; exit 1; }
}
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 10:50:53 +01:00
test_known_broken_ok_ () {
if test -n "$write_junit_xml"
then
write_junit_xml_testcase "$* (breakage fixed)"
fi
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 10:50:53 +01:00
test_fixed=$(($test_fixed+1))
say_color error "ok $test_count - $@ # TODO known breakage vanished"
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 10:50:53 +01:00
}
test_known_broken_failure_ () {
if test -n "$write_junit_xml"
then
write_junit_xml_testcase "$* (known breakage)"
fi
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 10:50:53 +01:00
test_broken=$(($test_broken+1))
say_color warn "not ok $test_count - $@ # TODO known breakage"
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 10:50:53 +01:00
}
test_debug () {
test "$debug" = "" || eval "$1"
}
match_pattern_list () {
arg="$1"
shift
test -z "$*" && return 1
for pattern_
do
case "$arg" in
$pattern_)
return 0
esac
done
return 1
}
match_test_selector_list () {
operation="$1"
shift
title="$1"
shift
arg="$1"
shift
test -z "$1" && return 0
# Commas are accepted as separators.
OLDIFS=$IFS
IFS=','
set -- $1
IFS=$OLDIFS
# If the first selector is negative we include by default.
include=
case "$1" in
!*) include=t ;;
esac
for selector
do
orig_selector=$selector
positive=t
case "$selector" in
!*)
positive=
selector=${selector##?}
;;
esac
test -z "$selector" && continue
case "$selector" in
*-*)
if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
then
echo "error: $operation: invalid non-numeric in range" \
"start: '$orig_selector'" >&2
exit 1
fi
if expr "z${selector#*-}" : "z[0-9]*[^0-9]" >/dev/null
then
echo "error: $operation: invalid non-numeric in range" \
"end: '$orig_selector'" >&2
exit 1
fi
;;
*)
if expr "z$selector" : "z[0-9]*[^0-9]" >/dev/null
then
case "$title" in *${selector}*)
include=$positive
;;
esac
continue
fi
esac
# Short cut for "obvious" cases
test -z "$include" && test -z "$positive" && continue
test -n "$include" && test -n "$positive" && continue
case "$selector" in
-*)
if test $arg -le ${selector#-}
then
include=$positive
fi
;;
*-)
if test $arg -ge ${selector%-}
then
include=$positive
fi
;;
*-*)
if test ${selector%%-*} -le $arg \
&& test $arg -le ${selector#*-}
then
include=$positive
fi
;;
*)
if test $arg -eq $selector
then
include=$positive
fi
;;
esac
done
test -n "$include"
}
maybe_teardown_verbose () {
test -z "$verbose_only" && return
exec 4>/dev/null 3>/dev/null
verbose=
}
last_verbose=t
maybe_setup_verbose () {
test -z "$verbose_only" && return
if match_pattern_list $test_count $verbose_only
then
exec 4>&2 3>&1
# Emit a delimiting blank line when going from
# non-verbose to verbose. Within verbose mode the
# delimiter is printed by test_expect_*. The choice
# of the initial $last_verbose is such that before
# test 1, we do not print it.
test -z "$last_verbose" && echo >&3 ""
verbose=t
else
exec 4>/dev/null 3>/dev/null
verbose=
fi
last_verbose=$verbose
}
maybe_teardown_valgrind () {
test -z "$GIT_VALGRIND" && return
GIT_VALGRIND_ENABLED=
}
maybe_setup_valgrind () {
test -z "$GIT_VALGRIND" && return
if test -z "$valgrind_only"
then
GIT_VALGRIND_ENABLED=t
return
fi
GIT_VALGRIND_ENABLED=
if match_pattern_list $test_count $valgrind_only
then
GIT_VALGRIND_ENABLED=t
fi
}
trace_level_=0
test-lib: disable trace when test is not verbose The "-x" test-script option turns on the shell's "-x" tracing, which can help show why a particular test is failing. Unfortunately, this can create false negatives in some tests if they invoke a shell function with its stderr redirected. t5512.10 is such a test, as it does: test_must_fail git ls-remote refs*master >actual 2>&1 && test_cmp exp actual The "actual" file gets the "-x" trace for the test_must_fail function, which prevents it from matching the expected output. There's no way to avoid this without managing the trace flag inside each sub-function, which isn't really a workable solution. But unless you specifically care about t5512.10, we can work around it by enabling tracing only for the specific tests we want. You can already do: ./t5512-ls-remote.sh -x --verbose-only=16 to see the trace only for a specific test. But that doesn't _disable_ the tracing in the other tests; it just sends it to /dev/null. However, there's no point in generating a trace that the user won't see, so we can simply disable tracing whenever it doesn't have a matching verbose flag. The normal case of just "./t5512-ls-remote.sh -x" stays the same, as "-x" already implies "--verbose" (and "--verbose-only" overrides "--verbose", which is why this works at all). And for our test, we need only check $verbose, as maybe_setup_verbose will have already set that flag based on the $verbose_only list). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-06 07:33:57 +02:00
want_trace () {
test "$trace" = t && {
test "$verbose" = t || test "$verbose_log" = t
}
test-lib: disable trace when test is not verbose The "-x" test-script option turns on the shell's "-x" tracing, which can help show why a particular test is failing. Unfortunately, this can create false negatives in some tests if they invoke a shell function with its stderr redirected. t5512.10 is such a test, as it does: test_must_fail git ls-remote refs*master >actual 2>&1 && test_cmp exp actual The "actual" file gets the "-x" trace for the test_must_fail function, which prevents it from matching the expected output. There's no way to avoid this without managing the trace flag inside each sub-function, which isn't really a workable solution. But unless you specifically care about t5512.10, we can work around it by enabling tracing only for the specific tests we want. You can already do: ./t5512-ls-remote.sh -x --verbose-only=16 to see the trace only for a specific test. But that doesn't _disable_ the tracing in the other tests; it just sends it to /dev/null. However, there's no point in generating a trace that the user won't see, so we can simply disable tracing whenever it doesn't have a matching verbose flag. The normal case of just "./t5512-ls-remote.sh -x" stays the same, as "-x" already implies "--verbose" (and "--verbose-only" overrides "--verbose", which is why this works at all). And for our test, we need only check $verbose, as maybe_setup_verbose will have already set that flag based on the $verbose_only list). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-06 07:33:57 +02:00
}
# This is a separate function because some tests use
# "return" to end a test_expect_success block early
# (and we want to make sure we run any cleanup like
# "set +x").
test_eval_inner_ () {
# Do not add anything extra (including LF) after '$*'
eval "
want_trace && trace_level_=$(($trace_level_+1)) && set -x
$*"
}
test_eval_ () {
# If "-x" tracing is in effect, then we want to avoid polluting stderr
# with non-test commands. But once in "set -x" mode, we cannot prevent
# the shell from printing the "set +x" to turn it off (nor the saving
# of $? before that). But we can make sure that the output goes to
# /dev/null.
#
# There are a few subtleties here:
#
# - we have to redirect descriptor 4 in addition to 2, to cover
# BASH_XTRACEFD
#
# - the actual eval has to come before the redirection block (since
# it needs to see descriptor 4 to set up its stderr)
#
# - likewise, any error message we print must be outside the block to
# access descriptor 4
#
# - checking $? has to come immediately after the eval, but it must
# be _inside_ the block to avoid polluting the "set -x" output
#
test_eval_inner_ "$@" </dev/null >&3 2>&4
{
test_eval_ret_=$?
test-lib: disable trace when test is not verbose The "-x" test-script option turns on the shell's "-x" tracing, which can help show why a particular test is failing. Unfortunately, this can create false negatives in some tests if they invoke a shell function with its stderr redirected. t5512.10 is such a test, as it does: test_must_fail git ls-remote refs*master >actual 2>&1 && test_cmp exp actual The "actual" file gets the "-x" trace for the test_must_fail function, which prevents it from matching the expected output. There's no way to avoid this without managing the trace flag inside each sub-function, which isn't really a workable solution. But unless you specifically care about t5512.10, we can work around it by enabling tracing only for the specific tests we want. You can already do: ./t5512-ls-remote.sh -x --verbose-only=16 to see the trace only for a specific test. But that doesn't _disable_ the tracing in the other tests; it just sends it to /dev/null. However, there's no point in generating a trace that the user won't see, so we can simply disable tracing whenever it doesn't have a matching verbose flag. The normal case of just "./t5512-ls-remote.sh -x" stays the same, as "-x" already implies "--verbose" (and "--verbose-only" overrides "--verbose", which is why this works at all). And for our test, we need only check $verbose, as maybe_setup_verbose will have already set that flag based on the $verbose_only list). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-06 07:33:57 +02:00
if want_trace
then
test 1 = $trace_level_ && set +x
trace_level_=$(($trace_level_-1))
fi
} 2>/dev/null 4>&2
if test "$test_eval_ret_" != 0 && want_trace
then
say_color error >&4 "error: last command exited with \$?=$test_eval_ret_"
fi
return $test_eval_ret_
}
test_run_ () {
test_cleanup=:
expecting_failure=$2
t/test-lib: introduce --chain-lint option It's easy to miss an "&&"-chain in a test script, like: test_expect_success 'check something important' ' cmd1 && cmd2 cmd3 ' The test harness will notice if cmd3 fails, but a failure of cmd1 or cmd2 will go unnoticed, as their exit status is lost after cmd3 runs. The toy example above is easy to spot because the "cmds" are all the same length, but real code is much more complicated. It's also difficult to detect these situations by statically analyzing the shell code with regexps (like the check-non-portable-shell script does); there's too much context required to know whether a &&-chain is appropriate on a given line or not. This patch instead lets the shell check each test by sticking a command with a specific and unusual return code at the top of each test, like: (exit 117) && cmd1 && cmd2 cmd3 In a well-formed test, the non-zero exit from the first command prevents any of the rest from being run, and the test's exit code is 117. In a bad test (like the one above), the 117 is lost, and cmd3 is run. When we encounter a failure of this check, we abort the test script entirely. For one thing, we have no clue which subset of the commands in the test snippet were actually run. Running further tests would be pointless, because we're now in an unknown state. And two, this is not a "test failure" in the traditional sense. The test script is buggy, not the code it is testing. We should be able to fix these problems in the script once, and not have them come back later as a regression in git's code. After checking a test snippet for --chain-lint, we do still run the test itself. We could actually have a pure-lint mode which just checks each test, but there are a few reasons not to. One, because the tests are executing arbitrary code, which could impact the later environment (e.g., that could impact which set of tests we run at all). And two, because a pure-lint mode would still be expensive to run, because a significant amount of code runs outside of the test_expect_* blocks. Instead, this option is designed to be used as part of a normal test suite run, where it adds very little overhead. Turning on this option detects quite a few problems in existing tests, which will be fixed in subsequent patches. However, there are a number of places it cannot reach: - it cannot find a failure to break out of loops on error, like: cmd1 && for i in a b c; do cmd2 $i done && cmd3 which will not notice failures of "cmd2 a" or "cmd b" - it cannot find a missing &&-chain inside a block or subfunction, like: foo () { cmd1 cmd2 } foo && bar which will not notice a failure of cmd1. - it only checks tests that you run; every platform will have some tests skipped due to missing prequisites, so it's impossible to say from one run that the test suite is free of broken &&-chains. However, all tests get run by _somebody_, so eventually we will notice problems. - it does not operate on test_when_finished or prerequisite blocks. It could, but these tends to be much shorter and less of a problem, so I punted on them in this patch. This patch was inspired by an earlier patch by Jonathan Nieder: http://article.gmane.org/gmane.comp.version-control.git/235913 This implementation and all bugs are mine. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20 11:05:48 +01:00
if test "${GIT_TEST_CHAIN_LINT:-1}" != 0; then
# turn off tracing for this test-eval, as it simply creates
# confusing noise in the "-x" output
trace_tmp=$trace
trace=
t/test-lib: introduce --chain-lint option It's easy to miss an "&&"-chain in a test script, like: test_expect_success 'check something important' ' cmd1 && cmd2 cmd3 ' The test harness will notice if cmd3 fails, but a failure of cmd1 or cmd2 will go unnoticed, as their exit status is lost after cmd3 runs. The toy example above is easy to spot because the "cmds" are all the same length, but real code is much more complicated. It's also difficult to detect these situations by statically analyzing the shell code with regexps (like the check-non-portable-shell script does); there's too much context required to know whether a &&-chain is appropriate on a given line or not. This patch instead lets the shell check each test by sticking a command with a specific and unusual return code at the top of each test, like: (exit 117) && cmd1 && cmd2 cmd3 In a well-formed test, the non-zero exit from the first command prevents any of the rest from being run, and the test's exit code is 117. In a bad test (like the one above), the 117 is lost, and cmd3 is run. When we encounter a failure of this check, we abort the test script entirely. For one thing, we have no clue which subset of the commands in the test snippet were actually run. Running further tests would be pointless, because we're now in an unknown state. And two, this is not a "test failure" in the traditional sense. The test script is buggy, not the code it is testing. We should be able to fix these problems in the script once, and not have them come back later as a regression in git's code. After checking a test snippet for --chain-lint, we do still run the test itself. We could actually have a pure-lint mode which just checks each test, but there are a few reasons not to. One, because the tests are executing arbitrary code, which could impact the later environment (e.g., that could impact which set of tests we run at all). And two, because a pure-lint mode would still be expensive to run, because a significant amount of code runs outside of the test_expect_* blocks. Instead, this option is designed to be used as part of a normal test suite run, where it adds very little overhead. Turning on this option detects quite a few problems in existing tests, which will be fixed in subsequent patches. However, there are a number of places it cannot reach: - it cannot find a failure to break out of loops on error, like: cmd1 && for i in a b c; do cmd2 $i done && cmd3 which will not notice failures of "cmd2 a" or "cmd b" - it cannot find a missing &&-chain inside a block or subfunction, like: foo () { cmd1 cmd2 } foo && bar which will not notice a failure of cmd1. - it only checks tests that you run; every platform will have some tests skipped due to missing prequisites, so it's impossible to say from one run that the test suite is free of broken &&-chains. However, all tests get run by _somebody_, so eventually we will notice problems. - it does not operate on test_when_finished or prerequisite blocks. It could, but these tends to be much shorter and less of a problem, so I punted on them in this patch. This patch was inspired by an earlier patch by Jonathan Nieder: http://article.gmane.org/gmane.comp.version-control.git/235913 This implementation and all bugs are mine. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20 11:05:48 +01:00
# 117 is magic because it is unlikely to match the exit
# code of other programs
t: avoid sed-based chain-linting in some expensive cases Commit 878f988350 (t/test-lib: teach --chain-lint to detect broken &&-chains in subshells, 2018-07-11) introduced additional chain-lint tests which add an extra "sed" pipeline to each test we run. This has a measurable impact on runtime. Here are timings with and without a new environment variable (added by this patch) that lets you disable just the additional sed-based chain-lint tests: Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER=1 make test Time (mean ± σ): 64.202 s ± 1.030 s [User: 622.469 s, System: 301.402 s] Range (min … max): 61.571 s … 65.662 s 10 runs Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER=0 make test Time (mean ± σ): 57.591 s ± 0.333 s [User: 529.368 s, System: 270.618 s] Range (min … max): 57.143 s … 58.309 s 10 runs Summary 'GIT_TEST_CHAIN_LINT_HARDER=0 make test' ran 1.11 ± 0.02 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=1 make test' Of course those extra lint checks are doing something useful, so paying a few extra seconds (at least on Linux) isn't so bad (though note the CPU time; we're bounded in our parallel run here by the slowest test, so it really is ~120s of CPU improvement). But we can observe that there are some test scripts where they produce a much stronger effect, and provide less value. In t0027 and t3070 we run a very large number of small tests, all driven by a series of functions/loops which are filling in the test bodies. There we get much less bang for our buck in terms of bug-finding versus CPU cost. This patch introduces a mechanism for controlling when those extra lint checks are run, at two levels: - a user can ask to disable or to force-enable the checks by setting GIT_TEST_CHAIN_LINT_HARDER - if the user hasn't specified a preference, individual scripts can disable the checks by setting GIT_TEST_CHAIN_LINT_HARDER_DEFAULT; scripts which don't set that get the current behavior of enabling them. In addition, this patch flips the default for t0027 and t3070's mass-generated sections to disable the extra checks. Here are the timing results for t0027: Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER=1 ./t0027-auto-crlf.sh Time (mean ± σ): 17.078 s ± 0.848 s [User: 14.878 s, System: 7.075 s] Range (min … max): 15.952 s … 18.421 s 10 runs Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER=0 ./t0027-auto-crlf.sh Time (mean ± σ): 9.063 s ± 0.759 s [User: 7.890 s, System: 3.362 s] Range (min … max): 7.747 s … 10.619 s 10 runs Benchmark #3: ./t0027-auto-crlf.sh Time (mean ± σ): 9.186 s ± 0.881 s [User: 7.957 s, System: 3.427 s] Range (min … max): 7.796 s … 10.498 s 10 runs Summary 'GIT_TEST_CHAIN_LINT_HARDER=0 ./t0027-auto-crlf.sh' ran 1.01 ± 0.13 times faster than './t0027-auto-crlf.sh' 1.88 ± 0.18 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=1 ./t0027-auto-crlf.sh' We can see that disabling the checks for the whole script buys us an almost 2x speedup. But the new default behavior, disabling them only for the mass-generated part, gets us most of that speedup (but still leaves the checks on for further manual tests people might write). As a side note, I'd caution about comparing runtimes and CPU seconds between this timing and the earlier "make test" one. In "make test", we're running a lot of scripts in parallel, so the CPU is throttling down (and thus a CPU second saved here would count for more during a parallel run; the same work takes more CPU seconds there). We get similar results for t3070: Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER=1 ./t3070-wildmatch.sh Time (mean ± σ): 20.054 s ± 3.967 s [User: 16.003 s, System: 8.286 s] Range (min … max): 11.891 s … 23.671 s 10 runs Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER=0 ./t3070-wildmatch.sh Time (mean ± σ): 12.399 s ± 2.256 s [User: 7.542 s, System: 5.342 s] Range (min … max): 9.606 s … 15.727 s 10 runs Benchmark #3: ./t3070-wildmatch.sh Time (mean ± σ): 10.726 s ± 3.476 s [User: 6.790 s, System: 4.365 s] Range (min … max): 5.444 s … 15.376 s 10 runs Summary './t3070-wildmatch.sh' ran 1.16 ± 0.43 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=0 ./t3070-wildmatch.sh' 1.87 ± 0.71 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=1 ./t3070-wildmatch.sh' Again, we get almost a 2x speedup disabling these. In this case, there are no tests not covered by the script's "default to disable" behavior, so the second two benchmarks should be the same (and while they do differ, you can see the variance is quite high but they're within one standard deviation). So it seems like for these two scripts, at least, disabling the extra checks is a reasonable tradeoff. Sadly, the overall runtime of "make test" on my system doesn't get much faster. But that's because we're mostly limited by the cost of the single biggest test. Here are the top-5 tests by wall-clock time from a parallel run, before my patch: 57.9192368984222 t9001-send-email.sh 45.6329638957977 t0027-auto-crlf.sh 32.5278220176697 t3070-wildmatch.sh 22.2701289653778 t7610-mergetool.sh 20.8635759353638 t1701-racy-split-index.sh And after: 57.1476998329163 t9001-send-email.sh 33.776211977005 t0027-auto-crlf.sh 21.3116669654846 t7610-mergetool.sh 20.7748689651489 t1701-racy-split-index.sh 19.6957249641418 t7112-reset-submodule.sh We dropped 12s from t0027, and t3070 dropped off our list entirely at around 16s. In both cases we're bound by t9001, but its slowness is due to the actual tests, so we'll have to deal with it in a different way. But this reduces overall CPU, and means that dealing with t9001 (by improving the speed of send-email or splitting it apart) will let us reduce our overall runtime even on multi-core machines. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13 08:25:53 +02:00
if test "OK-117" != "$(test_eval_ "(exit 117) && $1${LF}${LF}echo OK-\$?" 3>&1)" ||
{
test "${GIT_TEST_CHAIN_LINT_HARDER:-${GIT_TEST_CHAIN_LINT_HARDER_DEFAULT:-1}}" != 0 &&
$(printf '%s\n' "$1" | sed -f "$GIT_BUILD_DIR/t/chainlint.sed" | grep -q '?![A-Z][A-Z]*?!')
}
then
tests: send "bug in the test script" errors to the script's stderr Some of the functions in our test library check that they were invoked properly with conditions like this: test "$#" = 2 || error "bug in the test script: not 2 parameters to test-expect-success" If this particular condition is triggered, then 'error' will abort the whole test script with a bold red error message [1] right away. However, under certain circumstances the test script will be aborted completely silently, namely if: - a similar condition in a test helper function like 'test_line_count' is triggered, - which is invoked from the test script's "main" shell [2], - and the test script is run manually (i.e. './t1234-foo.sh' as opposed to 'make t1234-foo.sh' or 'make test') [3] - and without the '--verbose' option, because the error message is printed from within 'test_eval_', where standard output is redirected either to /dev/null or to a log file. The only indication that something is wrong is that not all tests in the script are executed and at the end of the test script's output there is no "# passed all N tests" message, which are subtle and can easily go unnoticed, as I had to experience myself. Send these "bug in the test script" error messages directly to the test scripts standard error and thus to the terminal, so those bugs will be much harder to overlook. Instead of updating all ~20 such 'error' calls with a redirection, let's add a BUG() function to 'test-lib.sh', wrapping an 'error' call with the proper redirection and also including the common prefix of those error messages, and convert all those call sites [4] to use this new BUG() function instead. [1] That particular error message from 'test_expect_success' is printed in color only when running with or without '--verbose'; with '--tee' or '--verbose-log' the error is printed without color, but it is printed to the terminal nonetheless. [2] If such a condition is triggered in a subshell of a test, then 'error' won't be able to abort the whole test script, but only the subshell, which in turn causes the test to fail in the usual way, indicating loudly and clearly that something is wrong. [3] Well, 'error' aborts the test script the same way when run manually or by 'make' or 'prove', but both 'make' and 'prove' pay attention to the test script's exit status, and even a silently aborted test script would then trigger those tools' usual noticable error messages. [4] Strictly speaking, not all those 'error' calls need that redirection to send their output to the terminal, see e.g. 'test_expect_success' in the opening example, but I think it's better to be consistent. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-19 14:13:26 +01:00
BUG "broken &&-chain or run-away HERE-DOC: $1"
t/test-lib: introduce --chain-lint option It's easy to miss an "&&"-chain in a test script, like: test_expect_success 'check something important' ' cmd1 && cmd2 cmd3 ' The test harness will notice if cmd3 fails, but a failure of cmd1 or cmd2 will go unnoticed, as their exit status is lost after cmd3 runs. The toy example above is easy to spot because the "cmds" are all the same length, but real code is much more complicated. It's also difficult to detect these situations by statically analyzing the shell code with regexps (like the check-non-portable-shell script does); there's too much context required to know whether a &&-chain is appropriate on a given line or not. This patch instead lets the shell check each test by sticking a command with a specific and unusual return code at the top of each test, like: (exit 117) && cmd1 && cmd2 cmd3 In a well-formed test, the non-zero exit from the first command prevents any of the rest from being run, and the test's exit code is 117. In a bad test (like the one above), the 117 is lost, and cmd3 is run. When we encounter a failure of this check, we abort the test script entirely. For one thing, we have no clue which subset of the commands in the test snippet were actually run. Running further tests would be pointless, because we're now in an unknown state. And two, this is not a "test failure" in the traditional sense. The test script is buggy, not the code it is testing. We should be able to fix these problems in the script once, and not have them come back later as a regression in git's code. After checking a test snippet for --chain-lint, we do still run the test itself. We could actually have a pure-lint mode which just checks each test, but there are a few reasons not to. One, because the tests are executing arbitrary code, which could impact the later environment (e.g., that could impact which set of tests we run at all). And two, because a pure-lint mode would still be expensive to run, because a significant amount of code runs outside of the test_expect_* blocks. Instead, this option is designed to be used as part of a normal test suite run, where it adds very little overhead. Turning on this option detects quite a few problems in existing tests, which will be fixed in subsequent patches. However, there are a number of places it cannot reach: - it cannot find a failure to break out of loops on error, like: cmd1 && for i in a b c; do cmd2 $i done && cmd3 which will not notice failures of "cmd2 a" or "cmd b" - it cannot find a missing &&-chain inside a block or subfunction, like: foo () { cmd1 cmd2 } foo && bar which will not notice a failure of cmd1. - it only checks tests that you run; every platform will have some tests skipped due to missing prequisites, so it's impossible to say from one run that the test suite is free of broken &&-chains. However, all tests get run by _somebody_, so eventually we will notice problems. - it does not operate on test_when_finished or prerequisite blocks. It could, but these tends to be much shorter and less of a problem, so I punted on them in this patch. This patch was inspired by an earlier patch by Jonathan Nieder: http://article.gmane.org/gmane.comp.version-control.git/235913 This implementation and all bugs are mine. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20 11:05:48 +01:00
fi
trace=$trace_tmp
t/test-lib: introduce --chain-lint option It's easy to miss an "&&"-chain in a test script, like: test_expect_success 'check something important' ' cmd1 && cmd2 cmd3 ' The test harness will notice if cmd3 fails, but a failure of cmd1 or cmd2 will go unnoticed, as their exit status is lost after cmd3 runs. The toy example above is easy to spot because the "cmds" are all the same length, but real code is much more complicated. It's also difficult to detect these situations by statically analyzing the shell code with regexps (like the check-non-portable-shell script does); there's too much context required to know whether a &&-chain is appropriate on a given line or not. This patch instead lets the shell check each test by sticking a command with a specific and unusual return code at the top of each test, like: (exit 117) && cmd1 && cmd2 cmd3 In a well-formed test, the non-zero exit from the first command prevents any of the rest from being run, and the test's exit code is 117. In a bad test (like the one above), the 117 is lost, and cmd3 is run. When we encounter a failure of this check, we abort the test script entirely. For one thing, we have no clue which subset of the commands in the test snippet were actually run. Running further tests would be pointless, because we're now in an unknown state. And two, this is not a "test failure" in the traditional sense. The test script is buggy, not the code it is testing. We should be able to fix these problems in the script once, and not have them come back later as a regression in git's code. After checking a test snippet for --chain-lint, we do still run the test itself. We could actually have a pure-lint mode which just checks each test, but there are a few reasons not to. One, because the tests are executing arbitrary code, which could impact the later environment (e.g., that could impact which set of tests we run at all). And two, because a pure-lint mode would still be expensive to run, because a significant amount of code runs outside of the test_expect_* blocks. Instead, this option is designed to be used as part of a normal test suite run, where it adds very little overhead. Turning on this option detects quite a few problems in existing tests, which will be fixed in subsequent patches. However, there are a number of places it cannot reach: - it cannot find a failure to break out of loops on error, like: cmd1 && for i in a b c; do cmd2 $i done && cmd3 which will not notice failures of "cmd2 a" or "cmd b" - it cannot find a missing &&-chain inside a block or subfunction, like: foo () { cmd1 cmd2 } foo && bar which will not notice a failure of cmd1. - it only checks tests that you run; every platform will have some tests skipped due to missing prequisites, so it's impossible to say from one run that the test suite is free of broken &&-chains. However, all tests get run by _somebody_, so eventually we will notice problems. - it does not operate on test_when_finished or prerequisite blocks. It could, but these tends to be much shorter and less of a problem, so I punted on them in this patch. This patch was inspired by an earlier patch by Jonathan Nieder: http://article.gmane.org/gmane.comp.version-control.git/235913 This implementation and all bugs are mine. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20 11:05:48 +01:00
fi
setup_malloc_check
test_eval_ "$1"
eval_ret=$?
teardown_malloc_check
if test -z "$immediate" || test $eval_ret = 0 ||
test -n "$expecting_failure" && test "$test_cleanup" != ":"
then
setup_malloc_check
test_eval_ "$test_cleanup"
teardown_malloc_check
fi
if test "$verbose" = "t" && test -n "$HARNESS_ACTIVE"
then
echo ""
fi
return "$eval_ret"
}
test_start_ () {
test_count=$(($test_count+1))
maybe_setup_verbose
maybe_setup_valgrind
if test -n "$write_junit_xml"
then
junit_start=$(test-tool date getnanos)
fi
}
test_finish_ () {
echo >&3 ""
maybe_teardown_valgrind
maybe_teardown_verbose
tests: include detailed trace logs with --write-junit-xml upon failure The JUnit XML format lends itself to be presented in a powerful UI, where you can drill down to the information you are interested in very quickly. For test failures, this usually means that you want to see the detailed trace of the failing tests. With Travis CI, we passed the `--verbose-log` option to get those traces. However, that seems excessive, as we do not need/use the logs in almost all of those cases: only when a test fails do we have a way to include the trace. So let's do something different when using Azure DevOps: let's run all the tests with `--quiet` first, and only if a failure is encountered, try to trace the commands as they are executed. Of course, we cannot turn on `--verbose-log` after the fact. So let's just re-run the test with all the same options, adding `--verbose-log`. And then munging the output file into the JUnit XML on the fly. Note: there is an off chance that re-running the test in verbose mode "fixes" the failures (and this does happen from time to time!). That is a possibility we should be able to live with. Ideally, we would label this as "Passed upon rerun", and Azure Pipelines even know about that outcome, but it is not available when using the JUnit XML format for now: https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:34 +01:00
if test -n "$GIT_TEST_TEE_OFFSET"
then
GIT_TEST_TEE_OFFSET=$(test-tool path-utils file-size \
"$GIT_TEST_TEE_OUTPUT_FILE")
fi
}
test_skip () {
to_skip=
skipped_reason=
if match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS
then
to_skip=t
skipped_reason="GIT_SKIP_TESTS"
fi
if test -z "$to_skip" && test -n "$run_list" &&
! match_test_selector_list '--run' "$1" $test_count "$run_list"
then
to_skip=t
skipped_reason="--run"
fi
if test -z "$to_skip" && test -n "$test_prereq" &&
! test_have_prereq "$test_prereq"
then
to_skip=t
of_prereq=
if test "$missing_prereq" != "$test_prereq"
then
of_prereq=" of $test_prereq"
fi
skipped_reason="missing $missing_prereq${of_prereq}"
fi
case "$to_skip" in
t)
if test -n "$write_junit_xml"
then
message="$(xml_attr_encode "$skipped_reason")"
write_junit_xml_testcase "$1" \
" <skipped message=\"$message\" />"
fi
say_color skip "ok $test_count # skip $1 ($skipped_reason)"
: true
;;
*)
false
;;
esac
}
Introduce a performance testing framework This introduces a performance testing framework under t/perf/. It tries to be as close to the test-lib.sh infrastructure as possible, and thus should be easy to get used to for git developers. The following points were considered for the implementation: 1. You usually want to compare arbitrary revisions/build trees against each other. They may not have the performance test under consideration, or even the perf-lib.sh infrastructure. To cope with this, the 'run' script lets you specify arbitrary build dirs and revisions. It even automatically builds the revisions if it doesn't have them at hand yet. 2. Usually you would not want to run all tests. It would take too long anyway. The 'run' script lets you specify which tests to run; or you can also do it manually. There is a Makefile for discoverability and 'make clean', but it is not meant for real-world use. 3. Creating test repos from scratch in every test is extremely time-consuming, and shipping or downloading such large/weird repos is out of the question. We leave this decision to the user. Two different sizes of test repos can be configured, and the scripts just copy one or more of those (using hardlinks for the object store). By default it tries to use the build tree's git.git repository. This is fairly fast and versatile. Using a copy instead of a clone preserves many properties that the user may want to test for, such as lots of loose objects, unpacked refs, etc. Signed-off-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-17 11:25:09 +01:00
# stub; perf-lib overrides it
test_at_end_hook_ () {
:
}
write_junit_xml () {
case "$1" in
--truncate)
>"$junit_xml_path"
junit_have_testcase=
shift
;;
esac
printf '%s\n' "$@" >>"$junit_xml_path"
}
xml_attr_encode () {
printf '%s\n' "$@" | test-tool xml-encode
}
write_junit_xml_testcase () {
junit_attrs="name=\"$(xml_attr_encode "$this_test.$test_count $1")\""
shift
junit_attrs="$junit_attrs classname=\"$this_test\""
junit_attrs="$junit_attrs time=\"$(test-tool \
date getnanos $junit_start)\""
write_junit_xml "$(printf '%s\n' \
" <testcase $junit_attrs>" "$@" " </testcase>")"
junit_have_testcase=t
}
finalize_junit_xml () {
if test -n "$write_junit_xml" && test -n "$junit_xml_path"
then
test -n "$junit_have_testcase" || {
junit_start=$(test-tool date getnanos)
write_junit_xml_testcase "all tests skipped"
}
# adjust the overall time
junit_time=$(test-tool date getnanos $junit_suite_start)
sed -e "s/\(<testsuite.*\) time=\"[^\"]*\"/\1/" \
-e "s/<testsuite [^>]*/& time=\"$junit_time\"/" \
-e '/^ *<\/testsuite/d' \
<"$junit_xml_path" >"$junit_xml_path.new"
mv "$junit_xml_path.new" "$junit_xml_path"
write_junit_xml " </testsuite>" "</testsuites>"
write_junit_xml=
fi
}
test-lib: introduce 'test_atexit' When running Apache, 'git daemon', or p4d, we want to kill them at the end of the test script, otherwise a leftover daemon process will keep its port open indefinitely, and thus will interfere with subsequent executions of the same test script. So far, we stop these daemon processes "manually", i.e.: - by registering functions or commands in the trap on EXIT to stop the daemon while preserving the last seen exit code before the trap (to deal with a failure when run with '--immediate' or with interrupts by ctrl-C), - and by invoking these functions/commands last thing before 'test_done' (and sometimes restoring the test framework's default trap on EXIT, to prevent the daemons from being killed twice). On one hand, we do this inconsistently, e.g. 'git p4' tests invoke different functions in the trap on EXIT and in the last test before 'test_done', and they neither restore the test framework's default trap on EXIT nor preserve the last seen exit code. On the other hand, this is error prone, because, as shown in a previous patch in this series, any output from the cleanup commands in the trap on EXIT can prevent a proper cleanup when a test script run with '--verbose-log' and certain shells, notably 'dash', is interrupted. Let's introduce 'test_atexit', which is loosely modeled after 'test_when_finished', but has a broader scope: rather than running the commands after the current test case, run them when the test script finishes, and also run them when the test is interrupted, or exits early in case of a failure while the '--immediate' option is in effect. When running the cleanup commands at the end of a successful test, then they will be run in 'test_done' before it removes the trash directory, i.e. the cleanup commands will still be able to access any pidfiles or socket files in there. When running the cleanup commands after an interrupt or failure with '--immediate', then they will be run in the trap on EXIT. In both cases they will be run in 'test_eval_', i.e. both standard error and output of all cleanup commands will go where they should according to the '-v' or '--verbose-log' options, and thus won't cause any troubles when interrupting a test script run with '--verbose-log'. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-13 13:24:11 +01:00
test_atexit_cleanup=:
test_atexit_handler () {
# In a succeeding test script 'test_atexit_handler' is invoked
# twice: first from 'test_done', then from 'die' in the trap on
# EXIT.
# This condition and resetting 'test_atexit_cleanup' below makes
# sure that the registered cleanup commands are run only once.
test : != "$test_atexit_cleanup" || return 0
setup_malloc_check
test_eval_ "$test_atexit_cleanup"
test_atexit_cleanup=:
teardown_malloc_check
}
test_done () {
GIT_EXIT_OK=t
test-lib: introduce 'test_atexit' When running Apache, 'git daemon', or p4d, we want to kill them at the end of the test script, otherwise a leftover daemon process will keep its port open indefinitely, and thus will interfere with subsequent executions of the same test script. So far, we stop these daemon processes "manually", i.e.: - by registering functions or commands in the trap on EXIT to stop the daemon while preserving the last seen exit code before the trap (to deal with a failure when run with '--immediate' or with interrupts by ctrl-C), - and by invoking these functions/commands last thing before 'test_done' (and sometimes restoring the test framework's default trap on EXIT, to prevent the daemons from being killed twice). On one hand, we do this inconsistently, e.g. 'git p4' tests invoke different functions in the trap on EXIT and in the last test before 'test_done', and they neither restore the test framework's default trap on EXIT nor preserve the last seen exit code. On the other hand, this is error prone, because, as shown in a previous patch in this series, any output from the cleanup commands in the trap on EXIT can prevent a proper cleanup when a test script run with '--verbose-log' and certain shells, notably 'dash', is interrupted. Let's introduce 'test_atexit', which is loosely modeled after 'test_when_finished', but has a broader scope: rather than running the commands after the current test case, run them when the test script finishes, and also run them when the test is interrupted, or exits early in case of a failure while the '--immediate' option is in effect. When running the cleanup commands at the end of a successful test, then they will be run in 'test_done' before it removes the trash directory, i.e. the cleanup commands will still be able to access any pidfiles or socket files in there. When running the cleanup commands after an interrupt or failure with '--immediate', then they will be run in the trap on EXIT. In both cases they will be run in 'test_eval_', i.e. both standard error and output of all cleanup commands will go where they should according to the '-v' or '--verbose-log' options, and thus won't cause any troubles when interrupting a test script run with '--verbose-log'. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-13 13:24:11 +01:00
# Run the atexit commands _before_ the trash directory is
# removed, so the commands can access pidfiles and socket files.
test_atexit_handler
finalize_junit_xml
if test -z "$HARNESS_ACTIVE"
then
mkdir -p "$TEST_RESULTS_DIR"
cat >"$TEST_RESULTS_BASE.counts" <<-EOF
total $test_count
success $test_success
fixed $test_fixed
broken $test_broken
failed $test_failure
EOF
fi
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 10:50:53 +01:00
if test "$test_fixed" != 0
then
say_color error "# $test_fixed known breakage(s) vanished; please update test(s)"
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 10:50:53 +01:00
fi
if test "$test_broken" != 0
then
say_color warn "# still have $test_broken known breakage(s)"
fi
if test "$test_broken" != 0 || test "$test_fixed" != 0
then
test_remaining=$(( $test_count - $test_broken - $test_fixed ))
msg="remaining $test_remaining test(s)"
else
test_remaining=$test_count
msg="$test_count test(s)"
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 10:50:53 +01:00
fi
case "$test_failure" in
0)
if test $test_external_has_tap -eq 0
then
if test $test_remaining -gt 0
then
say_color pass "# passed all $msg"
fi
# Maybe print SKIP message
test -z "$skip_all" || skip_all="# SKIP $skip_all"
case "$test_count" in
0)
say "1..$test_count${skip_all:+ $skip_all}"
;;
*)
test -z "$skip_all" ||
say_color warn "$skip_all"
say "1..$test_count"
;;
esac
fi
if test -z "$debug"
then
test -d "$TRASH_DIRECTORY" ||
error "Tests passed but trash directory already removed before test cleanup; aborting"
cd "$TRASH_DIRECTORY/.." &&
rm -fr "$TRASH_DIRECTORY" || {
# try again in a bit
sleep 5;
rm -fr "$TRASH_DIRECTORY"
} ||
error "Tests passed but test cleanup failed; aborting"
fi
Introduce a performance testing framework This introduces a performance testing framework under t/perf/. It tries to be as close to the test-lib.sh infrastructure as possible, and thus should be easy to get used to for git developers. The following points were considered for the implementation: 1. You usually want to compare arbitrary revisions/build trees against each other. They may not have the performance test under consideration, or even the perf-lib.sh infrastructure. To cope with this, the 'run' script lets you specify arbitrary build dirs and revisions. It even automatically builds the revisions if it doesn't have them at hand yet. 2. Usually you would not want to run all tests. It would take too long anyway. The 'run' script lets you specify which tests to run; or you can also do it manually. There is a Makefile for discoverability and 'make clean', but it is not meant for real-world use. 3. Creating test repos from scratch in every test is extremely time-consuming, and shipping or downloading such large/weird repos is out of the question. We leave this decision to the user. Two different sizes of test repos can be configured, and the scripts just copy one or more of those (using hardlinks for the object store). By default it tries to use the build tree's git.git repository. This is fairly fast and versatile. Using a copy instead of a clone preserves many properties that the user may want to test for, such as lots of loose objects, unpacked refs, etc. Signed-off-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-17 11:25:09 +01:00
test_at_end_hook_
exit 0 ;;
*)
if test $test_external_has_tap -eq 0
then
say_color error "# failed $test_failure among $msg"
say "1..$test_count"
fi
test-lib: Adjust output to be valid TAP format TAP, the Test Anything Protocol, is a simple text-based interface between testing modules in a test harness. test-lib.sh's output was already very close to being valid TAP. This change brings it all the way there. Before: $ ./t0005-signals.sh * ok 1: sigchain works * passed all 1 test(s) And after: $ ./t0005-signals.sh ok 1 - sigchain works # passed all 1 test(s) 1..1 The advantage of using TAP is that any program that reads the format (a "test harness") can run the tests. The most popular of these is the prove(1) utility that comes with Perl. It can run tests in parallel, display colored output, format the output to console, file, HTML etc., and much more. An example: $ prove ./t0005-signals.sh ./t0005-signals.sh .. ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.01 cusr 0.02 csys = 0.06 CPU) Result: PASS prove(1) gives you human readable output without being too verbose. Running the test suite in parallel with `make test -j15` produces a flood of text. Running them with `prove -j 15 ./t[0-9]*.sh` makes it easy to follow what's going on. All this patch does is re-arrange the output a bit so that it conforms with the TAP spec, everything that the test suite did before continues to work. That includes aggregating results in t/test-results/, the --verbose, --debug and other options for tests, and the test color output. TAP harnesses ignore everything that they don't know about, so running the tests with --verbose works: $ prove ./t0005-signals.sh :: --verbose --debug ./t0005-signals.sh .. Terminated ./t0005-signals.sh .. ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.01 cusr 0.01 csys = 0.05 CPU) Result: PASS Just supply the -v option to prove itself to get all the verbose output that it suppresses: $ prove -v ./t0005-signals.sh :: --verbose --debug ./t0005-signals.sh .. Initialized empty Git repository in /home/avar/g/git/t/trash directory.t0005-signals/.git/ expecting success: test-sigchain >actual case "$?" in 143) true ;; # POSIX w/ SIGTERM=15 3) true ;; # Windows *) false ;; esac && test_cmp expect actual Terminated ok 1 - sigchain works # passed all 1 test(s) 1..1 ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.01 cusr 0.01 csys = 0.04 CPU) Result: PASS As a further example, consider this test script that uses a lot of test-lib.sh features by Jakub Narebski: #!/bin/sh test_description='this is a sample test. This test is here to see various test outputs.' . ./test-lib.sh say 'diagnostic message' test_expect_success 'true test' 'true' test_expect_success 'false test' 'false' test_expect_failure 'true test (todo)' 'true' test_expect_failure 'false test (todo)' 'false' test_debug 'echo "debug message"' test_done The output of that was previously: * diagnostic message # yellow * ok 1: true test * FAIL 2: false test # bold red false * FIXED 3: true test (todo) * still broken 4: false test (todo) # bold green * fixed 1 known breakage(s) # green * still have 1 known breakage(s) # bold red * failed 1 among remaining 3 test(s) # bold red But is now: diagnostic message # yellow ok 1 - true test not ok - 2 false test # bold red # false ok 3 - true test (todo) # TODO known breakage not ok 4 - false test (todo) # TODO known breakage # bold green # fixed 1 known breakage(s) # green # still have 1 known breakage(s) # bold red # failed 1 among remaining 3 test(s) # bold red 1..4 All the coloring is preserved when the test is run manually. Under prove(1) the test performs as expected, even with --debug and --verbose options: $ prove ./example.sh :: --debug --verbose ./example.sh .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/4 subtests (1 TODO test unexpectedly succeeded) Test Summary Report ------------------- ./example.sh (Wstat: 256 Tests: 4 Failed: 1) Failed test: 2 TODO passed: 3 Non-zero exit status: 1 Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.01 csys = 0.03 CPU) Result: FAIL The TAP harness itself doesn't get confused by the color output, they aren't used by test-lib.sh stdout isn't open to a terminal (test -t 1). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-06-24 23:52:12 +02:00
exit 1 ;;
esac
}
if test -n "$valgrind"
Add valgrind support in test scripts This patch adds the ability to use valgrind's memcheck tool to diagnose memory problems in Git while running the test scripts. It requires valgrind 3.4.0 or newer. It works by creating symlinks to a valgrind script, which have the same name as our Git binaries, and then putting that directory in front of the test script's PATH as well as set GIT_EXEC_PATH to that directory. Git scripts are symlinked from that directory directly. That way, Git binaries called by Git scripts are valgrinded, too. Valgrind can be used by specifying "GIT_TEST_OPTS=--valgrind" in the make invocation. Any invocation of git that finds any errors under valgrind will exit with failure code 126. Any valgrind output will go to the usual stderr channel for tests (i.e., /dev/null, unless -v has been specified). If you need to pass options to valgrind -- you might want to run another tool than memcheck, for example -- you can set the environment variable GIT_VALGRIND_OPTIONS. A few default suppressions are included, since libz seems to trigger quite a few false positives. We'll assume that libz works and that we can ignore any errors which are reported there. Note: it is safe to run the valgrind tests in parallel, as the links in t/valgrind/bin/ are created using proper locking. Initial patch and all the hard work by Jeff King. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-02-04 00:25:59 +01:00
then
make_symlink () {
test -h "$2" &&
test "$1" = "$(readlink "$2")" || {
# be super paranoid
if mkdir "$2".lock
then
rm -f "$2" &&
ln -s "$1" "$2" &&
rm -r "$2".lock
else
while test -d "$2".lock
do
say "Waiting for lock on $2."
sleep 1
done
fi
}
}
make_valgrind_symlink () {
# handle only executables, unless they are shell libraries that
remove #!interpreter line from shell libraries In a shell snippet meant to be sourced by other shell scripts, an opening #! line does more harm than good. The harm: - When the shell library is sourced, the interpreter and options from the #! line are not used. Specifying a particular shell can confuse the reader into thinking it is safe for the shell library to rely on idiosyncrasies of that shell. - Using #! instead of a plain comment drops a helpful visual clue that this is a shell library and not a self-contained script. - Tools such as lintian can use a #! line to tell when an installation script has failed by forgetting to set a script executable. This check does not work if shell libraries also start with a #! line. The good: - Text editors notice the #! line and use it for syntax highlighting if you try to edit the installed scripts (without ".sh" suffix) in place. The use of the #! for file type detection is not needed because Git's shell libraries are meant to be edited in source form (with ".sh" suffix). Replace the opening #! lines with comments. This involves tweaking the test harness's valgrind support to find shell libraries by looking for "# " in the first line instead of "#!" (see v1.7.6-rc3~7, 2011-06-17). Suggested by Russ Allbery through lintian. Thanks to Jeff King and Clemens Buchacher for further analysis. Tested by searching for non-executable scripts with #! line: find . -name .git -prune -o -type f -not -executable | while read file do read line <"$file" case $line in '#!'*) echo "$file" ;; esac done The only remaining scripts found are templates for shell scripts (unimplemented.sh, wrap-for-bin.sh) and sample input used in tests (t/t4034/perl/{pre,post}). Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-11-25 22:03:52 +01:00
# need to be in the exec-path.
test -x "$1" ||
test "# " = "$(test_copy_bytes 2 <"$1")" ||
return;
Add valgrind support in test scripts This patch adds the ability to use valgrind's memcheck tool to diagnose memory problems in Git while running the test scripts. It requires valgrind 3.4.0 or newer. It works by creating symlinks to a valgrind script, which have the same name as our Git binaries, and then putting that directory in front of the test script's PATH as well as set GIT_EXEC_PATH to that directory. Git scripts are symlinked from that directory directly. That way, Git binaries called by Git scripts are valgrinded, too. Valgrind can be used by specifying "GIT_TEST_OPTS=--valgrind" in the make invocation. Any invocation of git that finds any errors under valgrind will exit with failure code 126. Any valgrind output will go to the usual stderr channel for tests (i.e., /dev/null, unless -v has been specified). If you need to pass options to valgrind -- you might want to run another tool than memcheck, for example -- you can set the environment variable GIT_VALGRIND_OPTIONS. A few default suppressions are included, since libz seems to trigger quite a few false positives. We'll assume that libz works and that we can ignore any errors which are reported there. Note: it is safe to run the valgrind tests in parallel, as the links in t/valgrind/bin/ are created using proper locking. Initial patch and all the hard work by Jeff King. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-02-04 00:25:59 +01:00
base=$(basename "$1")
case "$base" in
test-*)
symlink_target="$GIT_BUILD_DIR/t/helper/$base"
;;
*)
symlink_target="$GIT_BUILD_DIR/$base"
;;
esac
Add valgrind support in test scripts This patch adds the ability to use valgrind's memcheck tool to diagnose memory problems in Git while running the test scripts. It requires valgrind 3.4.0 or newer. It works by creating symlinks to a valgrind script, which have the same name as our Git binaries, and then putting that directory in front of the test script's PATH as well as set GIT_EXEC_PATH to that directory. Git scripts are symlinked from that directory directly. That way, Git binaries called by Git scripts are valgrinded, too. Valgrind can be used by specifying "GIT_TEST_OPTS=--valgrind" in the make invocation. Any invocation of git that finds any errors under valgrind will exit with failure code 126. Any valgrind output will go to the usual stderr channel for tests (i.e., /dev/null, unless -v has been specified). If you need to pass options to valgrind -- you might want to run another tool than memcheck, for example -- you can set the environment variable GIT_VALGRIND_OPTIONS. A few default suppressions are included, since libz seems to trigger quite a few false positives. We'll assume that libz works and that we can ignore any errors which are reported there. Note: it is safe to run the valgrind tests in parallel, as the links in t/valgrind/bin/ are created using proper locking. Initial patch and all the hard work by Jeff King. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-02-04 00:25:59 +01:00
# do not override scripts
if test -x "$symlink_target" &&
test ! -d "$symlink_target" &&
test "#!" != "$(test_copy_bytes 2 <"$symlink_target")"
Add valgrind support in test scripts This patch adds the ability to use valgrind's memcheck tool to diagnose memory problems in Git while running the test scripts. It requires valgrind 3.4.0 or newer. It works by creating symlinks to a valgrind script, which have the same name as our Git binaries, and then putting that directory in front of the test script's PATH as well as set GIT_EXEC_PATH to that directory. Git scripts are symlinked from that directory directly. That way, Git binaries called by Git scripts are valgrinded, too. Valgrind can be used by specifying "GIT_TEST_OPTS=--valgrind" in the make invocation. Any invocation of git that finds any errors under valgrind will exit with failure code 126. Any valgrind output will go to the usual stderr channel for tests (i.e., /dev/null, unless -v has been specified). If you need to pass options to valgrind -- you might want to run another tool than memcheck, for example -- you can set the environment variable GIT_VALGRIND_OPTIONS. A few default suppressions are included, since libz seems to trigger quite a few false positives. We'll assume that libz works and that we can ignore any errors which are reported there. Note: it is safe to run the valgrind tests in parallel, as the links in t/valgrind/bin/ are created using proper locking. Initial patch and all the hard work by Jeff King. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-02-04 00:25:59 +01:00
then
symlink_target=../valgrind.sh
fi
case "$base" in
*.sh|*.perl)
symlink_target=../unprocessed-script
esac
Add valgrind support in test scripts This patch adds the ability to use valgrind's memcheck tool to diagnose memory problems in Git while running the test scripts. It requires valgrind 3.4.0 or newer. It works by creating symlinks to a valgrind script, which have the same name as our Git binaries, and then putting that directory in front of the test script's PATH as well as set GIT_EXEC_PATH to that directory. Git scripts are symlinked from that directory directly. That way, Git binaries called by Git scripts are valgrinded, too. Valgrind can be used by specifying "GIT_TEST_OPTS=--valgrind" in the make invocation. Any invocation of git that finds any errors under valgrind will exit with failure code 126. Any valgrind output will go to the usual stderr channel for tests (i.e., /dev/null, unless -v has been specified). If you need to pass options to valgrind -- you might want to run another tool than memcheck, for example -- you can set the environment variable GIT_VALGRIND_OPTIONS. A few default suppressions are included, since libz seems to trigger quite a few false positives. We'll assume that libz works and that we can ignore any errors which are reported there. Note: it is safe to run the valgrind tests in parallel, as the links in t/valgrind/bin/ are created using proper locking. Initial patch and all the hard work by Jeff King. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-02-04 00:25:59 +01:00
# create the link, or replace it if it is out of date
make_symlink "$symlink_target" "$GIT_VALGRIND/bin/$base" || exit
}
# override all git executables in TEST_DIRECTORY/..
GIT_VALGRIND=$TEST_DIRECTORY/valgrind
mkdir -p "$GIT_VALGRIND"/bin
for file in $GIT_BUILD_DIR/git* $GIT_BUILD_DIR/t/helper/test-*
do
make_valgrind_symlink $file
done
# special-case the mergetools loadables
make_symlink "$GIT_BUILD_DIR"/mergetools "$GIT_VALGRIND/bin/mergetools"
OLDIFS=$IFS
IFS=:
for path in $PATH
do
ls "$path"/git-* 2> /dev/null |
while read file
do
make_valgrind_symlink "$file"
done
done
IFS=$OLDIFS
Add valgrind support in test scripts This patch adds the ability to use valgrind's memcheck tool to diagnose memory problems in Git while running the test scripts. It requires valgrind 3.4.0 or newer. It works by creating symlinks to a valgrind script, which have the same name as our Git binaries, and then putting that directory in front of the test script's PATH as well as set GIT_EXEC_PATH to that directory. Git scripts are symlinked from that directory directly. That way, Git binaries called by Git scripts are valgrinded, too. Valgrind can be used by specifying "GIT_TEST_OPTS=--valgrind" in the make invocation. Any invocation of git that finds any errors under valgrind will exit with failure code 126. Any valgrind output will go to the usual stderr channel for tests (i.e., /dev/null, unless -v has been specified). If you need to pass options to valgrind -- you might want to run another tool than memcheck, for example -- you can set the environment variable GIT_VALGRIND_OPTIONS. A few default suppressions are included, since libz seems to trigger quite a few false positives. We'll assume that libz works and that we can ignore any errors which are reported there. Note: it is safe to run the valgrind tests in parallel, as the links in t/valgrind/bin/ are created using proper locking. Initial patch and all the hard work by Jeff King. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-02-04 00:25:59 +01:00
PATH=$GIT_VALGRIND/bin:$PATH
GIT_EXEC_PATH=$GIT_VALGRIND/bin
export GIT_VALGRIND
GIT_VALGRIND_MODE="$valgrind"
export GIT_VALGRIND_MODE
GIT_VALGRIND_ENABLED=t
test -n "$valgrind_only" && GIT_VALGRIND_ENABLED=
export GIT_VALGRIND_ENABLED
elif test -n "$GIT_TEST_INSTALLED"
then
GIT_EXEC_PATH=$($GIT_TEST_INSTALLED/git --exec-path) ||
error "Cannot run git from $GIT_TEST_INSTALLED."
PATH=$GIT_TEST_INSTALLED:$GIT_BUILD_DIR/t/helper:$PATH
GIT_EXEC_PATH=${GIT_TEST_EXEC_PATH:-$GIT_EXEC_PATH}
else # normal case, use ../bin-wrappers only unless $with_dashes:
if test -n "$no_bin_wrappers"
then
with_dashes=t
else
git_bin_dir="$GIT_BUILD_DIR/bin-wrappers"
if ! test -x "$git_bin_dir/git"
then
if test -z "$with_dashes"
then
say "$git_bin_dir/git is not executable; using GIT_EXEC_PATH"
fi
with_dashes=t
fi
PATH="$git_bin_dir:$PATH"
fi
GIT_EXEC_PATH=$GIT_BUILD_DIR
if test -n "$with_dashes"
then
PATH="$GIT_BUILD_DIR:$GIT_BUILD_DIR/t/helper:$PATH"
fi
Add valgrind support in test scripts This patch adds the ability to use valgrind's memcheck tool to diagnose memory problems in Git while running the test scripts. It requires valgrind 3.4.0 or newer. It works by creating symlinks to a valgrind script, which have the same name as our Git binaries, and then putting that directory in front of the test script's PATH as well as set GIT_EXEC_PATH to that directory. Git scripts are symlinked from that directory directly. That way, Git binaries called by Git scripts are valgrinded, too. Valgrind can be used by specifying "GIT_TEST_OPTS=--valgrind" in the make invocation. Any invocation of git that finds any errors under valgrind will exit with failure code 126. Any valgrind output will go to the usual stderr channel for tests (i.e., /dev/null, unless -v has been specified). If you need to pass options to valgrind -- you might want to run another tool than memcheck, for example -- you can set the environment variable GIT_VALGRIND_OPTIONS. A few default suppressions are included, since libz seems to trigger quite a few false positives. We'll assume that libz works and that we can ignore any errors which are reported there. Note: it is safe to run the valgrind tests in parallel, as the links in t/valgrind/bin/ are created using proper locking. Initial patch and all the hard work by Jeff King. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-02-04 00:25:59 +01:00
fi
GIT_TEMPLATE_DIR="$GIT_BUILD_DIR"/templates/blt
GIT_CONFIG_NOSYSTEM=1
GIT_ATTR_NOSYSTEM=1
export PATH GIT_EXEC_PATH GIT_TEMPLATE_DIR GIT_CONFIG_NOSYSTEM GIT_ATTR_NOSYSTEM
if test -z "$GIT_TEST_CMP"
then
if test -n "$GIT_TEST_CMP_USE_COPIED_CONTEXT"
then
GIT_TEST_CMP="$DIFF -c"
else
GIT_TEST_CMP="$DIFF -u"
fi
fi
Makefile: replace perl/Makefile.PL with simple make rules Replace the perl/Makefile.PL and the fallback perl/Makefile used under NO_PERL_MAKEMAKER=NoThanks with a much simpler implementation heavily inspired by how the i18n infrastructure's build process works[1]. The reason for having the Makefile.PL in the first place is that it was initially[2] building a perl C binding to interface with libgit, this functionality, that was removed[3] before Git.pm ever made it to the master branch. We've since since started maintaining a fallback perl/Makefile, as MakeMaker wouldn't work on some platforms[4]. That's just the tip of the iceberg. We have the PM.stamp hack in the top-level Makefile[5] to detect whether we need to regenerate the perl/perl.mak, which I fixed just recently to deal with issues like the perl version changing from under us[6]. There is absolutely no reason for why this needs to be so complex anymore. All we're getting out of this elaborate Rube Goldberg machine was copying perl/* to perl/blib/* as we do a string-replacement on the *.pm files to hardcode @@LOCALEDIR@@ in the source, as well as pod2man-ing Git.pm & friends. So replace the whole thing with something that's pretty much a copy of how we generate po/build/**.mo from po/*.po, just with a small sed(1) command instead of msgfmt. As that's being done rename the files from *.pm to *.pmc just to indicate that they're generated (see "perldoc -f require"). While I'm at it, change the fallback for Error.pm from being something where we'll ship our own Error.pm if one doesn't exist at build time to one where we just use a Git::Error wrapper that'll always prefer the system-wide Error.pm, only falling back to our own copy if it really doesn't exist at runtime. It's now shipped as Git::FromCPAN::Error, making it easy to add other modules to Git::FromCPAN::* in the future if that's needed. Functional changes: * This will not always install into perl's idea of its global "installsitelib". This only potentially matters for packagers that need to expose Git.pm for non-git use, and as explained in the INSTALL file there's a trivial workaround. * The scripts themselves will 'use lib' the target directory, but if INSTLIBDIR is set it overrides it. It doesn't have to be this way, it could be set in addition to INSTLIBDIR, but my reading of [7] is that this is the desired behavior. * We don't build man pages for all of the perl modules as we used to, only Git(3pm). As discussed on-list[8] that we were building installed manpages for purely internal APIs like Git::I18N or private-Error.pm was always a bug anyway, and all the Git::SVN::* ones say they're internal APIs. There are apparently external users of Git.pm, but I don't expect there to be any of the others. As a side-effect of these general changes the perl documentation now only installed by install-{doc,man}, not a mere "install" as before. 1. 5e9637c629 ("i18n: add infrastructure for translating Git with gettext", 2011-11-18) 2. b1edc53d06 ("Introduce Git.pm (v4)", 2006-06-24) 3. 18b0fc1ce1 ("Git.pm: Kill Git.xs for now", 2006-09-23) 4. f848718a69 ("Make perl/ build procedure ActiveState friendly.", 2006-12-04) 5. ee9be06770 ("perl: detect new files in MakeMaker builds", 2012-07-27) 6. c59c4939c2 ("perl: regenerate perl.mak if perl -V changes", 2017-03-29) 7. 0386dd37b1 ("Makefile: add PERLLIB_EXTRA variable that adds to default perl path", 2013-11-15) 8. 87bmjjv1pu.fsf@evledraar.booking.com ("Re: [PATCH] Makefile: replace perl/Makefile.PL with simple make rules" Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-12-10 22:13:33 +01:00
GITPERLLIB="$GIT_BUILD_DIR"/perl/build/lib
export GITPERLLIB
test -d "$GIT_BUILD_DIR"/templates/blt || {
error "You haven't built things yet, have you?"
}
if ! test -x "$GIT_BUILD_DIR"/t/helper/test-tool$X
then
echo >&2 'You need to build test-tool:'
echo >&2 'Run "make t/helper/test-tool" in the source (toplevel) directory'
exit 1
fi
# Test repository
rm -fr "$TRASH_DIRECTORY" || {
GIT_EXIT_OK=t
echo >&5 "FATAL: Cannot prepare test area"
exit 1
}
HOME="$TRASH_DIRECTORY"
signed push: teach smart-HTTP to pass "git push --signed" around The "--signed" option received by "git push" is first passed to the transport layer, which the native transport directly uses to notice that a push certificate needs to be sent. When the transport-helper is involved, however, the option needs to be told to the helper with set_helper_option(), and the helper needs to take necessary action. For the smart-HTTP helper, the "necessary action" involves spawning the "git send-pack" subprocess with the "--signed" option. Once the above all gets wired in, the smart-HTTP transport now can use the push certificate mechanism to authenticate its pushes. Add a test that is modeled after tests for the native transport in t5534-push-signed.sh to t5541-http-push-smart.sh. Update the test Apache configuration to pass GNUPGHOME environment variable through. As PassEnv would trigger warnings for an environment variable that is not set, export it from test-lib.sh set to a harmless value when GnuPG is not being used in the tests. Note that the added test is deliberately loose and does not check the nonce in this step. This is because the stateless RPC mode is inevitably flaky and a nonce that comes back in the actual push processing is one issued by a different process; if the two interactions with the server crossed a second boundary, the nonces will not match and such a check will fail. A later patch in the series will work around this shortcoming. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-09-15 23:59:00 +02:00
GNUPGHOME="$HOME/gnupg-home-not-used"
export HOME GNUPGHOME
if test -z "$TEST_NO_CREATE_REPO"
then
test_create_repo "$TRASH_DIRECTORY"
Introduce a performance testing framework This introduces a performance testing framework under t/perf/. It tries to be as close to the test-lib.sh infrastructure as possible, and thus should be easy to get used to for git developers. The following points were considered for the implementation: 1. You usually want to compare arbitrary revisions/build trees against each other. They may not have the performance test under consideration, or even the perf-lib.sh infrastructure. To cope with this, the 'run' script lets you specify arbitrary build dirs and revisions. It even automatically builds the revisions if it doesn't have them at hand yet. 2. Usually you would not want to run all tests. It would take too long anyway. The 'run' script lets you specify which tests to run; or you can also do it manually. There is a Makefile for discoverability and 'make clean', but it is not meant for real-world use. 3. Creating test repos from scratch in every test is extremely time-consuming, and shipping or downloading such large/weird repos is out of the question. We leave this decision to the user. Two different sizes of test repos can be configured, and the scripts just copy one or more of those (using hardlinks for the object store). By default it tries to use the build tree's git.git repository. This is fairly fast and versatile. Using a copy instead of a clone preserves many properties that the user may want to test for, such as lots of loose objects, unpacked refs, etc. Signed-off-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-17 11:25:09 +01:00
else
mkdir -p "$TRASH_DIRECTORY"
Introduce a performance testing framework This introduces a performance testing framework under t/perf/. It tries to be as close to the test-lib.sh infrastructure as possible, and thus should be easy to get used to for git developers. The following points were considered for the implementation: 1. You usually want to compare arbitrary revisions/build trees against each other. They may not have the performance test under consideration, or even the perf-lib.sh infrastructure. To cope with this, the 'run' script lets you specify arbitrary build dirs and revisions. It even automatically builds the revisions if it doesn't have them at hand yet. 2. Usually you would not want to run all tests. It would take too long anyway. The 'run' script lets you specify which tests to run; or you can also do it manually. There is a Makefile for discoverability and 'make clean', but it is not meant for real-world use. 3. Creating test repos from scratch in every test is extremely time-consuming, and shipping or downloading such large/weird repos is out of the question. We leave this decision to the user. Two different sizes of test repos can be configured, and the scripts just copy one or more of those (using hardlinks for the object store). By default it tries to use the build tree's git.git repository. This is fairly fast and versatile. Using a copy instead of a clone preserves many properties that the user may want to test for, such as lots of loose objects, unpacked refs, etc. Signed-off-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-17 11:25:09 +01:00
fi
# Use -P to resolve symlinks in our working directory so that the cwd
# in subprocesses like git equals our $PWD (for pathname comparisons).
cd -P "$TRASH_DIRECTORY" || exit 1
this_test=${0##*/}
this_test=${this_test%%-*}
if match_pattern_list "$this_test" $GIT_SKIP_TESTS
then
say_color info >&3 "skipping test $this_test altogether"
skip_all="skip all tests in $this_test"
test_done
fi
if test -n "$write_junit_xml"
then
junit_xml_dir="$TEST_OUTPUT_DIRECTORY/out"
mkdir -p "$junit_xml_dir"
junit_xml_base=${0##*/}
junit_xml_path="$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml"
junit_attrs="name=\"${junit_xml_base%.sh}\""
junit_attrs="$junit_attrs timestamp=\"$(TZ=UTC \
date +%Y-%m-%dT%H:%M:%S)\""
write_junit_xml --truncate "<testsuites>" " <testsuite $junit_attrs>"
junit_suite_start=$(test-tool date getnanos)
tests: include detailed trace logs with --write-junit-xml upon failure The JUnit XML format lends itself to be presented in a powerful UI, where you can drill down to the information you are interested in very quickly. For test failures, this usually means that you want to see the detailed trace of the failing tests. With Travis CI, we passed the `--verbose-log` option to get those traces. However, that seems excessive, as we do not need/use the logs in almost all of those cases: only when a test fails do we have a way to include the trace. So let's do something different when using Azure DevOps: let's run all the tests with `--quiet` first, and only if a failure is encountered, try to trace the commands as they are executed. Of course, we cannot turn on `--verbose-log` after the fact. So let's just re-run the test with all the same options, adding `--verbose-log`. And then munging the output file into the JUnit XML on the fly. Note: there is an off chance that re-running the test in verbose mode "fixes" the failures (and this does happen from time to time!). That is a possibility we should be able to live with. Ideally, we would label this as "Passed upon rerun", and Azure Pipelines even know about that outcome, but it is not available when using the JUnit XML format for now: https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:34 +01:00
if test -n "$GIT_TEST_TEE_OUTPUT_FILE"
then
GIT_TEST_TEE_OFFSET=0
fi
fi
# Convenience
# A regexp to match 5, 35 and 40 hexdigits
_x05='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
_x35="$_x05$_x05$_x05$_x05$_x05$_x05$_x05"
_x40="$_x35$_x05"
test_oid_init
ZERO_OID=$(test_oid zero)
OID_REGEX=$(echo $ZERO_OID | sed -e 's/0/[0-9a-f]/g')
OIDPATH_REGEX=$(test_oid_to_path $ZERO_OID | sed -e 's/0/[0-9a-f]/g')
EMPTY_TREE=$(test_oid empty_tree)
EMPTY_BLOB=$(test_oid empty_blob)
_z40=$ZERO_OID
# Provide an implementation of the 'yes' utility; the upper bound
# limit is there to help Windows that cannot stop this loop from
# wasting cycles when the downstream stops reading, so do not be
# tempted to turn it into an infinite loop. cf. 6129c930 ("test-lib:
# limit the output of the yes utility", 2016-02-02)
yes () {
if test $# = 0
then
y=y
else
y="$*"
fi
i=0
while test $i -lt 99
do
echo "$y"
i=$(($i+1))
done
}
# The GIT_TEST_FAIL_PREREQS code hooks into test_set_prereq(), and
# thus needs to be set up really early, and set an internal variable
# for convenience so the hot test_set_prereq() codepath doesn't need
tests: add 'test_bool_env' to catch non-bool GIT_TEST_* values Since 3b072c577b (tests: replace test_tristate with "git env--helper", 2019-06-21) we get the normalized bool values of various GIT_TEST_* environment variables via 'git env--helper'. Now, while the 'git env--helper' command itself does catch invalid values in the environment variable or in the given --default and exits with error (exit code 128 or 129, respectively), it's invoked in conditions like 'if ! git env--helper ...', which means that all invalid bool values are interpreted the same as the ordinary 'false' (exit code 1). This has led to inadvertently skipped httpd tests in our CI builds for a couple of weeks, see 3960290675 (ci: restore running httpd tests, 2019-09-06). Let's be more careful about what the test suite accepts as bool values in GIT_TEST_* environment variables, and error out loud and clear on invalid values instead of simply skipping tests. Add the 'test_bool_env' helper function to encapsulate the invocation of 'git env--helper' and the verification of its exit code, and replace all invocations of that command in our test framework and test suite with a call to this new helper (except in 't0017-env-helper.sh', of course). $ GIT_TEST_GIT_DAEMON=YesPlease ./t5570-git-daemon.sh fatal: bad numeric config value 'YesPlease' for 'GIT_TEST_GIT_DAEMON': invalid unit error: test_bool_env requires bool values both for $GIT_TEST_GIT_DAEMON and for the default fallback Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-22 14:14:36 +01:00
# to call "git env--helper" (via test_bool_env). Only do that work
# if needed by seeing if GIT_TEST_FAIL_PREREQS is set at all.
GIT_TEST_FAIL_PREREQS_INTERNAL=
if test -n "$GIT_TEST_FAIL_PREREQS"
then
tests: add 'test_bool_env' to catch non-bool GIT_TEST_* values Since 3b072c577b (tests: replace test_tristate with "git env--helper", 2019-06-21) we get the normalized bool values of various GIT_TEST_* environment variables via 'git env--helper'. Now, while the 'git env--helper' command itself does catch invalid values in the environment variable or in the given --default and exits with error (exit code 128 or 129, respectively), it's invoked in conditions like 'if ! git env--helper ...', which means that all invalid bool values are interpreted the same as the ordinary 'false' (exit code 1). This has led to inadvertently skipped httpd tests in our CI builds for a couple of weeks, see 3960290675 (ci: restore running httpd tests, 2019-09-06). Let's be more careful about what the test suite accepts as bool values in GIT_TEST_* environment variables, and error out loud and clear on invalid values instead of simply skipping tests. Add the 'test_bool_env' helper function to encapsulate the invocation of 'git env--helper' and the verification of its exit code, and replace all invocations of that command in our test framework and test suite with a call to this new helper (except in 't0017-env-helper.sh', of course). $ GIT_TEST_GIT_DAEMON=YesPlease ./t5570-git-daemon.sh fatal: bad numeric config value 'YesPlease' for 'GIT_TEST_GIT_DAEMON': invalid unit error: test_bool_env requires bool values both for $GIT_TEST_GIT_DAEMON and for the default fallback Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-22 14:14:36 +01:00
if test_bool_env GIT_TEST_FAIL_PREREQS false
then
GIT_TEST_FAIL_PREREQS_INTERNAL=true
test_set_prereq FAIL_PREREQS
fi
else
test_lazy_prereq FAIL_PREREQS '
tests: add 'test_bool_env' to catch non-bool GIT_TEST_* values Since 3b072c577b (tests: replace test_tristate with "git env--helper", 2019-06-21) we get the normalized bool values of various GIT_TEST_* environment variables via 'git env--helper'. Now, while the 'git env--helper' command itself does catch invalid values in the environment variable or in the given --default and exits with error (exit code 128 or 129, respectively), it's invoked in conditions like 'if ! git env--helper ...', which means that all invalid bool values are interpreted the same as the ordinary 'false' (exit code 1). This has led to inadvertently skipped httpd tests in our CI builds for a couple of weeks, see 3960290675 (ci: restore running httpd tests, 2019-09-06). Let's be more careful about what the test suite accepts as bool values in GIT_TEST_* environment variables, and error out loud and clear on invalid values instead of simply skipping tests. Add the 'test_bool_env' helper function to encapsulate the invocation of 'git env--helper' and the verification of its exit code, and replace all invocations of that command in our test framework and test suite with a call to this new helper (except in 't0017-env-helper.sh', of course). $ GIT_TEST_GIT_DAEMON=YesPlease ./t5570-git-daemon.sh fatal: bad numeric config value 'YesPlease' for 'GIT_TEST_GIT_DAEMON': invalid unit error: test_bool_env requires bool values both for $GIT_TEST_GIT_DAEMON and for the default fallback Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-22 14:14:36 +01:00
test_bool_env GIT_TEST_FAIL_PREREQS false
'
fi
# Fix some commands on Windows, and other OS-specific things
uname_s=$(uname -s)
case $uname_s in
*MINGW*)
# Windows has its own (incompatible) sort and find
sort () {
/usr/bin/sort "$@"
}
find () {
/usr/bin/find "$@"
}
# git sees Windows-style pwd
pwd () {
builtin pwd -W
}
# no POSIX permissions
# backslashes in pathspec are converted to '/'
# exec does not inherit the PID
test_set_prereq MINGW
test_set_prereq NATIVE_CRLF
test_set_prereq SED_STRIPS_CR
test_set_prereq GREP_STRIPS_CR
cygwin: disallow backslashes in file names The backslash character is not a valid part of a file name on Windows. If, in Windows, Git attempts to write a file that has a backslash character in the filename, it will be incorrectly interpreted as a directory separator. This caused CVE-2019-1354 in MinGW, as this behaviour can be manipulated to cause the checkout to write to files it ought not write to, such as adding code to the .git/hooks directory. This was fixed by e1d911dd4c (mingw: disallow backslash characters in tree objects' file names, 2019-09-12). However, the vulnerability also exists in Cygwin: while Cygwin mostly provides a POSIX-like path system, it will still interpret a backslash as a directory separator. To avoid this vulnerability, CVE-2021-29468, extend the previous fix to also apply to Cygwin. Similarly, extend the test case added by the previous version of the commit. The test suite doesn't have an easy way to say "run this test if in MinGW or Cygwin", so add a new test prerequisite that covers both. As well as checking behaviour in the presence of paths containing backslashes, the existing test also checks behaviour in the presence of paths that differ only by the presence of a trailing ".". MinGW follows normal Windows application behaviour and treats them as the same path, but Cygwin more closely emulates *nix systems (at the expense of compatibility with native Windows applications) and will create and distinguish between such paths. Gate the relevant bit of that test accordingly. Reported-by: RyotaK <security@ryotak.me> Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Adam Dinwoodie <adam@dinwoodie.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-29 22:11:44 +02:00
test_set_prereq WINDOWS
GIT_TEST_CMP=mingw_test_cmp
;;
*CYGWIN*)
test_set_prereq POSIXPERM
test_set_prereq EXECKEEPSPID
test_set_prereq CYGWIN
test_set_prereq SED_STRIPS_CR
test_set_prereq GREP_STRIPS_CR
cygwin: disallow backslashes in file names The backslash character is not a valid part of a file name on Windows. If, in Windows, Git attempts to write a file that has a backslash character in the filename, it will be incorrectly interpreted as a directory separator. This caused CVE-2019-1354 in MinGW, as this behaviour can be manipulated to cause the checkout to write to files it ought not write to, such as adding code to the .git/hooks directory. This was fixed by e1d911dd4c (mingw: disallow backslash characters in tree objects' file names, 2019-09-12). However, the vulnerability also exists in Cygwin: while Cygwin mostly provides a POSIX-like path system, it will still interpret a backslash as a directory separator. To avoid this vulnerability, CVE-2021-29468, extend the previous fix to also apply to Cygwin. Similarly, extend the test case added by the previous version of the commit. The test suite doesn't have an easy way to say "run this test if in MinGW or Cygwin", so add a new test prerequisite that covers both. As well as checking behaviour in the presence of paths containing backslashes, the existing test also checks behaviour in the presence of paths that differ only by the presence of a trailing ".". MinGW follows normal Windows application behaviour and treats them as the same path, but Cygwin more closely emulates *nix systems (at the expense of compatibility with native Windows applications) and will create and distinguish between such paths. Gate the relevant bit of that test accordingly. Reported-by: RyotaK <security@ryotak.me> Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Adam Dinwoodie <adam@dinwoodie.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-29 22:11:44 +02:00
test_set_prereq WINDOWS
;;
*)
test_set_prereq POSIXPERM
test_set_prereq BSLASHPSPEC
test_set_prereq EXECKEEPSPID
;;
esac
# Detect arches where a few things don't work
uname_m=$(uname -m)
case $uname_m in
parisc* | hppa*)
test_set_prereq HPPA
;;
esac
( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1
test -z "$NO_PERL" && test_set_prereq PERL
test -z "$NO_PTHREADS" && test_set_prereq PTHREADS
test -z "$NO_PYTHON" && test_set_prereq PYTHON
test -n "$USE_LIBPCRE2" && test_set_prereq PCRE
test -n "$USE_LIBPCRE2" && test_set_prereq LIBPCRE2
i18n: add infrastructure for translating Git with gettext Change the skeleton implementation of i18n in Git to one that can show localized strings to users for our C, Shell and Perl programs using either GNU libintl or the Solaris gettext implementation. This new internationalization support is enabled by default. If gettext isn't available, or if Git is compiled with NO_GETTEXT=YesPlease, Git falls back on its current behavior of showing interface messages in English. When using the autoconf script we'll auto-detect if the gettext libraries are installed and act appropriately. This change is somewhat large because as well as adding a C, Shell and Perl i18n interface we're adding a lot of tests for them, and for those tests to work we need a skeleton PO file to actually test translations. A minimal Icelandic translation is included for this purpose. Icelandic includes multi-byte characters which makes it easy to test various edge cases, and it's a language I happen to understand. The rest of the commit message goes into detail about various sub-parts of this commit. = Installation Gettext .mo files will be installed and looked for in the standard $(prefix)/share/locale path. GIT_TEXTDOMAINDIR can also be set to override that, but that's only intended to be used to test Git itself. = Perl Perl code that's to be localized should use the new Git::I18n module. It imports a __ function into the caller's package by default. Instead of using the high level Locale::TextDomain interface I've opted to use the low-level (equivalent to the C interface) Locale::Messages module, which Locale::TextDomain itself uses. Locale::TextDomain does a lot of redundant work we don't need, and some of it would potentially introduce bugs. It tries to set the $TEXTDOMAIN based on package of the caller, and has its own hardcoded paths where it'll search for messages. I found it easier just to completely avoid it rather than try to circumvent its behavior. In any case, this is an issue wholly internal Git::I18N. Its guts can be changed later if that's deemed necessary. See <AANLkTilYD_NyIZMyj9dHtVk-ylVBfvyxpCC7982LWnVd@mail.gmail.com> for a further elaboration on this topic. = Shell Shell code that's to be localized should use the git-sh-i18n library. It's basically just a wrapper for the system's gettext.sh. If gettext.sh isn't available we'll fall back on gettext(1) if it's available. The latter is available without the former on Solaris, which has its own non-GNU gettext implementation. We also need to emulate eval_gettext() there. If neither are present we'll use a dumb printf(1) fall-through wrapper. = About libcharset.h and langinfo.h We use libcharset to query the character set of the current locale if it's available. I.e. we'll use it instead of nl_langinfo if HAVE_LIBCHARSET_H is set. The GNU gettext manual recommends using langinfo.h's nl_langinfo(CODESET) to acquire the current character set, but on systems that have libcharset.h's locale_charset() using the latter is either saner, or the only option on those systems. GNU and Solaris have a nl_langinfo(CODESET), FreeBSD can use either, but MinGW and some others need to use libcharset.h's locale_charset() instead. =Credits This patch is based on work by Jeff Epler <jepler@unpythonic.net> who did the initial Makefile / C work, and a lot of comments from the Git mailing list, including Jonathan Nieder, Jakub Narebski, Johannes Sixt, Erik Faye-Lund, Peter Krefting, Junio C Hamano, Thomas Rast and others. [jc: squashed a small Makefile fix from Ramsay] Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-11-18 00:14:42 +01:00
test -z "$NO_GETTEXT" && test_set_prereq GETTEXT
if test -z "$GIT_TEST_CHECK_CACHE_TREE"
then
GIT_TEST_CHECK_CACHE_TREE=true
export GIT_TEST_CHECK_CACHE_TREE
fi
test_lazy_prereq PIPE '
# test whether the filesystem supports FIFOs
test_have_prereq !MINGW,!CYGWIN &&
rm -f testfifo && mkfifo testfifo
'
test: allow prerequisite to be evaluated lazily The test prerequisite mechanism is a useful way to allow some tests in a test script to be skipped in environments that do not support certain features (e.g. it is pointless to attempt checking how well symbolic links are handled by Git on filesystems that do not support them). It is OK for commonly used prerequisites to be always tested during start-up of a test script by having a codeblock that tests a feature and calls test_set_prereq, but for an uncommon feature, forcing 90% of scripts to pay the same probing overhead for prerequisite they do not care about is wasteful. Introduce a mechanism to probe the prerequiste lazily. Changes are: - test_lazy_prereq () function, which takes the name of the prerequisite it probes and the script to probe for it, is added. This only registers the name of the prerequiste that can be lazily probed and the script to eval (without running). - test_have_prereq() function (which is used by test_expect_success and also can be called directly by test scripts) learns to look at the list of prerequisites that can be lazily probed, and the prerequisites that have already been probed that way. When asked for a prerequiste that can be but haven't been probed, the script registered with an earlier call to test_lazy_prereq is evaluated and the prerequisite is set. - test_run_lazy_prereq_() function is a helper to run the probe script with the same kind of sandbox as regular tests, helped by Jeff King. Update the codeblock to probe and set SYMLINKS prerequisite using the new mechanism as an example. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-07-27 00:50:45 +02:00
test_lazy_prereq SYMLINKS '
# test whether the filesystem supports symbolic links
ln -s x y && test -h y
'
test_lazy_prereq FILEMODE '
test "$(git config --bool core.filemode)" = true
'
test_lazy_prereq CASE_INSENSITIVE_FS '
echo good >CamelCase &&
echo bad >camelcase &&
test "$(cat CamelCase)" != good
'
test_lazy_prereq FUNNYNAMES '
test_have_prereq !MINGW &&
touch -- \
"FUNNYNAMES tab embedded" \
"FUNNYNAMES \"quote embedded\"" \
"FUNNYNAMES newline
embedded" 2>/dev/null &&
rm -- \
"FUNNYNAMES tab embedded" \
"FUNNYNAMES \"quote embedded\"" \
"FUNNYNAMES newline
embedded" 2>/dev/null
'
test_lazy_prereq UTF8_NFD_TO_NFC '
# check whether FS converts nfd unicode to nfc
auml=$(printf "\303\244")
aumlcdiar=$(printf "\141\314\210")
>"$auml" &&
test -f "$aumlcdiar"
'
test_lazy_prereq AUTOIDENT '
sane_unset GIT_AUTHOR_NAME &&
sane_unset GIT_AUTHOR_EMAIL &&
git var GIT_AUTHOR_IDENT
'
test_lazy_prereq EXPENSIVE '
test -n "$GIT_TEST_LONG"
'
test_lazy_prereq EXPENSIVE_ON_WINDOWS '
test_have_prereq EXPENSIVE || test_have_prereq !MINGW,!CYGWIN
'
test_lazy_prereq USR_BIN_TIME '
test -x /usr/bin/time
'
test_lazy_prereq NOT_ROOT '
uid=$(id -u) &&
test "$uid" != 0
'
test_lazy_prereq JGIT '
jgit --version
'
# SANITY is about "can you correctly predict what the filesystem would
# do by only looking at the permission bits of the files and
# directories?" A typical example of !SANITY is running the test
# suite as root, where a test may expect "chmod -r file && cat file"
# to fail because file is supposed to be unreadable after a successful
# chmod. In an environment (i.e. combination of what filesystem is
# being used and who is running the tests) that lacks SANITY, you may
# be able to delete or create a file when the containing directory
# doesn't have write permissions, or access a file even if the
# containing directory doesn't have read or execute permissions.
test_lazy_prereq SANITY '
mkdir SANETESTD.1 SANETESTD.2 &&
chmod +w SANETESTD.1 SANETESTD.2 &&
>SANETESTD.1/x 2>SANETESTD.2/x &&
chmod -w SANETESTD.1 &&
chmod -r SANETESTD.1/x &&
chmod -rx SANETESTD.2 ||
tests: send "bug in the test script" errors to the script's stderr Some of the functions in our test library check that they were invoked properly with conditions like this: test "$#" = 2 || error "bug in the test script: not 2 parameters to test-expect-success" If this particular condition is triggered, then 'error' will abort the whole test script with a bold red error message [1] right away. However, under certain circumstances the test script will be aborted completely silently, namely if: - a similar condition in a test helper function like 'test_line_count' is triggered, - which is invoked from the test script's "main" shell [2], - and the test script is run manually (i.e. './t1234-foo.sh' as opposed to 'make t1234-foo.sh' or 'make test') [3] - and without the '--verbose' option, because the error message is printed from within 'test_eval_', where standard output is redirected either to /dev/null or to a log file. The only indication that something is wrong is that not all tests in the script are executed and at the end of the test script's output there is no "# passed all N tests" message, which are subtle and can easily go unnoticed, as I had to experience myself. Send these "bug in the test script" error messages directly to the test scripts standard error and thus to the terminal, so those bugs will be much harder to overlook. Instead of updating all ~20 such 'error' calls with a redirection, let's add a BUG() function to 'test-lib.sh', wrapping an 'error' call with the proper redirection and also including the common prefix of those error messages, and convert all those call sites [4] to use this new BUG() function instead. [1] That particular error message from 'test_expect_success' is printed in color only when running with or without '--verbose'; with '--tee' or '--verbose-log' the error is printed without color, but it is printed to the terminal nonetheless. [2] If such a condition is triggered in a subshell of a test, then 'error' won't be able to abort the whole test script, but only the subshell, which in turn causes the test to fail in the usual way, indicating loudly and clearly that something is wrong. [3] Well, 'error' aborts the test script the same way when run manually or by 'make' or 'prove', but both 'make' and 'prove' pay attention to the test script's exit status, and even a silently aborted test script would then trigger those tools' usual noticable error messages. [4] Strictly speaking, not all those 'error' calls need that redirection to send their output to the terminal, see e.g. 'test_expect_success' in the opening example, but I think it's better to be consistent. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-19 14:13:26 +01:00
BUG "cannot prepare SANETESTD"
! test -r SANETESTD.1/x &&
! rm SANETESTD.1/x && ! test -f SANETESTD.2/x
status=$?
chmod +rwx SANETESTD.1 SANETESTD.2 &&
rm -rf SANETESTD.1 SANETESTD.2 ||
tests: send "bug in the test script" errors to the script's stderr Some of the functions in our test library check that they were invoked properly with conditions like this: test "$#" = 2 || error "bug in the test script: not 2 parameters to test-expect-success" If this particular condition is triggered, then 'error' will abort the whole test script with a bold red error message [1] right away. However, under certain circumstances the test script will be aborted completely silently, namely if: - a similar condition in a test helper function like 'test_line_count' is triggered, - which is invoked from the test script's "main" shell [2], - and the test script is run manually (i.e. './t1234-foo.sh' as opposed to 'make t1234-foo.sh' or 'make test') [3] - and without the '--verbose' option, because the error message is printed from within 'test_eval_', where standard output is redirected either to /dev/null or to a log file. The only indication that something is wrong is that not all tests in the script are executed and at the end of the test script's output there is no "# passed all N tests" message, which are subtle and can easily go unnoticed, as I had to experience myself. Send these "bug in the test script" error messages directly to the test scripts standard error and thus to the terminal, so those bugs will be much harder to overlook. Instead of updating all ~20 such 'error' calls with a redirection, let's add a BUG() function to 'test-lib.sh', wrapping an 'error' call with the proper redirection and also including the common prefix of those error messages, and convert all those call sites [4] to use this new BUG() function instead. [1] That particular error message from 'test_expect_success' is printed in color only when running with or without '--verbose'; with '--tee' or '--verbose-log' the error is printed without color, but it is printed to the terminal nonetheless. [2] If such a condition is triggered in a subshell of a test, then 'error' won't be able to abort the whole test script, but only the subshell, which in turn causes the test to fail in the usual way, indicating loudly and clearly that something is wrong. [3] Well, 'error' aborts the test script the same way when run manually or by 'make' or 'prove', but both 'make' and 'prove' pay attention to the test script's exit status, and even a silently aborted test script would then trigger those tools' usual noticable error messages. [4] Strictly speaking, not all those 'error' calls need that redirection to send their output to the terminal, see e.g. 'test_expect_success' in the opening example, but I think it's better to be consistent. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-19 14:13:26 +01:00
BUG "cannot clean SANETESTD"
return $status
'
test FreeBSD != $uname_s || GIT_UNZIP=${GIT_UNZIP:-/usr/local/bin/unzip}
GIT_UNZIP=${GIT_UNZIP:-unzip}
test_lazy_prereq UNZIP '
"$GIT_UNZIP" -v
test $? -ne 127
'
run_with_limited_cmdline () {
(ulimit -s 128 && "$@")
}
test_lazy_prereq CMDLINE_LIMIT '
test_have_prereq !HPPA,!MINGW,!CYGWIN &&
run_with_limited_cmdline true
'
run_with_limited_stack () {
(ulimit -s 128 && "$@")
}
test_lazy_prereq ULIMIT_STACK_SIZE '
test_have_prereq !HPPA,!MINGW,!CYGWIN &&
run_with_limited_stack true
'
run_with_limited_open_files () {
(ulimit -n 32 && "$@")
}
test_lazy_prereq ULIMIT_FILE_DESCRIPTORS '
test_have_prereq !MINGW,!CYGWIN &&
run_with_limited_open_files true
'
build_option () {
git version --build-options |
sed -ne "s/^$1: //p"
}
test_lazy_prereq SIZE_T_IS_64BIT '
test 8 -eq "$(build_option sizeof-size_t)"
'
test_lazy_prereq LONG_IS_64BIT '
test 8 -le "$(build_option sizeof-long)"
'
test_lazy_prereq TIME_IS_64BIT 'test-tool date is64bit'
test_lazy_prereq TIME_T_IS_64BIT 'test-tool date time_t-is64bit'
test_lazy_prereq CURL '
curl --version
'
# SHA1 is a test if the hash algorithm in use is SHA-1. This is both for tests
# which will not work with other hash algorithms and tests that work but don't
# test anything meaningful (e.g. special values which cause short collisions).
test_lazy_prereq SHA1 '
case "$GIT_DEFAULT_HASH" in
sha1) true ;;
"") test $(git hash-object /dev/null) = e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 ;;
*) false ;;
esac
'
test_lazy_prereq REBASE_P '
test -z "$GIT_TEST_SKIP_REBASE_P"
'
maintenance: add start/stop subcommands Add new subcommands to 'git maintenance' that start or stop background maintenance using 'cron', when available. This integration is as simple as I could make it, barring some implementation complications. The schedule is laid out as follows: 0 1-23 * * * $cmd maintenance run --schedule=hourly 0 0 * * 1-6 $cmd maintenance run --schedule=daily 0 0 * * 0 $cmd maintenance run --schedule=weekly where $cmd is a properly-qualified 'git for-each-repo' execution: $cmd=$path/git --exec-path=$path for-each-repo --config=maintenance.repo where $path points to the location of the Git executable running 'git maintenance start'. This is critical for systems with multiple versions of Git. Specifically, macOS has a system version at '/usr/bin/git' while the version that users can install resides at '/usr/local/bin/git' (symlinked to '/usr/local/libexec/git-core/git'). This will also use your locally-built version if you build and run this in your development environment without installing first. This conditional schedule avoids having cron launch multiple 'git for-each-repo' commands in parallel. Such parallel commands would likely lead to the 'hourly' and 'daily' tasks competing over the object database lock. This could lead to to some tasks never being run! Since the --schedule=<frequency> argument will run all tasks with _at least_ the given frequency, the daily runs will also run the hourly tasks. Similarly, the weekly runs will also run the daily and hourly tasks. The GIT_TEST_CRONTAB environment variable is not intended for users to edit, but instead as a way to mock the 'crontab [-l]' command. This variable is set in test-lib.sh to avoid a future test from accidentally running anything with the cron integration from modifying the user's schedule. We use GIT_TEST_CRONTAB='test-tool crontab <file>' in our tests to check how the schedule is modified in 'git maintenance (start|stop)' commands. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-11 19:49:18 +02:00
# Ensure that no test accidentally triggers a Git command
# that runs the actual maintenance scheduler, affecting a user's
# system permanently.
# Tests that verify the scheduler integration must set this locally
maintenance: add start/stop subcommands Add new subcommands to 'git maintenance' that start or stop background maintenance using 'cron', when available. This integration is as simple as I could make it, barring some implementation complications. The schedule is laid out as follows: 0 1-23 * * * $cmd maintenance run --schedule=hourly 0 0 * * 1-6 $cmd maintenance run --schedule=daily 0 0 * * 0 $cmd maintenance run --schedule=weekly where $cmd is a properly-qualified 'git for-each-repo' execution: $cmd=$path/git --exec-path=$path for-each-repo --config=maintenance.repo where $path points to the location of the Git executable running 'git maintenance start'. This is critical for systems with multiple versions of Git. Specifically, macOS has a system version at '/usr/bin/git' while the version that users can install resides at '/usr/local/bin/git' (symlinked to '/usr/local/libexec/git-core/git'). This will also use your locally-built version if you build and run this in your development environment without installing first. This conditional schedule avoids having cron launch multiple 'git for-each-repo' commands in parallel. Such parallel commands would likely lead to the 'hourly' and 'daily' tasks competing over the object database lock. This could lead to to some tasks never being run! Since the --schedule=<frequency> argument will run all tasks with _at least_ the given frequency, the daily runs will also run the hourly tasks. Similarly, the weekly runs will also run the daily and hourly tasks. The GIT_TEST_CRONTAB environment variable is not intended for users to edit, but instead as a way to mock the 'crontab [-l]' command. This variable is set in test-lib.sh to avoid a future test from accidentally running anything with the cron integration from modifying the user's schedule. We use GIT_TEST_CRONTAB='test-tool crontab <file>' in our tests to check how the schedule is modified in 'git maintenance (start|stop)' commands. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-11 19:49:18 +02:00
# to avoid errors.
GIT_TEST_MAINT_SCHEDULER="none:exit 1"