Testing OpenWrt is important, and there is a test suite in the making.
For maximum convenience and minimal CI over-usage, make it simple to run
tests locally. The main Makefile now attempts to include
`tests/Makefile` and silently fails if it doesn't.
While the test suite[1] is still young, it provides good examples of how
to test things around OpenWrt: starting with shell scripts using
`bats`[2], followed by QEMU tests, and finally real device tests using
LabGrid[3]. This could lead to the creation of the best OpenWrt version
yet.
Please consult the `openwrt-tests.git` README.md for details on the
setup. Once installed you may run commands like the following:
* make tests/shell # run shell tests
* make tests/x86-64 # run and test x86/64 in QEMU
[1]: http://github.com/aparcar/openwrt-tests/
[2]: https://bats-core.readthedocs.io
[3]: https://labgrid.readthedocs.io
Signed-off-by: Paul Spooren <mail@aparcar.org>
Link: https://github.com/openwrt/openwrt/pull/15647
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
The output of command_all when inside a nix-shell looks like the below
where /usr does not match:
➜ scripts/command_all.sh pkg-config
/nix/store/ifr6srqgpvygd5vp14748d109ri31isv-pkg-config-wrapper-0.29.2/bin/pkg-config
Signed-off-by: Sandro Jäckel <sandro.jaeckel@gmail.com>
Add a variable that stores the original value of $PATH
in the host system's shell, before Make alters it.
This can be useful for when it is necessary
to ignore symlinks and programs made by the build system.
Define this new variable before all instances of
'export PATH:=' or similar.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
STAGING_DIR may be provided from command line. We currently hardcoded
STAGING_DIR_HOST and STAGING_DIR_HOSTPKG to the default location but we
currently have some relocatable patch that derive the path from
STAGING_DIR.
Fix this and correctly derive STAGING_DIR_HOST and STAGING_DIR_HOSTPKG
from STAGING_DIR.
The intention is to fix inconsistency from the relocatable patch and the
use of STAGING_DIR_HOST that is always hardcoded.
This with a wrong configuration may end up in broken state with some
host tools expecing a PATH from STAGING_DIR and others using library
from the default staging_dir/host path.
To save downstream project the original implementation is saved while
fixing the inconsistency between patch and .mk.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
We simply grep for "/usr". So no need for "-E" or "\/". Furthermore, in
the new grep versions this creates warnings.
As written in the grep-3.8 announcement:
Regular expressions with stray backslashes now cause warnings, as
their unspecified behavior can lead to unexpected results.
For example, '\a' and 'a' are not always equivalent
<https://bugs.gnu.org/39678>.
Fixes warnings in the form of:
grep: warning: stray \ before /
Signed-off-by: Nick Hainke <vincent@systemli.org>
targetclean should not remove BUILD_LOG
Fixes: db34b93331e9 (add a version that can be bumped to force toolchain/target rebuild)
Signed-off-by: Huangbin Zhan <zhanhb88@gmail.com>
Ubuntu started to flag which as deprecated and it
seems which is not really standard and may vary
across Distro.
Drop the use of which and use the standard 'command -v'
for this simple task.
Which is still present in the prereq if some package/script
still use which.
A utility script called command_all.sh is implemented that
will just mimic the output of which -a.
Signed-off-by: Ansuel Smith <ansuelsmth@gmail.com>
This can be used to ensure that the compiled code is up to date, when
something important changes in the toolchain.
A recent example of this is the gcc 11 fix for a code miscompilation issue
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This reverts commit c7aec47e5e3a3ff7b5fdaa11cd1e62cae6746acb.
The original commit replaces 'which' with 'command'. Sadly most of
them are not equivalent and for 'which -a', there is no easy
replacements that would not reimplement PATH parsing logic. Hence
revert. Keeping a dependency on which is absolutely fine.
Signed-off-by: Clemens Fruhwirth <clemens@endorphin.org>
The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
This keeps the configuration, like the size of the cache, and the
statistics intact. Move the removal of the cache directory to the
distclean target, but only delete the .ccache directory inside of our
build tree, as we should not mess with a user-configured external ccache
directory this destructively.
Signed-off-by: Sven Wegener <sven.wegener@stealer.net>
`which` utility is not shipped by default for example on recent Arch
Linux and then any steps relying on its presence fails, like for example
following Python3 prereq build check:
$ python3 --version
Python 3.9.1
$ make
/bin/sh: line 1: which: command not found
/bin/sh: line 1: which: command not found
/bin/sh: line 1: which: command not found
...
Checking 'python3'... failed.
...
Fix this by switching to Bash builtin `command` which should provide
same functionality.
Fixes: FS#3525
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Set CCACHE_DIR to $(TOPDIR)/.ccache and CCACHE_BASEDIR to $(TOPDIR).
This allows to do clean and dirclean. Cache hit rate for test build
after dirclean is ~65%.
If CCACHE is enabled stats are printed out at the end of building process.
CCACHE_DIR config variable allows to override default, which could be useful
when sharing cache with many builds.
cacheclean make target allows to clean the cache.
Changes from v1:
- remove ccache directory using CCACHE_DIR variable
- remove ccache leftovers from sdk and toolchain make files
- introduce CONFIG_CCACHE_DIR variable
- introduce cacheclean make target
Signed-off-by: Roman Yeryomin <roman@advem.lv>
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6e2
("build: create JSON files containing image info").
They are useful for firmware wizards and script checking for
reproducibility.
Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.
This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.
Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.
Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.
As before, this creation is enabled by default only if `BUILDBOT` is set.
Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
When sharing a common build directory between different build platforms
eg. macos v docker based linux v virtual machine, a 'make dirclean'
isn't quite enough to clean all the platform related binaries. The
'conf' and 'mconf' aka 'make menuconfig/defconfig & friends' utilities
remain.
Clean those as part of 'dirclean' so they get rebuilt for the current
platform on the next 'make'
Signed-off-by: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>
The prepare target was added some 11 years ago to build tools and
toolchain and was recently extended to create buildinfo files for
reproducibility, meaning {feeds,version,config}.buildinfo.
As the buildbot workflow is more complex than the single prepare (kmod
feed insertion), prepare is only used to create those buildinfo files.
Running prepare however runs `target/compile` as well, taking time even
everything is already compiled.
Splitting this allows the buildbot to run only the `buildinfo` target
while others can still use the convenience feature `prepare`.
Signed-off-by: Paul Spooren <mail@aparcar.org>
On a recent Gentoo Linux installation, invoking `make menuconfig`, `make
kernel_menuconfig` or `make kernel_nconfig` in the build system fails,
whereas for example `make menuconfig` in the kernel tree alone works as
expected.
This is happening because STAGING_PREFIX is not defined when kernel's
{menu,n}config target calls pkg-config from the toolchain/host and thus
pkg-config returns an empty value, and the fallback values in the kernel
config script are applied but those are off and the linking fails.
Solution is to use system's pkg-config for all ncurses based menu config
targets in order to provide proper compiler/linker flags.
Ref: FS#2423
Cc: Thomas Albers <thomas.gameiro@gmail.com>
Tested-by: Stijn Tintel <stijn@linux-ipv6.be>
Tested-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
generate feeds.buildinfo and version.buildinfo in build dir after
containing the feed revisions (via ./scripts/feeds list -sf) as well as
the current revision of buildroot (via ./scripts/getver.sh).
With this information it should be possible to reproduce any build,
especially the release builds.
Usage would be to move feeds.buildinfo to feeds.conf and git checkout the
revision hash of version.buildinfo.
Content of feeds.buildinfo would look similar to this:
src-git routing https://git.openwrt.org/feed/routing.git^bf475d6
src-git telephony https://git.openwrt.org/feed/telephony.git^470eb8e
...
Content of version.buildinfo would look similar to this:
r10203+1-c12bd3a21b
Without the exact feed revision it is not possible to determine
installed package versions.
Also rename config.seed to config.buildinfo to follow the recommended
style of https://reproducible-builds.org/docs/recording/
Signed-off-by: Paul Spooren <mail@aparcar.org>
Revert "mac80211: add new minstrel_ht patches to improve probing on mt76x2" (9861050b85e5381c93d73ffdbc24c130400e0fb8)
Revert "kernel: use bulk free in kfree_skb_list to improve performance" (98b654de2e7502507b31f0fb82befbb48f9c8542)
Revert "ramips: add preliminary support for WIO ONE" (085141dc5ba5e763e1bf4c1a2fd6957160810ccc)
Revert "ramips: add preliminary support for SGE AP-MTKH7-0006 developer board" (b1db6d0539511f60bf65fbde28b16afa31180e34)
Revert "build: use config.site generated by autoconf-lean, drop hardcoded sitefiles" (363ce4329d4f6c00b4a7cb83dd1821fafc0d40ad)
Revert "toolchain: add autoconf-lean" (fdb30eed03ae56cd87078b64d9b2fac00799e783)
Revert "build: allow overriding the filename on the remote server when downloading" (6fa0e07758524eddf20c9b66dfcb55c490e1961e)
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This reverts commit fe928c7a81810fd84210a5e9594b7fa8ce1064ce as it seems
to introduce build breakage when local.mk doesn't exist.
make --trace
toplevel.mk:216: target 'local.mk' does not exist
Signed-off-by: Petr Štetiar <ynezz@true.cz>
I tend to automate most of the boring and repetitive tasks like firmware
flashing, build config reconfiguration etc., so I always end up adding
`include local.mk` as a last line in my Makefile, where local.mk is
usually symlink to some other place, just to not accidentally delete it
during `git clean`.
Carrying this single uncommited modification along in the development
process is quite PITA, because it's causing problems during Git
workflow, while rebasing etc.
I hope, that I'm not alone using similar workflow, so I believe, that
this modification might be useful for others as well.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
In some places the output of commands, which include "cd" are used.
In case of CDPATH the new path is printed, which might not be expected.
Disable the variable to avoid these problem.
When CDPATH was set by the user to some value like "export CDPATH=."
the git checkout done by the build system did not work anymore, the
git cloning aborted with such an error message for example:
....
Packing checkout...
tar: /disk/fs1/tmp2/mehrtens/pon-ugw/ugw-haps/openwrt/tmp/dl/ppa-drv-1.0\n@1534240258: Cannot stat: No such file or directory
tar: Date sample file not found
Try 'tar --help' or 'tar --usage' for more information.
.....
To avoid this, this patch makes the build system unset CDPATH inside
the build system, so the build system will still work even when the
user set this variable in his local environment.
Signed-off-by: Thomas Langer <thomas.langer@intel.com>
Signed-off-by: Hauke Mehrtens <hauke.mehrtens@intel.com>
Acked-by: Hans Dedecker <dedeckeh@gmail.com>
In order to be able to better compare files to sync in the future, include
all BIN_DIR subdirectories in the checksum calculation.
To not break existing applications, restrict the recursive checksumming to
CONFIG_BUILDBOT for now.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
This reverts commit 2990a21058243b067bb56fed36ee69205595993e.
This introduces a race condition, let's fix this in buildbot instead.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Using a single host package staging dir (and build dir) significantly speeds up
builds when multiple targets are built in succession, especially for large host
packages like NodeJS.
$(STAGING_DIR)/host is kept in addition to $(STAGING_DIR_HOSTPKG) in most
places; it is still used as destination for host files in Build/InstallDev.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Ensure that BIN_DIR exists when the diffconfig target needs it.
Otherwise 'make diffconfig' fails after 'make clean'
Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
Add a "diffconfig" build target which stores the output of
"scripts/diffconfig.sh" as "config.seed" in the image output directory and
invoke that target by default.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
This is intended to be used for a wide array of package sanity checks.
The first check that is implemented is for the hash of downloaded files.
It checks:
- Missing hash
- Use of SHA256 instead of MD5
- dl/<file> hash not matching hash in makefile
- deprecated MD5SUM variable
The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Add a new "checksum" make target which generates an sha256sums file over the
image files produced in bin/targets/ and automatically call it during make
world after the package index generation.
The advantage of this new target is that it is guaranteed to run after the
images, the SDK and the ImageBuilder archives have been generated to ensure
that they all end up in the checksum file. Fixes FS#51.
Uses sed to postprocess the OpenSSL digest output into an sha256sum command
compatible format.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Currently "make clean" only clears the build_dir/target*, but leaves
staging_dir/target* intact. "make clean" should also clean the
staging_dir/target* directories, as in the current situation some old
packages or libraries may be linked into the firmware from staging_dir
despite a "make clean".
The patch reorganises clean / dirclean functionality so that
* "make clean" also clears the staging_dir/target* in addition to
build_dir/target*.
* "make dirclean" clears toolchain and host(=tools) directories from both
build_dir and staging_dir
signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
SVN-Revision: 45973
Makefile: remove non-existent STAGING_DIR_TOOLCHAIN from dirclean
Openwrt's top level Makefile uses STAGING_DIR_TOOLCHAIN in the make dirclean
statement.
https://dev.openwrt.org/browser/trunk/Makefile#L55
rm -rf $(STAGING_DIR) $(STAGING_DIR_HOST) $(STAGING_DIR_TOOLCHAIN)
$(TOOLCHAIN_DIR) $(BUILD_DIR_HOST) $(BUILD_DIR_TOOLCHAIN)
As far as I can determine, no such variable has been defined. I made a search
in Openwrt source repository and the one line in Makefile's dirclean command
is the only place where that variable exists.
The item has been introduced to Makefile by r8362, but even at that time
neither Makefile nor rules.mk defined such a variable. Most likely the goal
has been to set both staging_dir/toolchain and build_dir/toolchain to be
cleaned, but one of the variables has been erroneous. The correct variable
for build_dir/toolchain has been then added by r13494.
References:
https://dev.openwrt.org/changeset/8362/https://dev.openwrt.org/browser/trunk/Makefile?rev=8362https://dev.openwrt.org/browser/trunk/rules.mk?rev=8362https://lists.openwrt.org/pipermail/openwrt-devel/2007-August/001159.htmlhttps://dev.openwrt.org/changeset/13494
In current code,
TOOLCHAIN_DIR = $(TOPDIR)/staging_dir/$(TOOLCHAIN_DIR_NAME)
BUILD_DIR_TOOLCHAIN = $(TOPDIR)/build_dir/$(TOOLCHAIN_DIR_NAME)
so the item STAGING_DIR_TOOLCHAIN in the rm command is unnecessary.
signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
SVN-Revision: 45736
Most of the time, we want to make sure OpenWrt has been configured and
setup before start running make. However, in case of package/symlinks,
forcing prereq as a dependency creates multiple issues:
*when executed on a clean workspace, it will prompt for user input
and open a menuconfig window before executing the feeds command
*the only way around that is to provide a .config. However, the "prereq"
target would then run a "make defconfig", which will remove all the
packages in the .config but from external feeds, as feeds have not been
installed yet.
The only way to currently work around this, is to generate a fake config
by running "make defconfig", then "make package/symlinks", copy the real
config (which at this point disregards the previously generated config),
and run make defconfig again. Something like this:
make defconfig
make package/symlinks
cp real.config .config
make defconfig
This change is removing the need for the first defconfig, making the
process more logical for OpenWrt users using the package/symlinks target.
Signed-off-by: Mathieu Olivari <mathieu@qca.qualcomm.com>
SVN-Revision: 45657
