Currently, the relocate-sdk.sh script scans the whole extracted SDK tree
to find instances of paths it needs to replace, which can take a
significant amount of time when the SDK is large, particularly relative
to the number of files that actually need to change.
However, the resulting list only depends on the SDK tarball itself, so
we can calculate it at build time and ship it with the tarball so
relocate-sdk.sh can use it directly.
Testing this on my machine with somewhat IOPS-limited rotating media,
the time goes down from:
$ time ./relocate-sdk.sh
Relocating the buildroot SDK from [...] to [...] ...
./relocate-sdk.sh 5.19s user 26.21s system 9% cpu 5:34.40 total
To:
$ time ./relocate-sdk.sh
Relocating the buildroot SDK from [...] to [...] ...
./relocate-sdk.sh 0.49s user 0.29s system 103% cpu 0.749 total
Signed-off-by: Florian Larysch <fl@n621.de>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit 63877f9e86)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The meson upgrade from 1.1.1 to 1.2.0 have been reverted because
host-qemu (version v8.0.3) was searching libraries in the build machine
instead of buildroot host directory [1].
$ grep -r LINK_ARGS * | grep "\-L/usr"
build.ninja: LINK_ARGS =
-L/usr/lib/gcc/x86_64-linux-gnu/10 \
-L/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu \
-L/usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib -L/usr/lib \
-L/lib/x86_64-linux-gnu \
-L/lib/../lib \
-L/usr/lib/../lib \
-L/usr/lib/gcc/x86_64-linux-gnu/10/../../.. \
-L/lib \
-I/[...]/host/include \
-L/[...]/host/lib \
-I/[...]/host/include/libfdt \
...
Upgrading qemu to version v8.1.0 solves the problem:
$ grep -r LINK_ARGS * | grep "\-L/usr"
build.ninja: LINK_ARGS =
-I/[...]/host/include \
-L/[...]/host/lib \
-I/[...]/host/include/libfdt \
...
Those two tests were done with the same buildroot version (2023.08) on
the same machine were the bug appeared. Using, git bisect, in this
environment, it was found that the problem was fixed in qemu by the
commit [2].
Thus, it is possible that the problem was introduced by improper use of
meson by qemu <= v8.0.3. Also, meson has been updated to version 1.2.1
and 1.3.0 in yocto and it doesn't seem to create any issue. [3]
The original problem is no longer present and some projects require
meson > 1.1.1 (e.g. rusticl requires meson 1.2.0 since mesa 23.3.0 and
meson 1.3.1 since mesa 24.0.0).
"pkgconfig" field is deprecated by meson 1.3.0 and replaced by
"pkg-config". See: [4]
The patch that adds the pkg_config_static property has been rebased.
[1] acfdf21f0b
[2] https://gitlab.com/qemu-project/qemu/-/commit/Fc9a809e0d28417fa1e7f7efc19f845bda4c1be9
[3] https://github.com/yoctoproject/poky/commits/master/meta/recipes-devtools/meson
[4] https://mesonbuild.com/Release-notes-for-1-3-0.html#machine-files-pkgconfig-field-deprecated-and-replaced-by-pkgconfig
Signed-off-by: Antoine Coutant <antoine.coutant@smile.fr>
[yann.morin.1998@free.fr:
- keep our formatting in the patch
- reflow commit log (to keep busy while test-building!)
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When an error occurs, the gitlab-ci job log doesn't contain any useful
information than the name of the failing test:
FAIL: test_run (tests.package.test_python_paho_mqtt.TestPythonPahoMQTT)
In order to encourage contributors to investigate issues reported by
gitlab-ci, we want to print the last lines of the log file (build or
runtime).
Unfortunately, gitlab-ci job log completely strips lines ending with
CRCRLF [1][2]. We have to take a look at the gitlab-ci raw log to see
the complete log [3].
To workaround this issue, remove crlf from qemu serial stdio log
while printing in the gitlab-ci job log (we don't want to change
the log file generated by support/testing/run-tests and saved as
artefacts).
[1] https://gitlab.com/gitlab-org/gitlab/-/issues/218771
[2] https://gitlab.com/kubu93/buildroot/-/jobs/5492937691
[3] https://gitlab.com/kubu93/buildroot/-/jobs/5492937691/raw
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Each time a new pipeline is triggered, some jobs may fail due to
temporary issue with a Gitlab runner (network, power supply, docker or
maintainance).
Most of the problems are "runner system failure" [1] and require to
retart each failed jobs manually by maintainers to complete the
pipeline with only real failures if any.
The "retry" keyword allows to configure how many times a job is retried
if it fails. "retry:when" allows to retry a failed job only on
specific failure types like "runner_system_failure".
While at it, retry a job if it failed due to a timeout failure (this
timeout means that the job was pending for more than 24h) [2].
Such timeout failures occur on pipelines testing each Buildroot's
defconfig since there is not enough gitlab runner available to build
all of them within 24h.
Retry only jobs that are more likely to wait for a runner
(generate-gitlab-ci-yml, runtime_test_base, defconfig_base and test_pkg).
[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/4936949397 (runner system failure)
[2] https://gitlab.com/buildroot.org/buildroot/-/jobs/4936949530 (timeout failure or the job got stuck)
https://docs.gitlab.com/ee/ci/yaml/#retrywhen
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
"stages" is curently not defined in the .gitlab-ci.yml file,
so the default pipeline stages are:
.pre
build
test
deploy
.post
Since any jobs specify a stage, all jobs are assigned the "test"
stage [1]. All other stages defined by default are not used in the
gitlab-ci pipeline, they remain hidden.
In order to introduce a new custom stage, add the "test" stage
explicitely.
[1] https://docs.gitlab.com/ee/ci/yaml/#stages
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Teach check-package to detect python files by type and check them using
flake8.
Do not use subprocess to call 'python3 -m flake8' in order to avoid too
many spawned shells, which in its turn would slow down the check for
multiple files. (make check-package takes twice the time using a shell
for each flake8 call, when compared of importing the main application)
Expand the runtime test and the unit tests for check-package.
Remove check-flake8 from the makefile and also from the GitLab CI
because the exact same checks become part of check-package.
Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: add a comment to x-python to explain its purpose]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the container. However,
our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation.
It is most probably impossible to change the ownership of the files: we
run as non-root,and the files belong to root (in the container). So
we're stuck.
The alternative, is to do as git suggest, and tell it to ignore the
situation. In a local setup, this would be very insecure, but in the
pipelines, this is in a throw-away container, where a single user exists
and is running, so we don't care much (if at all).
Add a global before_script that registers the git config to ignore
ownership issues in the buildroot repository; see [0] for the definition
of the CI_PROJECT_DIR variable. Note: unlike what is said in there, and
in [1], the value actually seen in CI_PROJECT_DIR is already prefixed
with CI_BUILDS_DIR (the documentation is unclear about that point).
[0] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
[1] https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the
container. However, our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation. In either case, we'll need to run a scriptlet before all
our jobs.
Gitlab-ci allows to provide a global before_script, that is inherited by
all jobs. However, some of our jobs already declare a before_script, and
that would shadow the global before_script.
There is no technical reason to do our before_script separately from
the actual script, so we move the code from the before_scripts to the
corresponding scripts.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This script checks for inconsistencies on symbols declared in Config.in
and used in .mk files.
Currently it checks only symbols following the pattern BR2_\w+ .
The script first gets the list of all files in the repository (using git
ls-files like 'make check-flake8' already do).
Then it parses all relevant files, searching for symbol definitions and
usages, and add entries into a database.
At the end, the database is searched for inconsistencies:
- symbol that is part of "choice" and is referenced with "select";
- legacy symbol being referenced in packages;
- legacy symbol being redefined in packages;
- symbol referenced but not defined;
- symbol defined but not referenced;
- legacy symbol that has a Note stating it is referenced by a package
(for legacy handling) but is referenced in the package without a
comment "# legacy";
- legacy symbol that has a Note stating it is referenced by a package
but it is not actually referenced.
There is also a debug parameter --search that dumps any filename or
symbol entries from the database that matches a regexp.
Sample usages:
$ utils/check-symbols
$ utils/docker-run utils/check-symbols
$ utils/check-symbols --search 'GETTEXT\b|\/openssl'
At same time the script is created:
- add unit tests for it, they can be run using:
utils/docker-run python3 -m pytest -v utils/checksymbolslib/
- add two more GitLab CI jobs: check-symbols (to check current tree
using the script) and check-check-symbols (to check the script against
its unit tests)
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Peter: print warnings to stderr, rename change_current_dir() to
change_to_top_dir()]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
