We have our own GitLab-CI runner, but with only one we can't run many
jobs in parallel so it takes a very long time before all the tests have
completed. In addition, if that runner goes down, we have nothing at
all.
GitLab offers the following machine types for hosted runners on Linux
x86-64 [1]. The default is the "small" runner. Using a larger runner
increases the "Cost factor" [2].
For opensource projects, the Cost factor is reduced to 0.5 (1 minute per
2 minutes of job time) whatever the runner type.
Runner Tag vCPUs Memory Storage Cost factor (OSS)
saas-linux-small-amd64 (default) 2 8 GB 30 GB 1 (0.5)
saas-linux-medium-amd64 4 16 GB 50 GB 2 (0.5)
saas-linux-large-amd64 (Premium and Ultimate only) 8 32 GB 100 GB 3 (0.5)
saas-linux-xlarge-amd64 (Premium and Ultimate only) 16 64 GB 200 GB 6 (0.5)
saas-linux-2xlarge-amd64 (Premium and Ultimate only) 32 128 GB 200 GB 12 (0.5)
Compute minutes consumed by a job is calculated by:
Job duration / 60 * Cost factor
(Job duration: The time, in seconds, that a job took to run, not
including time spent in the created or pending statuses.)
Thanks to the GitLab OSS program [3], Buildroot benefits from a free
Ultimate subscription and can use GitLab shared runners tagged with
saas-linux-{large, xlarge, 2xlarge}-amd64. In addition, we receive
50,000 free runner minutes per month.
In order to use one of those tags in Buildroot GitLab-CI jobs, we have
to classify all tests by resource requirement, to make sure the job
doesn't fail because it times out or has insufficient memory or disk
space. While we usually shouldn't use the largest runner for
everything, we can use larger runner without cost penalty thanks to
the cost factor reduced to 0.5 for opensource projects. This will
reduce the CI minutes consumed by a CPU intensive job.
First we introduce some new templates used to add the corresponding
runner tag to a runtime test job (reusing the GitLab terminology).
.runner-{small,medium,large,xlarge,2xlarge}
Most of our tests are fast (checkpackage, test_external_bootlin...), so
saas-linux-small-amd64 runner tag is enough. Default to this tag if
nothing else is specified.
Add a comment next to the test class to provide the runner tag.
This runner tag is retrieved when generating the
generated-gitlab-ci.yml file used to create the child pipeline where
the runtime test jobs are executed.
We use the list of runtime tests returned by node2:
"tests.boot.test_edk2.TestEdk2.test_run"
We convert each element of this list to get the path to the test source
file and the name of the test:
"support/testing/tests/boot/test_edk2.py"
TestEdk2
With that, we can grep into the test source file to retrieve the runner
tag placed one line above the test class:
# GitLab-runner: large
class TestEdk2(infra.basetest.BRTest):
Once the runner tag is retrieved, it's used to use the corresponding
runner template to the runtime test job:
tests.boot.test_edk2.TestEdk2.test_run: { extends: [ .runtime_test_base, .runner-large ]}
GitLab runners hosted by the Buildroot project should be able to run
any jobs, so they should be tagged with Gitlab runner tags
(saas-linux-{small,medium,large,xlarge,2xlarge}-amd64).
A specific runner tag "buildroot-runner" can be used to allow running
a job only on such runners.
If a test can't be executed by any shared GitLab-CI runners, we have
to use a runner owned by the Buildroot project. In this case we have
to use a specific template ".runner-buildroot-runner-only" in order to
add the specific runner tag "buildroot-runner" to the job running the
test. There is no such runtime test at the moment.
The proposed classification is based on a previous pipeline analysis
[5]:
- Tests lasting more than 3 hours will use 2xlarge runners.
- Tests lasting more than 2 hours will use xlarge runners.
- Tests lasting more than 1 hours will use large runners.
- Tests building a kernel or a toolchain will use medium runners.
- All other tests will use small runners when possible.
CI minute cost estimate:
tests.package.test_clang.TestClangCompilerRT.test_run lasts 4h25 on the
Buildroot runner. If we this duration for 2xlarge runners, the CI
minute consumed would be:
(15900 / 60) * 0.5 = 133
With 6 jobs using a 2xlarge runners we used ~795 CI minutes.
tests.package.test_kmscube.TestKmsCube.test_run list 2h04 on the
Buildroot runner. If we this duration for xlarge runners, the CI
minute consumed would be:
(7440 / 60) * 0.5 = 62
With 4 jobs using a xlarge runners we used ~248 CI minutes.
tests.package.test_weston.TestWeston.test_run last 1h15 on th
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(4500 / 60) * 0.5 = 37.5
With 28 jobs using a large runners we used ~1050 CI minutes.
tests.package.test_gstreamer1.TestGstreamer1.test_run last 46min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(2760 / 60) * 0.5 = 23
With 31 jobs using a medium runners we used ~713 CI minutes.
tests.package.test_python.TestPython3Py.test_run last 13min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(780 / 60) * 0.5 = 6.5
With 691 jobs using a medium runners we used ~4491 CI minutes.
In total, one pipeline for the runtime tests cost ~7297 CI minutes.
After a first try [6], we are actually using 8000 CI minutes per
pipeline.
We run such pipeline once a week (on Monday), one for each Buildroot
releases every 3 month, one for each stable and LTS release per month,
and one for each release candidate (3).
Worst case (release month):
(4 weeks + 1 release + 1 stable + 1 LTS + 3 release candidate) * 8000 CI
minutes: 80000 CI minutes / 50000.
So we would spend the minutes very quickly in the worst case scenario.
We have to keep one pipeline under 5000 CI minutes.
[1] https://docs.gitlab.com/ci/runners/hosted_runners/linux/#machine-types-available-for-linux---x86-64
[2] https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factorshttps://docs.gitlab.com/ci/pipelines/compute_minutes/#compute-usage-calculationhttps://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors-of-hosted-runners-for-gitlabcom
[3] https://gitlab.com/buildroot.org/gitlab-oss
[4] https://docs.gitlab.com/ci/runners/hosted_runners/#gitlabcom-hosted-runner-workflow
[5] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2416603721
[6] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2562421098
Signed-off-by: Romain Naour <romain.naour@smile.fr>
[Arnout:
- simplify parsing of test_file and test_name;
- match the entire test_name instead of substring;
- assume "small" by default;
- remove the "small" tags;
- use "gitlab-runner" instead of "Gitlab-runner".
]
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit c9df1b64b2)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Currently, the relocate-sdk.sh script scans the whole extracted SDK tree
to find instances of paths it needs to replace, which can take a
significant amount of time when the SDK is large, particularly relative
to the number of files that actually need to change.
However, the resulting list only depends on the SDK tarball itself, so
we can calculate it at build time and ship it with the tarball so
relocate-sdk.sh can use it directly.
Testing this on my machine with somewhat IOPS-limited rotating media,
the time goes down from:
$ time ./relocate-sdk.sh
Relocating the buildroot SDK from [...] to [...] ...
./relocate-sdk.sh 5.19s user 26.21s system 9% cpu 5:34.40 total
To:
$ time ./relocate-sdk.sh
Relocating the buildroot SDK from [...] to [...] ...
./relocate-sdk.sh 0.49s user 0.29s system 103% cpu 0.749 total
Signed-off-by: Florian Larysch <fl@n621.de>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit 63877f9e86)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The meson upgrade from 1.1.1 to 1.2.0 have been reverted because
host-qemu (version v8.0.3) was searching libraries in the build machine
instead of buildroot host directory [1].
$ grep -r LINK_ARGS * | grep "\-L/usr"
build.ninja: LINK_ARGS =
-L/usr/lib/gcc/x86_64-linux-gnu/10 \
-L/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu \
-L/usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib -L/usr/lib \
-L/lib/x86_64-linux-gnu \
-L/lib/../lib \
-L/usr/lib/../lib \
-L/usr/lib/gcc/x86_64-linux-gnu/10/../../.. \
-L/lib \
-I/[...]/host/include \
-L/[...]/host/lib \
-I/[...]/host/include/libfdt \
...
Upgrading qemu to version v8.1.0 solves the problem:
$ grep -r LINK_ARGS * | grep "\-L/usr"
build.ninja: LINK_ARGS =
-I/[...]/host/include \
-L/[...]/host/lib \
-I/[...]/host/include/libfdt \
...
Those two tests were done with the same buildroot version (2023.08) on
the same machine were the bug appeared. Using, git bisect, in this
environment, it was found that the problem was fixed in qemu by the
commit [2].
Thus, it is possible that the problem was introduced by improper use of
meson by qemu <= v8.0.3. Also, meson has been updated to version 1.2.1
and 1.3.0 in yocto and it doesn't seem to create any issue. [3]
The original problem is no longer present and some projects require
meson > 1.1.1 (e.g. rusticl requires meson 1.2.0 since mesa 23.3.0 and
meson 1.3.1 since mesa 24.0.0).
"pkgconfig" field is deprecated by meson 1.3.0 and replaced by
"pkg-config". See: [4]
The patch that adds the pkg_config_static property has been rebased.
[1] acfdf21f0b
[2] https://gitlab.com/qemu-project/qemu/-/commit/Fc9a809e0d28417fa1e7f7efc19f845bda4c1be9
[3] https://github.com/yoctoproject/poky/commits/master/meta/recipes-devtools/meson
[4] https://mesonbuild.com/Release-notes-for-1-3-0.html#machine-files-pkgconfig-field-deprecated-and-replaced-by-pkgconfig
Signed-off-by: Antoine Coutant <antoine.coutant@smile.fr>
[yann.morin.1998@free.fr:
- keep our formatting in the patch
- reflow commit log (to keep busy while test-building!)
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When an error occurs, the gitlab-ci job log doesn't contain any useful
information than the name of the failing test:
FAIL: test_run (tests.package.test_python_paho_mqtt.TestPythonPahoMQTT)
In order to encourage contributors to investigate issues reported by
gitlab-ci, we want to print the last lines of the log file (build or
runtime).
Unfortunately, gitlab-ci job log completely strips lines ending with
CRCRLF [1][2]. We have to take a look at the gitlab-ci raw log to see
the complete log [3].
To workaround this issue, remove crlf from qemu serial stdio log
while printing in the gitlab-ci job log (we don't want to change
the log file generated by support/testing/run-tests and saved as
artefacts).
[1] https://gitlab.com/gitlab-org/gitlab/-/issues/218771
[2] https://gitlab.com/kubu93/buildroot/-/jobs/5492937691
[3] https://gitlab.com/kubu93/buildroot/-/jobs/5492937691/raw
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Each time a new pipeline is triggered, some jobs may fail due to
temporary issue with a Gitlab runner (network, power supply, docker or
maintainance).
Most of the problems are "runner system failure" [1] and require to
retart each failed jobs manually by maintainers to complete the
pipeline with only real failures if any.
The "retry" keyword allows to configure how many times a job is retried
if it fails. "retry:when" allows to retry a failed job only on
specific failure types like "runner_system_failure".
While at it, retry a job if it failed due to a timeout failure (this
timeout means that the job was pending for more than 24h) [2].
Such timeout failures occur on pipelines testing each Buildroot's
defconfig since there is not enough gitlab runner available to build
all of them within 24h.
Retry only jobs that are more likely to wait for a runner
(generate-gitlab-ci-yml, runtime_test_base, defconfig_base and test_pkg).
[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/4936949397 (runner system failure)
[2] https://gitlab.com/buildroot.org/buildroot/-/jobs/4936949530 (timeout failure or the job got stuck)
https://docs.gitlab.com/ee/ci/yaml/#retrywhen
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
"stages" is curently not defined in the .gitlab-ci.yml file,
so the default pipeline stages are:
.pre
build
test
deploy
.post
Since any jobs specify a stage, all jobs are assigned the "test"
stage [1]. All other stages defined by default are not used in the
gitlab-ci pipeline, they remain hidden.
In order to introduce a new custom stage, add the "test" stage
explicitely.
[1] https://docs.gitlab.com/ee/ci/yaml/#stages
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Teach check-package to detect python files by type and check them using
flake8.
Do not use subprocess to call 'python3 -m flake8' in order to avoid too
many spawned shells, which in its turn would slow down the check for
multiple files. (make check-package takes twice the time using a shell
for each flake8 call, when compared of importing the main application)
Expand the runtime test and the unit tests for check-package.
Remove check-flake8 from the makefile and also from the GitLab CI
because the exact same checks become part of check-package.
Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: add a comment to x-python to explain its purpose]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the container. However,
our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation.
It is most probably impossible to change the ownership of the files: we
run as non-root,and the files belong to root (in the container). So
we're stuck.
The alternative, is to do as git suggest, and tell it to ignore the
situation. In a local setup, this would be very insecure, but in the
pipelines, this is in a throw-away container, where a single user exists
and is running, so we don't care much (if at all).
Add a global before_script that registers the git config to ignore
ownership issues in the buildroot repository; see [0] for the definition
of the CI_PROJECT_DIR variable. Note: unlike what is said in there, and
in [1], the value actually seen in CI_PROJECT_DIR is already prefixed
with CI_BUILDS_DIR (the documentation is unclear about that point).
[0] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
[1] https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When gitlab prepares a job to run, it checks out the repository with a
non-root user, and spawns a container that runs as root, with some UID
mapping that makes the files be owned by root in the
container. However, our pipelines run as a nont-root user.
Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated
the docker image that is used to run in our pipelines.
That new image includes a git version that is stricter about the
ownership of the git tree it is acting in: git aborts in error when the
user running it does not own the repository.
We use `git ls-tree` quite a lot in our check-{flake8,package,symbols}
rules, so they all fail (in various ways).
To fix this, we either need to fix the ownership or tell git to ignore
the situation. In either case, we'll need to run a scriptlet before all
our jobs.
Gitlab-ci allows to provide a global before_script, that is inherited by
all jobs. However, some of our jobs already declare a before_script, and
that would shadow the global before_script.
There is no technical reason to do our before_script separately from
the actual script, so we move the code from the before_scripts to the
corresponding scripts.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This script checks for inconsistencies on symbols declared in Config.in
and used in .mk files.
Currently it checks only symbols following the pattern BR2_\w+ .
The script first gets the list of all files in the repository (using git
ls-files like 'make check-flake8' already do).
Then it parses all relevant files, searching for symbol definitions and
usages, and add entries into a database.
At the end, the database is searched for inconsistencies:
- symbol that is part of "choice" and is referenced with "select";
- legacy symbol being referenced in packages;
- legacy symbol being redefined in packages;
- symbol referenced but not defined;
- symbol defined but not referenced;
- legacy symbol that has a Note stating it is referenced by a package
(for legacy handling) but is referenced in the package without a
comment "# legacy";
- legacy symbol that has a Note stating it is referenced by a package
but it is not actually referenced.
There is also a debug parameter --search that dumps any filename or
symbol entries from the database that matches a regexp.
Sample usages:
$ utils/check-symbols
$ utils/docker-run utils/check-symbols
$ utils/check-symbols --search 'GETTEXT\b|\/openssl'
At same time the script is created:
- add unit tests for it, they can be run using:
utils/docker-run python3 -m pytest -v utils/checksymbolslib/
- add two more GitLab CI jobs: check-symbols (to check current tree
using the script) and check-check-symbols (to check the script against
its unit tests)
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Peter: print warnings to stderr, rename change_current_dir() to
change_to_top_dir()]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
