We have our own GitLab-CI runner, but with only one we can't run many
jobs in parallel so it takes a very long time before all the tests have
completed. In addition, if that runner goes down, we have nothing at
all.
GitLab offers the following machine types for hosted runners on Linux
x86-64 [1]. The default is the "small" runner. Using a larger runner
increases the "Cost factor" [2].
For opensource projects, the Cost factor is reduced to 0.5 (1 minute per
2 minutes of job time) whatever the runner type.
Runner Tag vCPUs Memory Storage Cost factor (OSS)
saas-linux-small-amd64 (default) 2 8 GB 30 GB 1 (0.5)
saas-linux-medium-amd64 4 16 GB 50 GB 2 (0.5)
saas-linux-large-amd64 (Premium and Ultimate only) 8 32 GB 100 GB 3 (0.5)
saas-linux-xlarge-amd64 (Premium and Ultimate only) 16 64 GB 200 GB 6 (0.5)
saas-linux-2xlarge-amd64 (Premium and Ultimate only) 32 128 GB 200 GB 12 (0.5)
Compute minutes consumed by a job is calculated by:
Job duration / 60 * Cost factor
(Job duration: The time, in seconds, that a job took to run, not
including time spent in the created or pending statuses.)
Thanks to the GitLab OSS program [3], Buildroot benefits from a free
Ultimate subscription and can use GitLab shared runners tagged with
saas-linux-{large, xlarge, 2xlarge}-amd64. In addition, we receive
50,000 free runner minutes per month.
In order to use one of those tags in Buildroot GitLab-CI jobs, we have
to classify all tests by resource requirement, to make sure the job
doesn't fail because it times out or has insufficient memory or disk
space. While we usually shouldn't use the largest runner for
everything, we can use larger runner without cost penalty thanks to
the cost factor reduced to 0.5 for opensource projects. This will
reduce the CI minutes consumed by a CPU intensive job.
First we introduce some new templates used to add the corresponding
runner tag to a runtime test job (reusing the GitLab terminology).
.runner-{small,medium,large,xlarge,2xlarge}
Most of our tests are fast (checkpackage, test_external_bootlin...), so
saas-linux-small-amd64 runner tag is enough. Default to this tag if
nothing else is specified.
Add a comment next to the test class to provide the runner tag.
This runner tag is retrieved when generating the
generated-gitlab-ci.yml file used to create the child pipeline where
the runtime test jobs are executed.
We use the list of runtime tests returned by node2:
"tests.boot.test_edk2.TestEdk2.test_run"
We convert each element of this list to get the path to the test source
file and the name of the test:
"support/testing/tests/boot/test_edk2.py"
TestEdk2
With that, we can grep into the test source file to retrieve the runner
tag placed one line above the test class:
# GitLab-runner: large
class TestEdk2(infra.basetest.BRTest):
Once the runner tag is retrieved, it's used to use the corresponding
runner template to the runtime test job:
tests.boot.test_edk2.TestEdk2.test_run: { extends: [ .runtime_test_base, .runner-large ]}
GitLab runners hosted by the Buildroot project should be able to run
any jobs, so they should be tagged with Gitlab runner tags
(saas-linux-{small,medium,large,xlarge,2xlarge}-amd64).
A specific runner tag "buildroot-runner" can be used to allow running
a job only on such runners.
If a test can't be executed by any shared GitLab-CI runners, we have
to use a runner owned by the Buildroot project. In this case we have
to use a specific template ".runner-buildroot-runner-only" in order to
add the specific runner tag "buildroot-runner" to the job running the
test. There is no such runtime test at the moment.
The proposed classification is based on a previous pipeline analysis
[5]:
- Tests lasting more than 3 hours will use 2xlarge runners.
- Tests lasting more than 2 hours will use xlarge runners.
- Tests lasting more than 1 hours will use large runners.
- Tests building a kernel or a toolchain will use medium runners.
- All other tests will use small runners when possible.
CI minute cost estimate:
tests.package.test_clang.TestClangCompilerRT.test_run lasts 4h25 on the
Buildroot runner. If we this duration for 2xlarge runners, the CI
minute consumed would be:
(15900 / 60) * 0.5 = 133
With 6 jobs using a 2xlarge runners we used ~795 CI minutes.
tests.package.test_kmscube.TestKmsCube.test_run list 2h04 on the
Buildroot runner. If we this duration for xlarge runners, the CI
minute consumed would be:
(7440 / 60) * 0.5 = 62
With 4 jobs using a xlarge runners we used ~248 CI minutes.
tests.package.test_weston.TestWeston.test_run last 1h15 on th
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(4500 / 60) * 0.5 = 37.5
With 28 jobs using a large runners we used ~1050 CI minutes.
tests.package.test_gstreamer1.TestGstreamer1.test_run last 46min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(2760 / 60) * 0.5 = 23
With 31 jobs using a medium runners we used ~713 CI minutes.
tests.package.test_python.TestPython3Py.test_run last 13min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(780 / 60) * 0.5 = 6.5
With 691 jobs using a medium runners we used ~4491 CI minutes.
In total, one pipeline for the runtime tests cost ~7297 CI minutes.
After a first try [6], we are actually using 8000 CI minutes per
pipeline.
We run such pipeline once a week (on Monday), one for each Buildroot
releases every 3 month, one for each stable and LTS release per month,
and one for each release candidate (3).
Worst case (release month):
(4 weeks + 1 release + 1 stable + 1 LTS + 3 release candidate) * 8000 CI
minutes: 80000 CI minutes / 50000.
So we would spend the minutes very quickly in the worst case scenario.
We have to keep one pipeline under 5000 CI minutes.
[1] https://docs.gitlab.com/ci/runners/hosted_runners/linux/#machine-types-available-for-linux---x86-64
[2] https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factorshttps://docs.gitlab.com/ci/pipelines/compute_minutes/#compute-usage-calculationhttps://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors-of-hosted-runners-for-gitlabcom
[3] https://gitlab.com/buildroot.org/gitlab-oss
[4] https://docs.gitlab.com/ci/runners/hosted_runners/#gitlabcom-hosted-runner-workflow
[5] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2416603721
[6] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2562421098
Signed-off-by: Romain Naour <romain.naour@smile.fr>
[Arnout:
- simplify parsing of test_file and test_name;
- match the entire test_name instead of substring;
- assume "small" by default;
- remove the "small" tags;
- use "gitlab-runner" instead of "Gitlab-runner".
]
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
We have our own GitLab-CI runner, but with only one we can't run many
jobs in parallel so it takes a very long time before all the tests have
completed. In addition, if that runner goes down, we have nothing at
all.
GitLab offers the following machine types for hosted runners on Linux
x86-64 [1]. The default is the "small" runner. Using a larger runner
increases the "Cost factor" [2]:
Runner Tag vCPUs Memory Storage Cost factor
saas-linux-small-amd64 (default) 2 8 GB 30 GB 1
saas-linux-medium-amd64 4 16 GB 50 GB 2
saas-linux-large-amd64 (Premium and Ultimate only) 8 32 GB 100 GB 3
saas-linux-xlarge-amd64 (Premium and Ultimate only) 16 64 GB 200 GB 6
saas-linux-2xlarge-amd64 (Premium and Ultimate only) 32 128 GB 200 GB 12
Compute minutes consumed by a job is calculated by:
Job duration / 60 * Cost factor
(Job duration: The time, in seconds, that a job took to run, not
including time spent in the created or pending statuses.)
Thanks to the GitLab OSS program [3], Buildroot benefits from a free
Ultimate subscription and can use GitLab shared runners tagged with
saas-linux-{large, xlarge, 2xlarge}-amd64. In addition, we receive
50,000 free runner minutes per month.
For opensource projects, the Cost factor is reduced by 0,5 (1 minute per
2 minutes of job time).
In order to use one of those tags in Buildroot GitLab-CI jobs, we have
to classify all tests by resource requirement, to make sure the job
doesn't fail because it times out or has insufficient memory or disk
space. We also don't want to use the largest runner for everything
because that would spend the minutes very quickly while the resources
aren't used efficiently (mostly serialized).
First we introduce some new templates used to add the corresponding
runner tag to a runtime test job (reusing the GitLab terminology).
.runner-{small,medium,large,xlarge,2xlarge}
Most of our tests are fast (checkpackage, test_external_bootlin...), so
saas-linux-small-amd64 runner tag is enough. Default to this tag if
nothing else is specified.
Add a comment next to the test class to provide the runner tag.
This runner tag is retrieved when generating the
generated-gitlab-ci.yml file used to create the child pipeline where
the runtime test jobs are executed.
We use the list of runtime tests returned by node2:
"tests.boot.test_edk2.TestEdk2.test_run"
We convert each element of this list to get the path to the test source
file and the name of the test:
"support/testing/tests/boot/test_edk2.py"
TestEdk2
With that, we can grep into the test source file to retrieve the runner
tag placed one line above the test class:
# GitLab-runner: large
class TestEdk2(infra.basetest.BRTest):
Once the runner tag is retrieved, it's used to use the corresponding
runner template to the runtime test job:
tests.boot.test_edk2.TestEdk2.test_run: { extends: [ .runtime_test_base, .runner-large ]}
GitLab runners hosted by the Buildroot project should be able to run any
jobs, so a specific runner tag "buildroot-runner" is used to allow
running a job on such runners. The "buildroot-runner" tag is used in
addition to GitLab ones to run a job on the Buildroot runner or on a
shared GitLab-CI runner.
If a test can't be executed by any shared GitLab-CI runners, we have
to use a runner owned by the Buildroot project. In this case we have
to use a specific template ".runner-buildroot-runner-only". There is no
such runtime test at the moment.
The proposed classification is based on a previous pipeline analysis
[5]:
- Tests lasting more than 3 hours will use 2xlarge runners.
- Tests lasting more than 2 hours will use xlarge runners.
- Tests lasting more than 1 hours will use large runners.
- Tests building a kernel or a toolchain will use medium runners.
- All other tests will use small runners when possible.
CI minute cost estimate:
tests.package.test_clang.TestClangCompilerRT.test_run lasts 4h25 on the
Buildroot runner. If we this duration for 2xlarge runners, the CI
minute consumed would be:
(265 / 60) * 12 * 0.5 = 26,5
With 6 jobs using a 2xlarge runners we used ~160 CI minutes.
tests.package.test_kmscube.TestKmsCube.test_run list 2h04 on the
Buildroot runner. If we this duration for xlarge runners, the CI
minute consumed would be:
(124 / 60) * 6 * 0.5 = 6,2
With 4 jobs using a xlarge runners we used ~37 CI minutes.
tests.package.test_weston.TestWeston.test_run last 1h15 on th
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(75 / 60) * 3 * 0.5 = 1.87
With 28 jobs using a large runners we used ~53 CI minutes.
tests.package.test_gstreamer1.TestGstreamer1.test_run last 46min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(46 / 60) * 2 * 0.5 = 0.76
With 31 jobs using a medium runners we used ~24 CI minutes.
tests.package.test_python.TestPython3Py.test_run last 13min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(13 / 60) * 1 * 0.5 = 0.1
With 691 jobs using a medium runners we used ~69 CI minutes.
In total, one pipeline for the runtime tests cost ~350 CI minutes.
We run such pipeline once a week (on Monday), one for each Buildroot
releases every 3 month, one for each stable and LTS release per month,
and one for each release candidate (3).
Worst case (realse month):
(4 weeks + 1 release + 1 stable + 1 LTS + 3 release candidate) * 350 CI
minutes: 3500 CI minutes / 50000.
So we should be able to run our pipeline without exhausting our CI
minutes credit.
[1] https://docs.gitlab.com/ci/runners/hosted_runners/linux/#machine-types-available-for-linux---x86-64
[2] https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factorshttps://docs.gitlab.com/ci/pipelines/compute_minutes/#compute-usage-calculationhttps://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors-of-hosted-runners-for-gitlabcom
[3] https://gitlab.com/buildroot.org/gitlab-oss
[4] https://docs.gitlab.com/ci/runners/hosted_runners/#gitlabcom-hosted-runner-workflow
[5] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2416603721
Signed-off-by: Romain Naour <romain.naour@smile.fr>
[Arnout:
- simplify parsing of test_file and test_name;
- match the entire test_name instead of substring;
- assume "small" by default;
- remove the "small" tags;
- use "gitlab-runner" instead of "Gitlab-runner".
]
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
This reverts commit 559bb33ae7, which
was disabling BR2_BACKUP_SITE in our tests with the following
reasoning:
support/testing: do not use s.b.o
Currently, the runtime tests will use the sources.buildroot.org backup
mirror, which is the default setup.
However, in some cases we do not want to use the backup site, because we
want to ensure that the download actually works.
This is the case for vendored packages, like cargo or golang packages,
for whbich we want to check that gthe download still works when we
update the rust or go versions, or when our download helpers change.
So, disable the use of the backup site in all runtime tests, and drop
the no-longer needed special cases.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
However, not using the backup site is causing a LOT of random failures
in Gitlab CI, making the results barely usable. Since it's more
important to get real failures than random ones due to download
issues, let's re-enable BR2_BACKUP_SITE in our testing infrastructure.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Currently, the runtime tests will use the sources.buildroot.org backup
mirror, which is the default setup.
However, in some cases we do not want to use the backup site, because we
want to ensure that the download actually works.
This is the case for vendored packages, like cargo or golang packages,
for whbich we want to check that gthe download still works when we
update the rust or go versions, or when our download helpers change.
So, disable the use of the backup site in all runtime tests, and drop
the no-longer needed special cases.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently the tests TestRust and TestRustBin do check for vendoring by
requiring package ripgrep to be built but only if the download
directory is already empty, otherwise the existing contents of the download
directory will be reused and therefore not be redownloaded.
This new test will only verify that the required packages are downloaded
and vendored correctly without doing a runtime test. It does so by setting a
path to a folder "dl" inside the build directory (output-directory/testname/)
and then setting the environment variable BR2_DL_DIR to this path before the
build starts. BR2_DL_DIR is not set in the config options because it would be
overridden by the user's own environment variable if defined. This code was
essentially copied from the file test_gitforge.py which was added in commit
1ca6ab6ace
We want the package ripgrep to be built since it requires vendoring
directly. Additionally we want the package python-cryptography to be
built because it has rust dependencies and therefore indirectly also requires
vendoring.
Signed-off-by: Sebastian Weyer <sebastian.weyer@smile.fr>
Signed-off-by: Antoine Coutant <antoine.coutant@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
TestRust and TestRustBin has been introduced at the time when there was
no cargo package infrastructure or any package using rust compiler
(Buildroot 2018.02).
Since then the ripgrep package has been introduced, initially using
the generic package infrastructure and converted later to the cargo
package infrastructure.
Due a recent change in rust/cargo removing the cargo config file [1]
the test TestRust and TestRustBin now fail to compile since they build
an hello-world crate outside of the cargo package infrastructure
without the correct environment for cross-compiling.
Replace the 'hello-world' crate by ripgrep package and check if it
can run properly in Qemu.
Fixes tests.package.test_rust.TestRustBin:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2116202545
But doesn't fixes tests.package.test_rust.TestRust due another bug:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2116202544
[1] b6378631c2
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This test already use builtin kernel provided by the testsuite infra:
self.emulator.boot(arch="armv7",
kernel="builtin",
options=["-initrd", img])
But a second kernel is build from the its defconfig. This second kernel
is not used by the test.
The TestRust (using BR2_PACKAGE_HOST_RUST=y) is really long to build,
save some cpu time by removing the kernel build.
This unused kernel (based on 4.11.3 release) doesn't even build with
host gcc >= 10.
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The BRTest() class implements an assertRunOk() method that does the
very common work of running a command inside the emulator, and
checking that it is successful.
This commit changes all locations where this .assertRunOk() method can
be used, instead of open-coding the same logic.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Since commit "2927f412be support/testing: standardize defconfig
fragments style" all other test cases use the same style for defconfig
fragments:
- start after a backslash;
- be declared as a multi-line string literal;
- be indented one level more than the variable that contains it.
Do the same here for consistency.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Matt Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When the run-time tests to build rust and rust-bin packages are run via Docker,
the $USER environment variable is not set, which makes cargo fail when
initializing the test project.
So add it to make cargo happy.
Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
To test the support for the Rust language, the following tests are added:
- building Rust compiler and Cargo from source.
- installing a pre-built Rust compiler and building Cargo from source.
For each test, a Rust test program is built and installed in the root file
system of a ARM vexpress QEMU system. The test is declared OK if the program can
be run properly from the test system.
Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
