We have our own GitLab-CI runner, but with only one we can't run many
jobs in parallel so it takes a very long time before all the tests have
completed. In addition, if that runner goes down, we have nothing at
all.
GitLab offers the following machine types for hosted runners on Linux
x86-64 [1]. The default is the "small" runner. Using a larger runner
increases the "Cost factor" [2].
For opensource projects, the Cost factor is reduced to 0.5 (1 minute per
2 minutes of job time) whatever the runner type.
Runner Tag vCPUs Memory Storage Cost factor (OSS)
saas-linux-small-amd64 (default) 2 8 GB 30 GB 1 (0.5)
saas-linux-medium-amd64 4 16 GB 50 GB 2 (0.5)
saas-linux-large-amd64 (Premium and Ultimate only) 8 32 GB 100 GB 3 (0.5)
saas-linux-xlarge-amd64 (Premium and Ultimate only) 16 64 GB 200 GB 6 (0.5)
saas-linux-2xlarge-amd64 (Premium and Ultimate only) 32 128 GB 200 GB 12 (0.5)
Compute minutes consumed by a job is calculated by:
Job duration / 60 * Cost factor
(Job duration: The time, in seconds, that a job took to run, not
including time spent in the created or pending statuses.)
Thanks to the GitLab OSS program [3], Buildroot benefits from a free
Ultimate subscription and can use GitLab shared runners tagged with
saas-linux-{large, xlarge, 2xlarge}-amd64. In addition, we receive
50,000 free runner minutes per month.
In order to use one of those tags in Buildroot GitLab-CI jobs, we have
to classify all tests by resource requirement, to make sure the job
doesn't fail because it times out or has insufficient memory or disk
space. While we usually shouldn't use the largest runner for
everything, we can use larger runner without cost penalty thanks to
the cost factor reduced to 0.5 for opensource projects. This will
reduce the CI minutes consumed by a CPU intensive job.
First we introduce some new templates used to add the corresponding
runner tag to a runtime test job (reusing the GitLab terminology).
.runner-{small,medium,large,xlarge,2xlarge}
Most of our tests are fast (checkpackage, test_external_bootlin...), so
saas-linux-small-amd64 runner tag is enough. Default to this tag if
nothing else is specified.
Add a comment next to the test class to provide the runner tag.
This runner tag is retrieved when generating the
generated-gitlab-ci.yml file used to create the child pipeline where
the runtime test jobs are executed.
We use the list of runtime tests returned by node2:
"tests.boot.test_edk2.TestEdk2.test_run"
We convert each element of this list to get the path to the test source
file and the name of the test:
"support/testing/tests/boot/test_edk2.py"
TestEdk2
With that, we can grep into the test source file to retrieve the runner
tag placed one line above the test class:
# GitLab-runner: large
class TestEdk2(infra.basetest.BRTest):
Once the runner tag is retrieved, it's used to use the corresponding
runner template to the runtime test job:
tests.boot.test_edk2.TestEdk2.test_run: { extends: [ .runtime_test_base, .runner-large ]}
GitLab runners hosted by the Buildroot project should be able to run
any jobs, so they should be tagged with Gitlab runner tags
(saas-linux-{small,medium,large,xlarge,2xlarge}-amd64).
A specific runner tag "buildroot-runner" can be used to allow running
a job only on such runners.
If a test can't be executed by any shared GitLab-CI runners, we have
to use a runner owned by the Buildroot project. In this case we have
to use a specific template ".runner-buildroot-runner-only" in order to
add the specific runner tag "buildroot-runner" to the job running the
test. There is no such runtime test at the moment.
The proposed classification is based on a previous pipeline analysis
[5]:
- Tests lasting more than 3 hours will use 2xlarge runners.
- Tests lasting more than 2 hours will use xlarge runners.
- Tests lasting more than 1 hours will use large runners.
- Tests building a kernel or a toolchain will use medium runners.
- All other tests will use small runners when possible.
CI minute cost estimate:
tests.package.test_clang.TestClangCompilerRT.test_run lasts 4h25 on the
Buildroot runner. If we this duration for 2xlarge runners, the CI
minute consumed would be:
(15900 / 60) * 0.5 = 133
With 6 jobs using a 2xlarge runners we used ~795 CI minutes.
tests.package.test_kmscube.TestKmsCube.test_run list 2h04 on the
Buildroot runner. If we this duration for xlarge runners, the CI
minute consumed would be:
(7440 / 60) * 0.5 = 62
With 4 jobs using a xlarge runners we used ~248 CI minutes.
tests.package.test_weston.TestWeston.test_run last 1h15 on th
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(4500 / 60) * 0.5 = 37.5
With 28 jobs using a large runners we used ~1050 CI minutes.
tests.package.test_gstreamer1.TestGstreamer1.test_run last 46min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(2760 / 60) * 0.5 = 23
With 31 jobs using a medium runners we used ~713 CI minutes.
tests.package.test_python.TestPython3Py.test_run last 13min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(780 / 60) * 0.5 = 6.5
With 691 jobs using a medium runners we used ~4491 CI minutes.
In total, one pipeline for the runtime tests cost ~7297 CI minutes.
After a first try [6], we are actually using 8000 CI minutes per
pipeline.
We run such pipeline once a week (on Monday), one for each Buildroot
releases every 3 month, one for each stable and LTS release per month,
and one for each release candidate (3).
Worst case (release month):
(4 weeks + 1 release + 1 stable + 1 LTS + 3 release candidate) * 8000 CI
minutes: 80000 CI minutes / 50000.
So we would spend the minutes very quickly in the worst case scenario.
We have to keep one pipeline under 5000 CI minutes.
[1] https://docs.gitlab.com/ci/runners/hosted_runners/linux/#machine-types-available-for-linux---x86-64
[2] https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factorshttps://docs.gitlab.com/ci/pipelines/compute_minutes/#compute-usage-calculationhttps://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors-of-hosted-runners-for-gitlabcom
[3] https://gitlab.com/buildroot.org/gitlab-oss
[4] https://docs.gitlab.com/ci/runners/hosted_runners/#gitlabcom-hosted-runner-workflow
[5] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2416603721
[6] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2562421098
Signed-off-by: Romain Naour <romain.naour@smile.fr>
[Arnout:
- simplify parsing of test_file and test_name;
- match the entire test_name instead of substring;
- assume "small" by default;
- remove the "small" tags;
- use "gitlab-runner" instead of "Gitlab-runner".
]
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
We have our own GitLab-CI runner, but with only one we can't run many
jobs in parallel so it takes a very long time before all the tests have
completed. In addition, if that runner goes down, we have nothing at
all.
GitLab offers the following machine types for hosted runners on Linux
x86-64 [1]. The default is the "small" runner. Using a larger runner
increases the "Cost factor" [2]:
Runner Tag vCPUs Memory Storage Cost factor
saas-linux-small-amd64 (default) 2 8 GB 30 GB 1
saas-linux-medium-amd64 4 16 GB 50 GB 2
saas-linux-large-amd64 (Premium and Ultimate only) 8 32 GB 100 GB 3
saas-linux-xlarge-amd64 (Premium and Ultimate only) 16 64 GB 200 GB 6
saas-linux-2xlarge-amd64 (Premium and Ultimate only) 32 128 GB 200 GB 12
Compute minutes consumed by a job is calculated by:
Job duration / 60 * Cost factor
(Job duration: The time, in seconds, that a job took to run, not
including time spent in the created or pending statuses.)
Thanks to the GitLab OSS program [3], Buildroot benefits from a free
Ultimate subscription and can use GitLab shared runners tagged with
saas-linux-{large, xlarge, 2xlarge}-amd64. In addition, we receive
50,000 free runner minutes per month.
For opensource projects, the Cost factor is reduced by 0,5 (1 minute per
2 minutes of job time).
In order to use one of those tags in Buildroot GitLab-CI jobs, we have
to classify all tests by resource requirement, to make sure the job
doesn't fail because it times out or has insufficient memory or disk
space. We also don't want to use the largest runner for everything
because that would spend the minutes very quickly while the resources
aren't used efficiently (mostly serialized).
First we introduce some new templates used to add the corresponding
runner tag to a runtime test job (reusing the GitLab terminology).
.runner-{small,medium,large,xlarge,2xlarge}
Most of our tests are fast (checkpackage, test_external_bootlin...), so
saas-linux-small-amd64 runner tag is enough. Default to this tag if
nothing else is specified.
Add a comment next to the test class to provide the runner tag.
This runner tag is retrieved when generating the
generated-gitlab-ci.yml file used to create the child pipeline where
the runtime test jobs are executed.
We use the list of runtime tests returned by node2:
"tests.boot.test_edk2.TestEdk2.test_run"
We convert each element of this list to get the path to the test source
file and the name of the test:
"support/testing/tests/boot/test_edk2.py"
TestEdk2
With that, we can grep into the test source file to retrieve the runner
tag placed one line above the test class:
# GitLab-runner: large
class TestEdk2(infra.basetest.BRTest):
Once the runner tag is retrieved, it's used to use the corresponding
runner template to the runtime test job:
tests.boot.test_edk2.TestEdk2.test_run: { extends: [ .runtime_test_base, .runner-large ]}
GitLab runners hosted by the Buildroot project should be able to run any
jobs, so a specific runner tag "buildroot-runner" is used to allow
running a job on such runners. The "buildroot-runner" tag is used in
addition to GitLab ones to run a job on the Buildroot runner or on a
shared GitLab-CI runner.
If a test can't be executed by any shared GitLab-CI runners, we have
to use a runner owned by the Buildroot project. In this case we have
to use a specific template ".runner-buildroot-runner-only". There is no
such runtime test at the moment.
The proposed classification is based on a previous pipeline analysis
[5]:
- Tests lasting more than 3 hours will use 2xlarge runners.
- Tests lasting more than 2 hours will use xlarge runners.
- Tests lasting more than 1 hours will use large runners.
- Tests building a kernel or a toolchain will use medium runners.
- All other tests will use small runners when possible.
CI minute cost estimate:
tests.package.test_clang.TestClangCompilerRT.test_run lasts 4h25 on the
Buildroot runner. If we this duration for 2xlarge runners, the CI
minute consumed would be:
(265 / 60) * 12 * 0.5 = 26,5
With 6 jobs using a 2xlarge runners we used ~160 CI minutes.
tests.package.test_kmscube.TestKmsCube.test_run list 2h04 on the
Buildroot runner. If we this duration for xlarge runners, the CI
minute consumed would be:
(124 / 60) * 6 * 0.5 = 6,2
With 4 jobs using a xlarge runners we used ~37 CI minutes.
tests.package.test_weston.TestWeston.test_run last 1h15 on th
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(75 / 60) * 3 * 0.5 = 1.87
With 28 jobs using a large runners we used ~53 CI minutes.
tests.package.test_gstreamer1.TestGstreamer1.test_run last 46min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(46 / 60) * 2 * 0.5 = 0.76
With 31 jobs using a medium runners we used ~24 CI minutes.
tests.package.test_python.TestPython3Py.test_run last 13min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(13 / 60) * 1 * 0.5 = 0.1
With 691 jobs using a medium runners we used ~69 CI minutes.
In total, one pipeline for the runtime tests cost ~350 CI minutes.
We run such pipeline once a week (on Monday), one for each Buildroot
releases every 3 month, one for each stable and LTS release per month,
and one for each release candidate (3).
Worst case (realse month):
(4 weeks + 1 release + 1 stable + 1 LTS + 3 release candidate) * 350 CI
minutes: 3500 CI minutes / 50000.
So we should be able to run our pipeline without exhausting our CI
minutes credit.
[1] https://docs.gitlab.com/ci/runners/hosted_runners/linux/#machine-types-available-for-linux---x86-64
[2] https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factorshttps://docs.gitlab.com/ci/pipelines/compute_minutes/#compute-usage-calculationhttps://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors-of-hosted-runners-for-gitlabcom
[3] https://gitlab.com/buildroot.org/gitlab-oss
[4] https://docs.gitlab.com/ci/runners/hosted_runners/#gitlabcom-hosted-runner-workflow
[5] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2416603721
Signed-off-by: Romain Naour <romain.naour@smile.fr>
[Arnout:
- simplify parsing of test_file and test_name;
- match the entire test_name instead of substring;
- assume "small" by default;
- remove the "small" tags;
- use "gitlab-runner" instead of "Gitlab-runner".
]
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
BR2_LINUX_KERNEL_NEEDS_HOST_LIBELF=y :
While building the kernel tools, libelf header is missing because objtools
was not compiled by default on older kernel versions. This results in the following error:
[...]tools/objtool/include/objtool/elf.h:10:10: fatal error: gelf.h: No such file or directory
10 | #include <gelf.h>
That is why BR2_LINUX_KERNEL_NEEDS_HOST_LIBELF was selected to build host-libelf.
Signed-off-by: Dowan Gullient <dowan.gullient@smile.fr>
[Julien: updated kernel to 6.18.21]
Signed-off-by: Julien Olivain <ju.o@free.fr>
Python setuptools has removed the obsolete pkg_resources python module in
v82.0.0. This module is used by the binman command in U-Boot until
v2025.10.
Since the python-setuptools package has been updated to v82.0.0 in
Buildroot, the python test tests.boot.test_atf.TestATFAllwinner (using
U-Boot v2023.10 and binman) fails to build with the following error [1]:
ModuleNotFoundError: No module named 'pkg_resources'
Update test_atf to use a more recent U-Boot v2026.01 (and add the
dependency on GNU TLS), to fix the build.
Link: https://gitlab.com/buildroot.org/buildroot/-/jobs/13500946337 [1]
Fixes: 51365ff063 ("package/python-setuptools: bump to version 82.0.0")
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>
Cc: James Hilliard <james.hilliard1@gmail.com>
Cc: Julien Olivain <ju.o@free.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
EDK2 removed OVMF IA32 support in commit [1], which is included in
version edk2-stable202511. This commit removes the test relying on it.
[1] 1fb88ffe28
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
The SBSA Reference Platform was updated to neoverse-n2 (armv9.0a)
in Qemu v9.1 [1].
The Armv9-A and Neoverse N2 core support was added in Buildroot
commit [2].
Since the test_edk2 uses the Qemu sbsa-ref machine, this commit
aligns it to use the neoverse-n2 cpu.
[1] b1d592e7b0
[2] 9845bd4541
Signed-off-by: Julien Olivain <ju.o@free.fr>
This toolchain is more than 7 years old now, and not maintained anymore.
Let's therefore remove it from Buildroot. There are still some
alternatives in-tree, such as ARM and Bootlin toolchains
Signed-off-by: Titouan Christophe <titouan.christophe@mind.be>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
The syslinux runtime test (which is in fact a build-only test) ensures
that syslinux does get build at least once a week (via the gitlab-CI
weekly pipeline). Runtime testing would need much more work, though, but
nothing in syslinux is currently runtime tested anyway.
Reported-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
For release notes since edk2-stable202411, see:
https://github.com/tianocore/edk2/releases/tag/edk2-stable202502https://github.com/tianocore/edk2/releases/tag/edk2-stable202505
Note: the edk2-stable202505 release note mention a security fix.
This fix was already backported in Buildroot in a previous commit.
For that reason, this bump is not marked as securiy, and also
removes the backported patch and its associated _IGNORE_CVES entry.
Those versions include fixes when using gcc 15 (host and target).
This commit also updates the edk2-platforms and edk2-non-osi packages
with the last commit merged at the edk2 release date (2025-05-23).
In the case of edk2-platforms, the commit id used in this update
is few days later, to include a fix for the Arm Sgi575 platform.
Exact commit should have been [1] (to be exactly aligned to the edk2
tag date), but commit [2] is used instead. The package edk2-non-osi is
updated to commit [3].
Also, the SocioNext SynQuacer platforms (including the DeveloperBox)
were removed upstream in edk2-platforms commit [4]. This commit
reflect that change by removing those platforms choices and associated
runtime tests.
This commit has been runtime tested with tests using EDK2 package,
with commands:
support/testing/run-tests \
-d dl -o output_folder \
tests.boot.test_edk2 \
tests.boot.test_grub.TestGrubAArch64EFI \
tests.boot.test_grub.TestGrubi386EFI \
tests.boot.test_grub.TestGrubRiscV64EFI \
tests.boot.test_grub.TestGrubX8664EFI \
tests.fs.test_iso9660.TestIso9660Grub2EFI \
tests.fs.test_iso9660.TestIso9660Grub2Hybrid \
tests.package.test_fwts
It has also been runtime tested (by booting in qemu) with defconfigs
using EDK2 package:
qemu_aarch64_sbsa_defconfig
qemu_riscv64_virt_efi_defconfig
[1] 2bfe3fa99e
[2] 92f7d69e8a
[3] ea2040c2d4
[4] aaba5807f1
Cc: Dick Olsson <hi@senzilla.io>
Cc: Romain Naour <romain.naour@smile.fr>
Cc: Vincent Stehlé <vincent.stehle@arm.com>
Tested-by: Vincent Stehlé <vincent.stehle@arm.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
Some EDK II configurations have complex dependencies on several packages
and additional build options; build tests help keeping track of those
more easily.
Factorize some code common to all the build tests into a new
TestEdk2BuildBase class, which defines a base configuration and a method
to assert that binaries do indeed exist after the build.
While at it, add myself in DEVELOPERS.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>
Cc: Dick Olsson <hi@senzilla.io>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since commit [1], test_atf is failing. See [2].
Passing -no-pie to ld does not produce the bl31.elf file (and does not
generate an error). TF-A 2.11 reworked its makefiles and switched to
gcc to link files. This commit fixes this failing test by updating to
this newer TF-A version.
Fixes: [2].
[1] 248c2d045e
[2] https://gitlab.com/buildroot.org/buildroot/-/jobs/8609307891
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Remove the download fix introduced by [1] now the edk2 git repository
has been fixed in version stable202411 [2].
[1] 47fc9e5509.
[2] 95d8a1c255
Cc: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
Since edk2-202408, ATF >= 2.12 is required for qemu aarch64 sbsa target
to support CPU topology information provided by edk2 firmware [1].
If ATF <= 2.11 is used, the boot will hand with the following error:
sbsa_sip_smc_handler: unhandled SMC (0xc20000ca) (function id: 202)
In order to update edk2 package to a newer version, update ATF version
used by TestEdk2 to 2.12.
[1] c891b4d835
Cc: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
The SBSA Reference Platform was updated to use the neoverse-n1 cpu
(armv8.2a) in Qemu v8.1 [1].
Commit c58a2a968 "configs/qemu_aarch64_sbsa_defconfig: switch to
neoverse-n1 (armv8.2a)" [2] reflected this change in
qemu_aarch64_sbsa_defconfig.
This commit updates the test_edk2 runtime test the same way.
This update is possible with the help of commits [3] and [4].
[1] 1877272bad
[2] c58a2a9687
[3] 0d4177598c
[4] fba62f2ab1
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Commit f439b47ed "boot/grub2: add RISC-V 64bit EFI support" [1]
introduced the grub2 RISC-V UEFI support.
At that time, it was not possible to have a runtime test for this.
qemu-system-riscv64 is not in the Buildroot Docker reference image,
and the base distribution (Debian 11) does not provide a
sufficiently recent version. A qemu-system-riscv64 >= 8.0.0 is needed.
Commit 0d4177598 "support/testing/infra/emulator.py: add build host
dir to qemu search path" [2] added the capability to use the
Buildroot host-qemu package inside runtime test.
With this support, it is now possible to add a runtime test for a
grub2 UEFI boot on RISC-V 64bit.
Also, due to an upstream edk2 git submodule issue, this commit
includes a workaround to use https://sources.buildroot.net for this
test. See [3].
[1] f439b47ed6
[2] 0d4177598c
[3] 47fc9e5509
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The EDK2 bump to stable202405 introduced an incompatibility with the
Qemu version 5.2.0 present in the reference Docker image. For that
reason, this test was disabled.
This commit enables host-qemu in the build, to be used by the runtime
test infrastructure.
Since this commit switch to a more recent host-qemu, it also removes
firmware options to workaround old qemu bugs.
Those changes are fixing this test, which is why this commit re-enables
it by removing the skipTest() invocation.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
The EDK2 bump to stable202405 introduced an incompatibility with the
Qemu version 5.2.0 present in the reference Docker image. For that
reason, this test was disabled.
This commit updates ATF to 2.11 and kernel to 6.6.58, and enables
host-qemu in the build, to be used by the runtime test infrastructure.
Those changes are fixing this test, which is why this commit re-enables
it by removing the skipTest() invocation.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
EDK2 version edk2-stable202405 (not yet in Buildroot) has dependencies
on specific Qemu version.
Buildroot guidelines also recommend that "each patch must be complete".
See: [1]. This means, a patch in a series is not allowed to break
something knowingly, even in the same patch series. This rule is
mainly for using git bisect.
We also want to avoid bumping EDK2, update the runtime test infra,
and few tests in a big single commit, for readability. This would mix
too many changes in the same commit.
As a trade off, this commit disables the runtime tests which are known
to be broken by the upcoming edk2 update. Then, those tests will be
individually fixed and re-enabled in followup commits.
This EDK2/Qemu incompatibility only affects runtime tests using EDK2
on the Arm Aarch64 architecture. This commit disable the runtime
tests by adding a "skipTest()" call in:
- tests.boot.test_edk2
- tests.boot.test_grub.TestGrubAArch64EFI
- tests.package.test_fwts
[1] https://buildroot.org/downloads/manual/manual.html#submitting-patches
Signed-off-by: Julien Olivain <ju.o@free.fr>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
TestATFVexpress is using vexpress_aemv8a_juno as as u-boot defconfig
but the Buildroot defconfig of this board was removed in 2022.11 [1]
Since both TestATFVexpress and TestATFAllwinner are now using mainline
ATF, we don't really need several ATF test anymore. Initially [2],
several runtime test were added to test ATF/U-Boot combinations when
ATF was provided by a vendor: vexpress (mainline), Allwinner and
Marvell.
Keep TestATFAllwinner as ATF mainline test.
[1] 347c108738
[2] 8cf3ce04e9
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
u-boot-2021.04 seems to be broken when pylibfdt support is enabled
and the latest python3/setuptools are used.
Since the TestATFAllwinner is using bananapi_m64 as u-boot defconfig
but the Buildroot defconfig of this board was removed in 2022.11 [1]
update TestATFAllwinner to use a newer BSP. Use the one provided
by orangepi_zero_plus2_defconfig.
[1] daf3c6661f
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/6477656317 (TestATFAllwinner)
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In commit 7e0e6e3b86 (toolchain/toolchain-external/toolchain-external-bootlin:
update to 2023.11-1) the TestGrubX8664EFI fails with this error:
# efivar -l
traps: efivar[86] trap invalid opcode ip:7fc187f4c7f4 sp:7fff9bbaa930 error:0 in libefivar.so.1.38[7fc187f4c000+16000]
Illegal instruction
This error can be reproduced by installing other packages like "file".
The 2023.11-1 Bootlin toolchains are built for a corei7 CPU [0], which
is in fact a Nehalem CPU; we switched to the new names in commit
653fa001f3 (arch/Config.in.x86: add "newer" names for several Intel
x86 CPU variants). This means that the Bootlin toolchains may use
Nehalem-specific instructions.
The TestGrubX8664EFI test is also setup for BR2_x86_corei7, so our
executables will also contain Nehalem instructions.
However, the default Qemu x86_64 is not guaranteed to emulate all the
instructions specific to Nehalem, causing runtime issues as reported
above.
A similar issue has been fixed in toolchain-builder by adding Nehalem
cpu emulation on the qemu command line [0].
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/6093853712
[0] f2b253732b
Signed-off-by: Romain Naour <romain.naour@smile.fr>
[yann.morin.1998@free.fr:
- reword the commit log to explain corei7 -> Nehalem equivalence
- note that the toolchain-builder *and* our test target corei7, thus
Nehalem
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Following the toolchain-external-arm-aarch64 bump to version 12.2-rel1
by commit [1], binutils 2.39 is now provided by the toolchain.
This new binutils version break the ATF build due to new linker warnings as
reported by the EDK2 test:
ld.bfd: warning: bl2.elf has a LOAD segment with RWX permissions
From [2]
"Users of GNU ld (BPF) from binutils 2.39+ will observe multiple instaces
of a new warning when linking the bl*.elf in the form:
ld.bfd: warning: stm32mp1_helper.o: missing .note.GNU-stack section implies executable stack
ld.bfd: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
ld.bfd: warning: bl2.elf has a LOAD segment with RWX permissions
ld.bfd: warning: bl32.elf has a LOAD segment with RWX permissions
These new warnings are enbaled by default to secure elf binaries:
- https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=ba951afb99912da01a6e8434126b8fac7aa75107
- https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=0d38576a34ec64a1b4500c9277a8e9d0f07e6774
"
Bump the ATF custom version to 2.9 for binutils 2.39+ support.
[1] 90404e84c1
[2] 1f49db5f25
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/4581325484
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The kernel config board/qemu/aarch64-sbsa/linux.config has never been in
use by qemu_aarch64_sbsa_defconfig, neither via
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE, nor via
BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES.
test_edk2.py is using the kernel config
board/qemu/aarch64-sbsa/linux.config. However, storing a kernel config
that is not used by qemu_aarch64_sbsa_defconfig, in a directory that is
"owned" by qemu_aarch64_sbsa_defconfig, is bound to cause confusion.
Therefore, move the config file to a new subdirectory:
support/testing/tests/boot/test_edk2/
This is similar to how e.g. test_grub.py has a subdirectory:
support/testing/tests/boot/test_grub/
where it keeps the kernel config that is only used by test_grub.py.
Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adds four test cases for Grub:
- Grub i386 legacy BIOS
- Grub i386 UEFI
- Grub x86-64 UEFI
- Grub AArch64 UEFI
There is some overlap with the ISO9660 filesystem test cases, some of
which use Grub, but we found it relevant to have separate test cases
for Grub, which were useful to test Grub in non-ISO9660 situations.
The Grub ARM UEFI case is not tested, as it requires Grub to be
chain-loaded by U-Boot. Implementing this test case is left as an
exercise for the reader.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- use EDK2 to build the OVMF blurbs from source, instead of the
binary blobs
- add host-dosfstools
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This version bump is needed to pass the ATF test with
hardening option enabled (-fstack-protector-strong)
With the version v2.2, ATF fail due to undefined references:
./build/juno/release/bl2u/arm_tzc400.o: In function `arm_tzc400_setup':
arm_tzc400.c:(.text.arm_tzc400_setup+0x10): undefined reference to `__stack_chk_guard'
arm_tzc400.c:(.text.arm_tzc400_setup+0x18): undefined reference to `__stack_chk_guard'
arm_tzc400.c:(.text.arm_tzc400_setup+0xb8): undefined reference to `__stack_chk_guard'
arm_tzc400.c:(.text.arm_tzc400_setup+0xcc): undefined reference to `__stack_chk_fail'
Since commit ccac9a5bbb, Buildroot no
longer forces ENABLE_STACK_PROTECTOR. However, we rely on the ATF build
system to handle it correctly, and this wasn't the case in v2.2.
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/1524842591
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Bump U-Boot version in TestATFAllwinner. Updating U-Boot version to
2021.04 requires the following two changes.
First, after switching to binman, u-boot.itb is no more generated for
64-bit sunxi boards. Combined u-boot-sunxi-with-spl.bin image should
be used instead. This image contains SPL, U-Boot, and FIT image,
where FIT image contains other binaries such as BL31 and SCP.
Second, new U-Boot enables support for System Control Processor (SCP)
firmware. SCP firmware is included by default into FIT image in the
combined u-boot-sunxi-with-spl.bin binary. When SCP is not available
or not needed, it should be explicitly disabled by pointing to an
empty file. Support for Allwinner SCP firmware is not yet available
neither in Buildroot nor in mainline kernel. So disable it for now
using custom U-Boot build options.
Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Switch to mainline TF-A that provides basic support for H5 and A64.
Note that Allwinner platform layer in TF-A does not provide support
for GCC stack protection, so make sure to disable this TF-A feature.
Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Boot a QEMU sbsa-ref machine with ATF, EDK2, GRUB2 and a minimal
kernel. This is a simple but effective test of a compliant setup.
Signed-off-by: Dick Olsson <hi@senzilla.io>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Fix U-Boot config snippet in TestATFAllwinner. Bump U-Boot
version to fix DTC build on hosts with gcc 10 and add
pylibfdt dependency.
Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
There were only two users of Marvell ATF: SolidRun MacchiatoBin board
and ClearFrog GT 8k board. After mv-ddr-marvell package update both
boards switched to upstream ATF. Remove tests for now unused
Marvell ATF.
Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since the bump of ATF to 2.2 for the ATF Vexpress test case in commit
fc3d6a3ed0
("support/testing/tests/boot/test_atf: update U-Boot/ATF use in
TestATFVexpress"), DTC is now needed otherwise the build fails with:
make[2]: dtc: Command not found
Makefile:873: recipe for target 'build/juno/release/fdts/juno_tb_fw_config.dtb' failed
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/674934470
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The BR2_TOOLCHAIN_EXTERNAL_LOCALE is only defined for uClibc based
custom external toolchains.
The test_syslinux is using a glibc toolchain, so the
"# BR2_TOOLCHAIN_EXTERNAL_LOCALE is not set" line disapear from
the generated dot config (.config) leading to an error due to
a new check in the testsuite infra.
There is the same problem with:
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_NPTL=y
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adds four new tests for the syslinux bootloader:
- Building on x86, for legacy BIOS
- Building on x86, for EFI BIOS
- Building on x86-64, for legacy BIOS
- Building on x86-64, for EFI BIOS
Runtime testing in Qemu would certainly be possible, but is left as a
future addition to these tests.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since release 18.12 binaries-marvell repository provides
common firmware supporting both A7K and A8K SoC families.
This commit bumps package version to 18.12 and removes
platform specific binary selections from Config.in.
Single firmware image suitable for both A7K and A8K
platforms is now specified in mk file explicitely.
Legacy handling is not needed, as configs which did have
the option set will continue to work without change.
Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
[Arnout:
- Remove BINARIES_MARVELL_IMAGE entirely;
- Add remark about legacy handling;
- Remove the deprecated option from the defconfigs and test that use
it.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This test case uses a too old U-Boot version, which is affected by the
infamous libfdt header conflict issue. We update U-Boot and ATF to
what is used in the current version of
solidrun_macchiatobin_mainline_defconfig, for which the problem no
longer exists.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/107860312
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The version of the ARM Trusted Firmware from Marvell was a Git branch,
not a Git commit, leading to unreproducible results. So let's use a
Git commit instead, which is the latest available from the branch that
was previously used.
More specifically, this branch has recently seen a fix that is needed
for ATF to build properly with recent gcc versions:
c96ec59f8b
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
