mirror/buildroot
1
0
Fork 0
mirror of https://gitlab.com/buildroot.org/buildroot.git synced 2026-09-09 07:51:59 -09:00
Code Issues Packages Projects Releases Wiki Activity
83,367 Commits 6 Branches 483 Tags
master
Commit Graph

7 Commits

Author SHA1 Message Date
Romain Naour
c9df1b64b2 support/testing: generate runtime test jobs with runner tags 
We have our own GitLab-CI runner, but with only one we can't run many
jobs in parallel so it takes a very long time before all the tests have
completed. In addition, if that runner goes down, we have nothing at
all.

GitLab offers the following machine types for hosted runners on Linux
x86-64 [1]. The default is the "small" runner. Using a larger runner
increases the "Cost factor" [2].

For opensource projects, the Cost factor is reduced to 0.5 (1 minute per
2 minutes of job time) whatever the runner type.

Runner Tag                                             vCPUs   Memory  Storage  Cost factor (OSS)
saas-linux-small-amd64 (default)                           2     8 GB    30 GB     1 (0.5)
saas-linux-medium-amd64                                    4    16 GB    50 GB     2 (0.5)
saas-linux-large-amd64 (Premium and Ultimate only)         8    32 GB   100 GB     3 (0.5)
saas-linux-xlarge-amd64 (Premium and Ultimate only)       16    64 GB   200 GB     6 (0.5)
saas-linux-2xlarge-amd64 (Premium and Ultimate only)      32   128 GB   200 GB    12 (0.5)

Compute minutes consumed by a job is calculated by:

  Job duration / 60 * Cost factor

(Job duration: The time, in seconds, that a job took to run, not
including time spent in the created or pending statuses.)

Thanks to the GitLab OSS program [3], Buildroot benefits from a free
Ultimate subscription and can use GitLab shared runners tagged with
saas-linux-{large, xlarge, 2xlarge}-amd64. In addition, we receive
50,000 free runner minutes per month.

In order to use one of those tags in Buildroot GitLab-CI jobs, we have
to classify all tests by resource requirement, to make sure the job
doesn't fail because it times out or has insufficient memory or disk
space. While we usually shouldn't use the largest runner for
everything, we can use larger runner without cost penalty thanks to
the cost factor reduced to 0.5 for opensource projects. This will
reduce the CI minutes consumed by a CPU intensive job.

First we introduce some new templates used to add the corresponding
runner tag to a runtime test job (reusing the GitLab terminology).

    .runner-{small,medium,large,xlarge,2xlarge}

Most of our tests are fast (checkpackage, test_external_bootlin...), so
saas-linux-small-amd64 runner tag is enough. Default to this tag if
nothing else is specified.

Add a comment next to the test class to provide the runner tag.
This runner tag is retrieved when generating the
generated-gitlab-ci.yml file used to create the child pipeline where
the runtime test jobs are executed.

We use the list of runtime tests returned by node2:

  "tests.boot.test_edk2.TestEdk2.test_run"

We convert each element of this list to get the path to the test source
file and the name of the test:

  "support/testing/tests/boot/test_edk2.py"

  TestEdk2

With that, we can grep into the test source file to retrieve the runner
tag placed one line above the test class:

  # GitLab-runner: large
  class TestEdk2(infra.basetest.BRTest):

Once the runner tag is retrieved, it's used to use the corresponding
runner template to the runtime test job:

  tests.boot.test_edk2.TestEdk2.test_run: { extends: [ .runtime_test_base, .runner-large ]}

GitLab runners hosted by the Buildroot project should be able to run
any jobs, so they should be tagged with Gitlab runner tags
(saas-linux-{small,medium,large,xlarge,2xlarge}-amd64).
A specific runner tag "buildroot-runner" can be used to allow running
a job only on such runners.

If a test can't be executed by any shared GitLab-CI runners, we have
to use a runner owned by the Buildroot project. In this case we have
to use a specific template ".runner-buildroot-runner-only" in order to
add the specific runner tag "buildroot-runner" to the job running the
test. There is no such runtime test at the moment.

The proposed classification is based on a previous pipeline analysis
[5]:

  - Tests lasting more than 3 hours will use 2xlarge runners.
  - Tests lasting more than 2 hours will use xlarge runners.
  - Tests lasting more than 1 hours will use large runners.
  - Tests building a kernel or a toolchain will use medium runners.
  - All other tests will use small runners when possible.

CI minute cost estimate:

tests.package.test_clang.TestClangCompilerRT.test_run lasts 4h25 on the
Buildroot runner. If we this duration for 2xlarge runners, the CI
minute consumed would be:

  (15900 / 60) * 0.5 = 133

With 6 jobs using a 2xlarge runners we used ~795 CI minutes.

tests.package.test_kmscube.TestKmsCube.test_run list 2h04 on the
Buildroot runner. If we	this duration for xlarge runners, the CI
minute consumed would be:

  (7440 / 60) * 0.5 = 62

With 4 jobs using a xlarge runners we used ~248 CI minutes.

tests.package.test_weston.TestWeston.test_run last 1h15 on th
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:

  (4500 / 60) * 0.5 = 37.5

With 28 jobs using a large runners we used ~1050 CI minutes.

tests.package.test_gstreamer1.TestGstreamer1.test_run last 46min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:

  (2760 / 60) * 0.5 = 23

With 31 jobs using a medium runners we used ~713 CI minutes.

tests.package.test_python.TestPython3Py.test_run last 13min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:

  (780 / 60) * 0.5 = 6.5

With 691 jobs using a medium runners we used ~4491 CI minutes.

In total, one pipeline for the runtime tests cost ~7297 CI minutes.
After a first try [6], we are actually using 8000 CI minutes per
pipeline.

We run such pipeline once a week (on Monday), one for each Buildroot
releases every 3 month, one for each stable and LTS release per month,
and one for each release candidate (3).

Worst case (release month):
(4 weeks + 1 release + 1 stable + 1 LTS + 3 release candidate) * 8000 CI
minutes: 80000 CI minutes / 50000.

So we would spend the minutes very quickly in the worst case scenario.
We have to keep one pipeline under 5000 CI minutes.

[1] https://docs.gitlab.com/ci/runners/hosted_runners/linux/#machine-types-available-for-linux---x86-64
[2] https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors
    https://docs.gitlab.com/ci/pipelines/compute_minutes/#compute-usage-calculation
    https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors-of-hosted-runners-for-gitlabcom
[3] https://gitlab.com/buildroot.org/gitlab-oss
[4] https://docs.gitlab.com/ci/runners/hosted_runners/#gitlabcom-hosted-runner-workflow
[5] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2416603721
[6] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2562421098

Signed-off-by: Romain Naour <romain.naour@smile.fr>
[Arnout:
 - simplify parsing of test_file and test_name;
 - match the entire test_name instead of substring;
 - assume "small" by default;
 - remove the "small" tags;
 - use "gitlab-runner" instead of "Gitlab-runner".
]
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>

Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
 2026-05-30 22:26:13 +02:00
Romain Naour
3713c3e82c Revert "support/testing: generate runtime test jobs with runner tags" 
The CI minute cost estimate is wrong for two reasons:
  1) Job duration was in minutes instead of seconds
  2) The cost factor is really 0,5 [1] instead of
     based on runner type * 0,5.

Due to this two mistake, the real CI minute cost is 7982.1 [2] instead
of 350.

Since we want to backport this patch, having a correct CI minute cost
estimate in the commit log would be better. Lets revert last changes
about runner tag and try again.

[1] https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors-of-hosted-runners-for-gitlabcom
[2] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2562421098

This reverts commit 38039415f8.
This reverts commit 825abb2682

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
 2026-05-30 22:26:12 +02:00
Romain Naour
38039415f8 support/testing: generate runtime test jobs with runner tags 
We have our own GitLab-CI runner, but with only one we can't run many
jobs in parallel so it takes a very long time before all the tests have
completed. In addition, if that runner goes down, we have nothing at
all.

GitLab offers the following machine types for hosted runners on Linux
x86-64 [1]. The default is the "small" runner. Using a larger runner
increases the "Cost factor" [2]:

Runner Tag                                             vCPUs   Memory  Storage  Cost factor
saas-linux-small-amd64 (default)                           2     8 GB    30 GB     1
saas-linux-medium-amd64                                    4    16 GB    50 GB     2
saas-linux-large-amd64 (Premium and Ultimate only)         8    32 GB   100 GB     3
saas-linux-xlarge-amd64 (Premium and Ultimate only)       16    64 GB   200 GB     6
saas-linux-2xlarge-amd64 (Premium and Ultimate only)      32   128 GB   200 GB    12

Compute minutes consumed by a job is calculated by:

  Job duration / 60 * Cost factor

(Job duration: The time, in seconds, that a job took to run, not
including time spent in the created or pending statuses.)

Thanks to the GitLab OSS program [3], Buildroot benefits from a free
Ultimate subscription and can use GitLab shared runners tagged with
saas-linux-{large, xlarge, 2xlarge}-amd64. In addition, we receive
50,000 free runner minutes per month.

For opensource projects, the Cost factor is reduced by 0,5 (1 minute per
2 minutes of job time).

In order to use one of those tags in Buildroot GitLab-CI jobs, we have
to classify all tests by resource requirement, to make sure the job
doesn't fail because it times out or has insufficient memory or disk
space. We also don't want to use the largest runner for everything
because that would spend the minutes very quickly while the resources
aren't used efficiently (mostly serialized).

First we introduce some new templates used to add the corresponding
runner tag to a runtime test job (reusing the GitLab terminology).

    .runner-{small,medium,large,xlarge,2xlarge}

Most of our tests are fast (checkpackage, test_external_bootlin...), so
saas-linux-small-amd64 runner tag is enough. Default to this tag if
nothing else is specified.

Add a comment next to the test class to provide the runner tag.
This runner tag is retrieved when generating the
generated-gitlab-ci.yml file used to create the child pipeline where
the runtime test jobs are executed.

We use the list of runtime tests returned by node2:

  "tests.boot.test_edk2.TestEdk2.test_run"

We convert each element of this list to get the path to the test source
file and the name of the test:

  "support/testing/tests/boot/test_edk2.py"

  TestEdk2

With that, we can grep into the test source file to retrieve the runner
tag placed one line above the test class:

  # GitLab-runner: large
  class TestEdk2(infra.basetest.BRTest):

Once the runner tag is retrieved, it's used to use the corresponding
runner template to the runtime test job:

  tests.boot.test_edk2.TestEdk2.test_run: { extends: [ .runtime_test_base, .runner-large ]}

GitLab runners hosted by the Buildroot project should be able to run any
jobs, so a specific runner tag "buildroot-runner" is used to allow
running a job on such runners. The "buildroot-runner" tag is used in
addition to GitLab ones to run a job on the Buildroot runner or on a
shared GitLab-CI runner.

If a test can't be executed by any shared GitLab-CI runners, we have
to use a runner owned by the Buildroot project. In this case we have
to use a specific template ".runner-buildroot-runner-only". There is no
such runtime test at the moment.

The proposed classification is based on a previous pipeline analysis
[5]:

  - Tests lasting more than 3 hours will use 2xlarge runners.
  - Tests lasting more than 2 hours will use xlarge runners.
  - Tests lasting more than 1 hours will use large runners.
  - Tests building a kernel or a toolchain will use medium runners.
  - All other tests will use small runners when possible.

CI minute cost estimate:

tests.package.test_clang.TestClangCompilerRT.test_run lasts 4h25 on the
Buildroot runner. If we this duration for 2xlarge runners, the CI
minute consumed would be:

  (265 / 60) * 12 * 0.5 = 26,5

With 6 jobs using a 2xlarge runners we used ~160 CI minutes.

tests.package.test_kmscube.TestKmsCube.test_run list 2h04 on the
Buildroot runner. If we	this duration for xlarge runners, the CI
minute consumed would be:

  (124 / 60) * 6 * 0.5 = 6,2

With 4 jobs using a xlarge runners we used ~37 CI minutes.

tests.package.test_weston.TestWeston.test_run last 1h15 on th
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:

  (75 / 60) * 3 * 0.5 = 1.87

With 28 jobs using a large runners we used ~53 CI minutes.

tests.package.test_gstreamer1.TestGstreamer1.test_run last 46min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:

  (46 / 60) * 2 * 0.5 = 0.76

With 31 jobs using a medium runners we used ~24 CI minutes.

tests.package.test_python.TestPython3Py.test_run last 13min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:

  (13 / 60) * 1 * 0.5 = 0.1

With 691 jobs using a medium runners we used ~69 CI minutes.

In total, one pipeline for the runtime tests cost ~350 CI minutes.

We run such pipeline once a week (on Monday), one for each Buildroot
releases every 3 month, one for each stable and LTS release per month,
and one for each release candidate (3).

Worst case (realse month):
(4 weeks + 1 release + 1 stable + 1 LTS + 3 release candidate) * 350 CI
minutes: 3500 CI minutes / 50000.

So we should be able to run our pipeline without exhausting our CI
minutes credit.

[1] https://docs.gitlab.com/ci/runners/hosted_runners/linux/#machine-types-available-for-linux---x86-64
[2] https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors
    https://docs.gitlab.com/ci/pipelines/compute_minutes/#compute-usage-calculation
    https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors-of-hosted-runners-for-gitlabcom
[3] https://gitlab.com/buildroot.org/gitlab-oss
[4] https://docs.gitlab.com/ci/runners/hosted_runners/#gitlabcom-hosted-runner-workflow
[5] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2416603721

Signed-off-by: Romain Naour <romain.naour@smile.fr>
[Arnout:
 - simplify parsing of test_file and test_name;
 - match the entire test_name instead of substring;
 - assume "small" by default;
 - remove the "small" tags;
 - use "gitlab-runner" instead of "Gitlab-runner".
]
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
 2026-05-29 18:13:29 +02:00
Adam Duskett
67c6d1b12c package/mender: bump version to 3.5.3 
Updating directly to 4.x is too difficult and complex, as mender 4.x is
a complete rewrite in C++, with several new dependencies and changes.

As such, update to the last version that was written in Go, which is
3.5.3. The following changes are necessary:

The artifact_info file is no longer supported. Instead, mender now expects a
bootstrap.mender artifact created by mender-artifact /var/lib/mender.
See the following for more information:
https://northerntech.atlassian.net/browse/MEN-2585
https://northerntech.atlassian.net/browse/MEN-2583
https://docs.mender.io/release-information/release-notes-changelog/mender-client#mender-3-5-0-1
https://github.com/mendersoftware/mender/blob/3.5.3/Documentation/automatic-bootstrap-artifact.md

  - Remove all instances of creating or copying an artifact_info file
    from board examples and package/mender.

  - Add a generate_mender_bootstrap_artifact method to
    board/mender/x86_64/post-image-efi.sh which creates a proper
    bootstrap.mender file and places it in the data partition.

  - Add a post-build.sh script to support/testing/tests/package/test_mender
    with the same generate_mender_bootstrap_artifact method with one change:
    As the rootfs.ext4 file is not yet created; we omit the optional
    --provides "rootfs-image.checksum:${img_checksum}" argument when
    generating the bootstrap.mender file.

Mender expects the device_type file to exist in /var/lib/mender/device_type.
  This is further supported by the following line in tests/Dockerfile.daemon:
  `echo device_type=docker-client > /var/lib/mender/device_type`

Add a migration section in docs/manual/migrating.adoc

Update the package/mender/readme.txt to include a quick mention of the
bootstrap.mender artifact change.

License changes:

New:
vendor/github.com/klauspost/compress/internal/snapref/LICENSE (BSD-3-Clause)
vendor/github.com/klauspost/compress/zstd/internal/xxhash/LICENSE.txt (MIT)

Removed:
vendor/github.com/klauspost/compress/internal/snapref/LICENSE
vendor/github.com/klauspost/compress/zstd/internal/xxhash/LICENSE.txt

Modified:
LICENSE: Update year from 2022 to 2024
LIC_FILES_CHKSUM.sha256: Sum of all Licenses changed
vendor/github.com/mendersoftware/mender-artifact/LICENSE
  - Update year from 2022 to 2023

Signed-off-by: Adam Duskett <adam.duskett@amarulasolutions.com>
[Arnout:
  - fix LIC_FILES_CHKSUM.sha256 hash
  - Better URL for bootstrap artifact doc
  - Migrate to 2025.02 instead of 2024.11
  - Improve migrating text (editorial changes)
]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
 2025-02-04 15:00:19 +01:00
Adam Duskett
c4772ecef6 support/testing/tests/package/test_mender.py: Add a systemd test 
- Move the check for /var/run/mender.pid to before self.run_mender_test()
    for the two tests using sysvinit.

  - Instead of checking for /var/run/mender.pid during the systemd test,
    check to ensure "systemctl is-active mender-client" returns "active"

Signed-off-by: Adam Duskett <adam.duskett@amarulasolutions.com>
[Arnout: fix flake8 issue]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
 2025-02-04 13:57:11 +01:00
Adam Duskett
5389f9b1a2 support/testing/tests/package/test_mender.py: Add RO fs test 
Add a test for read-only file systems.

Signed-off-by: Adam Duskett <adam.duskett@amarulasolutions.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
 2025-02-04 13:19:56 +01:00
Mikael Bourhis-Cloarec
bc91d35da7 support/testing: add package/test_mender.py 
Create a test to check Mender client at runtime.

The aim of this test is:
  - to check the correct execution of simple Mender commands,
    in a minimal environment;
  - to validate there is no missing dependencies for runtime.

This test is not a board integration test for Mender,
including well-configured bootloader, partitioning, ...

Check:
  - the daemon is started;
  - the current 'artifact name' (name of the image or update) of the active
    partition is read, without error.

For that, we need to fake (see the 'overlay' directory):
  - some bootloader environment variables;
  - the name of an update.

Signed-off-by: Mikael Bourhis-Cloarec <mikael.bourhis@smile.fr>
[Romain: remove single hyphen command (Mender 3.0.0)]
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
 2022-01-07 19:05:46 +01:00