The Debian 12 Bookworm snapshot we use, 20250203, uses the new deb822
format [0] for source list files; it carries a source list file that
points to the current repository, not to the snapshot.
Even though we do inject an old-style source list that points to the
snapshot repository, the packages in the current repository are more
recent than the one in the snapshot, so when we install our packages,
they get retrieved mostly from the current repository rather than from
the snapshot. The image is not reproducible.
Switch to using the new deb822-style source list file.
Note: we do not need to carry the "Latest just before" trick: the
snapshot repository will use the most recent actual snapshot before the
requested dated, so we can just use the date of the image we use (stuck
at midnight because we don't have better).
Since the snapshot repository can be really slow, while at the same time
the remote http server not dropping connections, it can take a very long
while to build the image; add a timeout so that stale connections are
detected and re-attemped early-ish (the timeout applies to both the
connect and the actual download, so it should be large enough to
accomodate slowish network connections).
Fixes: c95d5b8e1e (support/docker: move to current bookworm (Debian
12) snapshot)
[0] https://manpages.debian.org/trixie/apt/sources.list.5.en.html#DEB822-STYLE_FORMAT
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
With the TARGETPLATFORM fix we need to rebuild the container so move to the
latest snapshot to get updated (security) fixes.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit 12e9fc1da5 ("support/docker: add tar 1.35") added logic to download
and build tar for the host, but mistyped a configure option:
configure: WARNING: unrecognized options: --disable-year2028
Fix that.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
buildx correctly complains about our ENV lines:
3 warnings found (use docker --debug to expand):
- LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 21)
- LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 94)
- LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 95)
The ENV syntax is defined as ENV FOO=BAR, not ENV FOO BAR, so adjust the
Dockerfile to match to get rid of this warning.
https://docs.docker.com/reference/dockerfile/#env
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
With the new 'buildx' command using 'BuildKit' since Docker 23.0 [1],
it become easy to build the same image for several architectures [2].
The only requirement is to use a Dockerfile without any architecture
specific definition.
Since our current Dockefile contains already some i386 specific
packages (g++-multilib,libc6:i386) to provide x86 32bits support,
we have to install them conditionally.
Update the build process described in the Dockerfile accordingly.
For now, Aarch64 hosts can't be used by the Buildroot testsuite (yet)
since the Bootlin external toolchain (used by default) currently only
support x86_64 hosts.
[1] https://docs.docker.com/engine/release-notes/23.0/
[2] https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/
[3] https://www.docker.com/blog/multi-arch-build-what-about-gitlab-ci/
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since commit b11956fb66 ("support/dependencies: require tar >= 1.35"),
Buildroot will build host-tar if the host does not have >= 1.35 available,
so add it to the container to save build time.
Debian 12 only provides tar 1.34. Tar 1.35 is available in testing - But
that then depends on a newer glibc, so instead build it from source using
the --disable-year2038 flag like we do for host-tar and install it into
/usr/local/bin so it shadows the Debian one.
This step runs as root (and needs to for make install), so add
FORCE_UNSAFE_CONFIGURE=1 as otherwise the configure script errors out.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit afece24a72 ("support/download: introduce curl backend for FTP
transfers") added curl download support, so add it to the container as well.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When check-package fail due to flake8 coding style checks, it suggest
to use flake8 command line:
run 'flake8' and fix the warnings
But flake8 is actually missing from the container used by
utils/docker-run, so add it to the container as well.
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The docker image currently contains qemu-system-arm and
qemu-system-x86. Each package contains the 32bit and 64bit variants.
This has been sufficient for the time being.
The RISC-V ecosystem is growing rapidly. It is starting to become
mainstream. To increase the diversity in Buildroot runtime tests,
this commit adds the qemu-system-misc package in the Docker image,
in order to have the commands qemu-system-riscv{32,64}. This package
also contains other architectures (for example: microblaze, nios2,
s390x, xtensa, ...).
For Debian package details, see:
https://packages.debian.org/bullseye/qemu-system-misc
Signed-off-by: Julien Olivain <ju.o@free.fr>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
snapshot.debian.org is notoriously slow, and quite often leads to
timeouts when downloading packages. To give us a better chance of
succeeding, let's retry 3 times before failing.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
It will be needed by check-package to run checks according to the file
type (the same determined by the command 'file').
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Allow developers to run check-package for init scripts, that call
shellcheck, without having to install the tool.
Since the docker have a fixed version of the tool, there will be no
difference between runs in different machines.
One can call:
$ utils/docker-run utils/check-package package/package/S*
$ utils/docker-run shellcheck package/package/S*
This change also allows to eventually run check-package for init scripts
in the GitLab CI.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Install the openssh-server package into the test container. This
package, as well as its dependency openssh-client, is required to test
SCP and SFTP download methods on the localhost, as if these tools were
already installed on the user's machine.
Signed-off-by: Thomas Preston <thomas.preston@codethink.co.uk>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Now that our pipelines are using the Docker image from the Gitlab
registry, there is no longer any reason to push the image to the
Docker hub.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
On a properly setup machine, it is totally useless to use sudo to run
docker; it is very bad practice. Instead, users really should add
themselves to the docker group.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since commit 4a40d36f13
("support/testing: switch to Python 3 only") our runtime testing
infrastructure is Python 3.x only.
Therefore, it is no longer needed to have python-nose2 and
python-pexpect in the Docker container used to run our Gitlab CI jobs.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
support/scripts/pkg-stats now uses some Python 3.x only constructs
("async" and related keywords), so we must use the Python 3.x flake8.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The test infra will soon be converted to Python 3 only.
So add the interpreter and also the Python 3 variant of modules nose2
and pexpect to the docker image used to run runtime tests.
Keep the Python 2 variant of those modules to allow a gradual
transition.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, we install flake8 and its dependencies via pip. We
tried to be reproducible by pinning the version of those python
packages, but we did forget quite a few of them, and thus some
dependencies for flake8 are installed as uncontrolled versions.
Furthermore, before we install flake8 and its dependencies, we
forcibly update pip, setuptools, and wheels packages to their
latest versions. This explicitly breaks reproducibility.
While we could enforce a specific version of all those packages
and still grab them from PyPI, we can simply grab them from the
distribution-provided packages instead.
Since we're using a pinned version of stretch, this already
guarantees we'll reproducibly get the same versions over and
over again. Besides, we just need to list flake8 as a package to
install to automatically get all its dependencies (again, in a
reproducible way).
This has the slight unfortunate drawback of downgrading flake8
to version 3.2.1, from version 3.5.0, as well as downgrading a
few of flake8's dependencies, as noticed by Ricardo:
http://lists.busybox.net/pipermail/buildroot/2018-May/222376.html
However, as Ricardo said, there isn't "any serious limitation of
this old version, the release notes for a version in the between
mentions 'Dramatically improve the performance' but we have a
limited number of scripts and running on Gitlab for all of them
still takes less than 5 minutes".
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Acked-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In commit 7517aef4d (support/docker: limit the number of layers),
we reduced the number of layers by coalescing multiple RUN commands
into less commands.
In doing so, we especially coalesced "apt-get update" with "apt-get
install".
However, the distribution we used is a pinned version of stretch, so
we know that running apt-get update will always yield the same apt
database.
If we split the two apt-get commands, then we can re-use any local
intermediate image when we need to update the list of packages to
install; this helps quite a bit when testing the docker files over
and over again, with just slight variants in the packages list.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some packages build C++ 32bits host-tools and need the g++-multilib to
be installed on the build machine. As example, qt5webengine builds a C++
host-tool when target is 32bits.
Add the check for g++-multilib to the dependencies script; and update
the Dockerfile to install g++-multilib package.
Signed-off-by: Gaël PORTAY <gael.portay@savoirfairelinux.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, we refer to the latest version of the image, which means we
can't guarantee any reproducibility. Also, it measn we can't have a
separate images for the maintenance branches (especially the LTS) and
master.
Update the comment in the Dockerfile to create and push tagged images.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since we're now using a specific base image tag, we need to also use a
specific, stable repository to get additional packages from for this
image.
As such, use the Debian snapshot that matches the base image.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, we are using debian:stable, which is subject to change with
time, as new stable versions of Debian are released/updated.
Use the latest tagged stable release, stretch-20171210 as of today, as
the base distribution to use.
This will ease reproducible builds in the future.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This image is not built very often, and when it is, it is important to
see what's going on, so don't be silent when installing packages from
the distro, and since that can take a bit of time it thus serves as
progress report...
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, our jobs on the gitlab-ci infra are running as root, which is
problematic for two reasons:
- this is not the usual way Buildroot is built;
- it may miss issues where running as non-root is problematic.
So, complement our Dockerfile with directives to add a new user and run
everything as that user, as demonstrated by this build job:
https://gitlab.com/ymorin/buildroot-ci/-/jobs/46929562
Additional, enforce an UTF-8 locale while running.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
For Gitlab-CI, we want to avoid re-generating the minimal install to
be able to run tests all the time. So let's create a docker image that
we can post on Docker Hub and then pull.
For the time being, this is just what we need for running our CI. Later
we can produce something that is also useful for users.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
