This commit adds the -D/--debug command line options to the
support/testing/run-tests script.
Using one of those options will pass "--log-level debug" to the
nose2 invocations. This can be useful to debug nose2 internal
behaviors. See:
https://github.com/nose-devs/nose2/blob/0.15.1/nose2/main.py#L209
For example, calling:
support/testing/run-tests --list --debug
will output:
List of tests
DEBUG:nose2.main:logging initialized debug
...
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit 7f0af11cee (support/testing/tests/package/test_flutter.py: new
runtime test) added a simple test that just checked that the systemd
unit launching the application, was active.
It is perfectly possible, from a systemd perspective, that the unit is
active, while the application actually crashes.
Instead, what we need to check, is that the application does actually
render "something"; we don't really care what, as long as we know it is
actually rendering, thus the graphical stack is working and the
aplication stack is running.
Extend the flutter runtime test to also check that the framebuffer is
modified by the application when it is running, similarly to what we do
in the weston test.
We drop the activation of the unit and start it manually, as we want to
check the state of the CRC before, while, and after the application
runs.
We also need to disable the blinking cursor on the console, or we would
not be able to detect whether a change in CRC is due to the application
starting rendering, or to the cursor blinking. We tell the kernel to
disable the cursor with the appropriate kernel command line parameter.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Adam Duskett <aduskett@gmail.com>
Reviewed-by: Adam Duskett <aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In 4edb0e3456 (support/testing/tests/package/test_weston.py: new
runtime test), the weston test was introduced, and thus was the first
that needed to test that rendering was happening.
Now we also have a test for a flutter application, and we'll want to
have it test the rendering too.
Move the corresponding code to a helper that can be reused by other
tests, rather than duplicate (or reinvent) it.
Switch weston to using that new helper.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit 9696d27756 "package/gpsd: condition python stuff to the proper
kconfig option" changed the condition in which the gpsd python scripts
are installed. After that change, the "gpsfake" command (which is a
python script) is no longer found and the runtime test is failing.
This commit fixes the issue by reflecting the change in the runtime
test Buildroot configuration.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This is a helper class providing a template for testing audio codec
programs such as lame mp3 encoder, flac tools, ogg vorbis-tools, ...
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adds a test case for the recently added swipl package. It
should have been part of commit
69710addd1 ("package/swipl: new
package") but was forgotten.
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Runtime tests running on test runners are subject to a high
variability in term of performance and timing. Most or the runtime
test commands are executed with a timeout, in pexpect.
Slow or very loaded test runners can use the timeout_multiplier to
globally increase those timeouts.
Some runtime test commands sometimes needs to poll or query a state,
rather than having purely sequential actions. It is sometimes hard to
know, from the test writer point of view, the maximum timeout to set, or
if a retry logic is needed.
In order to help debugging runtime tests failing due very slow
execution, this commit adds extra information on the host test runner
about its load in the run log. Relevant information are: number of
cpus, the load average at the moment the emulator is started and the
current timeout_multiplier.
Note: this change was discussed in:
https://lists.buildroot.org/pipermail/buildroot/2024-July/759119.html
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The previous commit removes s6-test from the runtime tests. This
commit adds the same test for eltest, which replaced s6-test.
Signed-off-by: J. Neuschäfer <j.neuschaefer@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
s6-test was removed in s6-portable-utils 2.3.0.0, and replaced with
eltest (from execline). This change broke the existing runtime tests
with commit 42fad03182 ("packages: bump skalibs-related packages").
Instead of s6-test simply use the shell built-in "test", which provides
sufficient functionality.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/7338388010
Signed-off-by: J. Neuschäfer <j.neuschaefer@gmx.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The TestInitSystemNone bypasses the normal init and instead launches
directly into a shell. So it needs to bypass the login code and go
directly to a shell.
Signed-off-by: Brandon Maier <brandon.maier@collins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This test started failing at commit
0cad947b96
"support/testing/infra/emulator.py: fix qemu prompt detection" with the
error message
AttributeError: 'NoneType' object has no attribute 'run_command'
This is because we changed emulator.run() so that emulator.login() must
be called first. But this test skips the login and goes directly to a
shell. Use the new emulator.connect_shell() function which prepares the
shell without logging in.
Signed-off-by: Brandon Maier <brandon.maier@collins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Note: this test was not working in Buildroot test infrastructure
before commit [1] was merged, because dmidecode has the string "# "
in its output.
[1] 0cad947b96
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Since we copied a piece from code from pexpect, we need to also follow
the license of pexpect. Effectively, that makes the file licensed under
both our default GPL-2.0 and pexpect's ISC license. This is already
hinted at in the comment inside _repl_sh_child, but let's make it
explicit by adding a license header to the file.
Ideally, of course, each file in Buildroot would have a license file.
This was done with the reuse tool:
reuse annotate -l GPL-2.0 -l ISC support/testing/infra/emulator.py
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
The qemu.run() method can break when a command happens to output the
string "# " to stdout. This is because qemu.run() detects when a command
has completed by searching for the shell prompt, which by default is
"# ". It then captures everything before the "# " as the commands
output, causing the rest of output to be lost.
Instead use the pexpect libraries REPLWrapper to handle running
commands. It has hooks to set a custom prompt and avoid some other
pitfalls of wrapping a shell.
We unfortunately can't reuse replwrap._repl_sh directly, because it
tries to spawn a command while we already have a qemu started. So we
need to copy that code into _repl_sh_child. While we're at it, also
define our own prompt strings.
Signed-off-by: Brandon Maier <brandon.maier@collins.com>
[Arnout:
- Make all arguments to _repl_sh_spawn non-optional.
- Move non_printable_insert to a local variable instead of an argument,
we don't need to override it.
- Copy the comment from _repl_sh that explains why non_printable_insert
is needed.
- Add a comment about timeouts.
- Rename spawn to child (we don't actually spawn anything so this felt
more natural, even though the class
- Use single quotes instead of triple quotes, and explicitly escape the
nested quotes.
]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
The python-pymupdf and mupdf packages do not follow the exact same
version numbers anymore. There is no specific mention of version
compatibilities in the upstream package, so update our comment to just
say that both should be "compatible" (to try to avoid one being
updated without the other).
The hardcoded paths we used to remove from setup.py are no longer
present, so the post-patch hook is removed. The new setup.py instead
uses new environment variables which we now provide.
This new python-pymupdf version fails at runtime when mupdf builds
static libraries.
Note also that python-pymupdf is gradually switching to a new
implementation that requires mupdf to be built with python
bindings. For now, both implementations are still available but we
only compile the old one. The runtime test is adapted accordingly as
the legacy implementation has to be imported with "fitz_old".
While at it, the dependencies are also split to one per line to make
them easier to diff in the future.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[Raphaël:
- fix cross-compilation
- remove unneeded dependencies
- update to 1.23.22
- update the commit message
]
Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
To give us a chance to catch runtime issues (such as missing
dependencies) more easily, add a test that writes a sample PDF file,
read it back and verify the text that was read.
Like similar packages that lead to a big
rootfs (e.g. python-botocore), this test requires a separate ext2
rootfs to avoid filling the default amount of RAM available
entirely (which would cause missing files from the root filesystem and
in turn, test failures).
Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add optional c extension for python-ruamel-yaml which can speed up yaml
loading/parsing.
Extend the ruamel-yaml runtime test to check if the c extension works
correctly.
Signed-off-by: Marcus Hoffmann <buildroot@bubu1.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Version 2.1.0 is a major bump, with an API change that should be
backward compatible with version up to 1.6.1.
The runtime test failed in timeout because the crng init did not have
time to complete in the default 5-second timeout; tests showed that the
trip-off would be between 6s and 7s here, so increase it enough to leave
some margin on less-capable machines.
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Davide Viti <zinosat@gmail.com>
Cc: James Hilliard <james.hilliard1@gmail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Marcus Hoffmann <buildroot@bubu1.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When running a test that uses host-python-setuptools using the Buildroot
Docker image, for example running the following command,
> ./utils/docker-run ./support/testing/run-tests -o output -s -k tests.package.test_python_pytest.TestPythonPy3Pytest
The build fails with the following error,
> File "/home/blmaier/buildroot/output/TestPythonPy3Pytest/build/host-python-setuptools-69.2.0/setuptools/_distutils/dist.py", line 354, in _gen_paths
> yield pathlib.Path('~').expanduser() / filename
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/blmaier/buildroot/output/TestPythonPy3Pytest/host/lib/python3.11/pathlib.py", line 1385, in expanduser
> raise RuntimeError("Could not determine home directory.")
> RuntimeError: Could not determine home directory.
>
> ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Python setuptools is looking for $HOME but failing to find it.
Signed-off-by: Brandon Maier <brandon.maier@collins.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
No functional change as we install a python symlink, but use python3 for
consistency with the other scripts.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Test the source provider of host-go to build a Go package.
The tests consist of building and installing a Go package in the root
file system of an ARM vexpress QEMU system.
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
[yann.morin.1998@free.fr: drop the go-bin test-case]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The python-magic-wormhole runtime test can randomly fail on slow
runners, see [1].
The issue is that the sending command is started first in background
_without_ redirecting its output to /dev/null. The receiving command
is started after, expecting the message to be printed on the first
standard output line. On slower systems, the sending command still
print messages while the test controller expect output from the
receiving command. The expected string finally appear, but not on the
first line. This makes the test fail.
This commit fixes the issue by redirecting all outputs (stdout, stderr)
of the sending command to /dev/null. To help even more, the sleep time
is moved from the emulator to the test controller. The sleep time is
also multiplied by the timeout_multiplier.
Fixes: [1]
[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/6888691508
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
