Some packages do not have a http/https download URL for a source tarball,
but are acquired over a version control system like git. If so, add
externalReferences of type "vcs" for such URLs.
As most git repositories use a https:// transport that may not indicated the
repository type, add a "comment" due to the lack of a better mechanism in
CycloneDX.
While the hashes are calculated over a tarball created locally, it still may
be useful, so add them for "vcs" externalReferences as well.
Signed-off-by: Martin Willi <martin@strongswan.org>
Acked-By: Thomas Perale <thomas.perale@mind.be>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit e8c54ffb3d)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Commit dc4af8bfa9 ("utils/generate-cyclonedx: use direct dependencies")
removes indirect dependencies from any listed component, as required by
CycloneDX. The root component, however, still includes indirect dependencies,
as it just takes the components from the show-info output.
Fix this by collecting all component dependencies, and then filter the root
component dependencies to include direct dependencies only.
Signed-off-by: Martin Willi <martin@strongswan.org>
Acked-By: Thomas Perale <thomas.perale@mind.be>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit cc41cc3fcd)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Since commit d10d22221f (utils/get-developers: read patch from stdin
when it's not a tty), get-developers accepts to read a patch fromn its
stdin when it is not a tty.
Add a test for this.
Reported-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
get-developers will check its stdin to decide whether it is a tty or
not, and behave differently whether it is or not. So, when we run the
tests, we need an actual tty.
However, when running in a CI pipeline, like on Gitlab-CI, there is no
tty available on stdin.
Fake one. We don't need anything too fancy, so just a slave pty will
suffice.
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/8830671800
Fixes: d10d22221f (utils/get-developers: read patch from stdin when
it's not a tty)
Reported-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
parser.error() reports a nice error message, that also displays a short
reminder of the available options.
Adapt the test-suite accordingly: previously, the error string was an
exact string in the stdout list, while it now is a substring in one of
the strings in stderr. The exit code changes, too.
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Julien Olivain <ju.o@free.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
The past participle for "to fix" is "fix". The "did you forget" got
eluded into "forget", so again a past participle.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Teach check-package to detect python files by type and check them using
flake8.
Do not use subprocess to call 'python3 -m flake8' in order to avoid too
many spawned shells, which in its turn would slow down the check for
multiple files. (make check-package takes twice the time using a shell
for each flake8 call, when compared of importing the main application)
Expand the runtime test and the unit tests for check-package.
Remove check-flake8 from the makefile and also from the GitLab CI
because the exact same checks become part of check-package.
Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: add a comment to x-python to explain its purpose]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Currently only SysV init scripts are checked using shellcheck and a few
other rules (e.g. variable naming, file naming).
Extend the check using shellcheck to all shell scripts in the tree.
This is actually limited to the list of directories that check-package
knows that can check, but that list can be expanded later.
In order to apply the check to all shell scripts, use python3-magic to
determine the file type. Unfortunately, there are two different python
modules called "magic". Support both by detecting which one is installed
and defining get_filetype accordingly.
Keep testing first for name pattern, and only in the case there is no
match, check the file type. This ensures, for instance, that SysV
init scripts follow specific rules.
Apply these checks for shell scripts:
- shellcheck;
- trailing space;
- consecutive empty lines;
- empty last line on file;
- newline at end of file.
Update the list of ignored warnings.
Do not add unit tests since no function was added, they were just
reused.
But expand the runtime test for check-package using as fixture a file
that generates a shellcheck warning.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: support both variants of the "magic" module]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Start counting the line numbers in 1 instead of 0, in case an error
must be printed.
Both the error about a developer entry with no file entry and the error
about a file entry with no developer entry actually belong to the
non-empty line previous the one being analysed, so in these cases print
the line number from the line before.
Also count empty and comment lines, so a developer fixing the file can
jump to the correct line (or the nearest one).
At same time standardize the messages, printing the line number
also in the case of a warning for a file that is not in the tree
anymore.
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently 4 types of parsing errors/warnings can be found:
- entry for a file that is not in the tree anymore (warning)
- developer entry with no file entry (error)
- file entry with no developer (error)
- entry that is not a developer, a file or a comment (hard error)
Currently only the last one ends the script with -v with error code.
Make all 3 error types into hard errors and bail out at the first error
found, because the rest of the state machine is not designed to handle
malformed input.
Suggested-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Extend test_check_package to also check the ignore list functionality.
Check:
- the entries in the ignore list use relative path;
- an entry in the ignore list actually ignores the warning;
- an outdated entry in the ignore list generates a warning by its own,
preventing the ignoring list to grow indefinitely.
For this to work, add 3 test fixtures, listing entries for an
pre-existing file in the br2-external used in the test.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Change the test into a characterization test for all warnings and errors
get-developers can return when parsing the DEVELOPERS files.
It will be helpful when changing the behavior of get-developers to bail
out on all syntax checking warnings.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add a runtime test in order to detect undesired changes in behavior of
the get-developers script.
The test uses a .patch file generated against the buildroot tree as a
fixture to check how get-developers operates when called to check it.
The test also overrides the DEVELOPERS file in order to be fully
reproducible and a -d option is added to get-developers in order to
allow this. Since get-developers only looks to already committed
files to compare against patch files, the fixture uses a package that
is very unlikely to be removed from buildroot tree: binutils.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Thomas: extracted from a larger patch from Ricardo, submitted at
https://patchwork.ozlabs.org/project/buildroot/patch/20220528014832.289907-1-ricardo.martincoski@gmail.com/]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Python 2.7 will not be maintained past 2020.
Many scripts on the tree are used during the build and should keep
Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in
modern distros only, so it can safely switch to support Python 3 only.
An advantage of this approach is to have less scenarios to test in.
Otherwise every change to the test infra or runtime tests would need to
be tested against both versions of the interpreter, increasing the
effort of the developers, to ensure the compatibility to Python 2 was
not broken.
In order to accomplish the change to Python 3:
- change the shebang for run-tests;
- use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
- when writing the downloaded binary files, explicitly open the output
file as binary;
- when subprocess is used to retrieve the text output from commands,
explicitly ask for text output. For this, use 'universal_newlines'
because 'text' was added only on Python 3.7;
- when pexpect is used to retrieve the text output from qemu or git,
explicitly ask for text output using 'encoding';
- the code using csv currently follows the example in the documentation
for the Python 2 module, change it to follow the example in the
documentation for the Python 3 module;
- fix the relative import for test_git.py to be Python 3 compliant.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: Romain Naour <romain.naour@smile.fr>
Tested-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Check external.mk is ignored only when in the root path of a
br2-external.
Add a file called external.mk as a fixture to be used by the test case.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: wrap at 80 columns]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Check the basic usage for check-package.
It can be called using either absolute path, relative path or from PATH.
Files to be checked can be passed with either absolute path or relative
path (also including files in the current directory).
Also check it ignores some special files when checking intree files,
i.e. package/pkg-generic.mk, while still generating warnings for out-of-tree
files when called with -b.
In order to allow the later, add an empty line to the Config.in in the
br2-external being tested so the script does generate a warning.
Catches bug #11271.
More tests can be added later, for example compatibility to Python 3.
Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: wrap at 80 columns where appropriate; merge into a single
class.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
