Specifying a list of br2-external trees is poorly documented, and the
only example uses a colon to separate the br2-external paths.
Adding the support for colon-separated list is the biggest mistake that
was made when introducing support for multiple br2-external [0]. Indeed,
both space and colon can be used to separate entries in the list, and it
is also possible to mix the two. However, internally, the list is stored
as a space-separated list, and all the code will split on spaces.
Besides, all other lists in Buildroot are a space-separated:
BR2_ROOTFS_DEVICE_TABLE
BR2_ROOTFS_STATIC_DEVICE_TABLE
BR2_TARGET_TZ_ZONELIST
BR2_ROOTFS_USERS_TABLES
BR2_ROOTFS_OVERLAY
BR2_ROOTFS_PRE_BUILD_SCRIPT
BR2_ROOTFS_POST_BUILD_SCRIPT
BR2_ROOTFS_POST_FAKEROOT_SCRIPT
BR2_ROOTFS_POST_IMAGE_SCRIPT
...
So, using colons is odd.
The fact that BR2_EXTERNAL is passed on the command line rather than
being a Kconfig item is not a reason enough to justify that it be
colon-separated.
Change the documentation to only mention using a space-separated list.
Of course, for backward compatibility, we keep the code as-is to accept
a colon-separated list, but we just do not advertise it.
Note that keeping the split on colons means that colons are not accepted
in pathnames of br2-external trees; in practice, this is not a new
restriction, or one that could lift as usign colons in Makefiles are
problematic anyway.
[0] in 20cd497387 core: add support for multiple br2-external trees
Reported-by: Fiona Klute (WIWA) <fiona.klute@gmx.de>
Reported-by: Brandon Maier <Brandon.Maier@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
(cherry picked from commit f9cdca48a5)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The trap was initially introduced in c5fa9308ea (core/br2-external:
properly report unexpected errors), in 2017, to catch all unexpected
errors, back when a single file was generated, and errors emitted to
stderr.
Since commit d027cd75d0 (core: generate all br2-external files in
one go), in 2019 the single output file 'ofile' is no longer created,
as multiple output files were then introduced, while messages for
*expected errors* were redirected to a Makefile variable assignment
emitted on stdout, at which point the script just exits (in error);
expected failures only occur in do_validate().
Unexpected errors can only occur on failure to create, or write to,
output files, either '.br2-external.mk' in do_validate() or do_mk(),
or any of the kconfig fragments in do_kconfig(). Cause for failure to
create those can only be a no-space-left-on-device condition, as they
are created in a directory that was just created by the script earlier
in main(), and thus has the necessary mode; failure to create that
directory is now caught explicitly.
A trap on ERR is not called when the shell exits explicitly with a call
to 'exit', thus, only failures to create or write to output file would
be caught. In that case, we are better off not trying to write to those
files anyway: failure to create the file would already be reported by
the shell on stderr, while disk-full would not allow to store the output
anyway...
In any case, the script exits in error, which is going to be caught by
the caller, which will terminate.
So, drop the trap altogether.
As a side effect, that squelches a shellcheck error.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
(cherry picked from commit 244e4283a9)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
This commit adds the -N/--needs-update option, disabled by default,
to list only packages with newer upstream versions. All other packages
will be excluded from the HTML or JSON output.
Signed-off-by: Kadambini Nema <kadambini.nema@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit ed9466e7f9)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
When running "make pkg-stats" on a host with Python 3.14 (e.g.
Fedora 43 for example), the execution fails with the error:
Checking URL status
Traceback (most recent call last):
File "/buildroot/support/scripts/pkg-stats", line 1387, in <module>
__main__()
~~~~~~~~^^
File "/buildroot/support/scripts/pkg-stats", line 1368, in __main__
loop = asyncio.get_event_loop()
File "/usr/lib64/python3.14/asyncio/events.py", line 715, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
% threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'MainThread'.
This is due to a breaking change introduced in Python 3.14
asyncio.get_event_loop(). See [1]. Before Python 3.14, this call was
creating and setting an event loop if there was none. This situation
is now a runtime error.
In order to fix this issue with newer Python version, while keeping
backward compatibility, this commit replaces the code:
loop = asyncio.get_event_loop()
by an explicit event loop creation:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
This commit was tested on a Fedora 43 host with Python-3.14.2, and
with the Buildroot Docker image plus the python3-aiohttp package
which is a Debian 12 with Python-3.11.2.
[1] https://docs.python.org/3.14/library/asyncio-eventloop.html#asyncio.get_event_loop
Signed-off-by: Julien Olivain <ju.o@free.fr>
Tested-by: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit e9f426aa52)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The NVD database has CVE entries that are not present but may be
referenced in other security trackers.
For instance the CVE-2024-12455 is documented in the Debian security
tracker [1]. However, the NVD page is empty [2] and this entry is not
present in the NVD database mirror.
The following command would make the script fail:
```
echo '{
"vulnerabilities": [
{
"id": "CVE-2024-12455"
}
]
}' | support/scripts/cve-check --enrich-only
```
No CVEs present in Buildroot ignored CVEs are affected. But when
enriching an SBOM with legitimate CVE not present on NVD, the script
will fail.
This patch change the behavior to just log to stderr unknown CVEs
instead of making the script fail.
[1] https://security-tracker.debian.org/tracker/CVE-2024-12455
[2] https://nvd.nist.gov/vuln/detail/CVE-2024-12455
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
[Peter: Tweak warning message]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit fa7fac0985)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
As requested by the gnu.org admins:
"
Since August 2024, we've been under DDoS attacks from common command line
tools.
To fix this, we would need to change the user-agent from "Python/3.11
aiohttp/3.8.4" to "buildroot.org pkg-stats" instead.
"
It indeed probably makes sense to use an unique user-agent string, so rework
the script to do that.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit 0981b4117d)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The gnu.org admins have been blocking the IP address of machines running
pkg-stats as the GET requests for the (many) packages with gnu.org URLs are
seen as abusive.
The resource body is not used, so use a HTTP HEAD request instead of a GET
to limit server load and bandwidth use.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit f9f3e6ccc6)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Enriches the input CycloneDX SBOM with vulnerability information and
analysis from the NVD database.
The NVD database is cloned using a mirror of it and the content is compared
locally. By default the path 'dl/buildroot-nvd' is used.
Example usage to analyse vulnerabilities of an input CycloneDX SBOM:
$ make show-info | utils/generate-cyclonedx | support/script/cve-check
The 'cve-check' can also be used to only enrich the vulnerabilities
present on the input SBOM with a set metadata (description, cvss,
references, ...) without applying an analysis.
With the following command the vulnerabilities ignored by Buildroot
present in the CycloneDX SBOM are enriched with description, cvss, etc
...
$ make show-info | utils/generate-cyclonedx | support/script/cve-check --enrich-only
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
[Peter: fix minor flake8 issues]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 6762c42e74)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
This patch move the 'download_nvd' call to the 'pkg-stats' script
instead of automatically calling 'read_nvd_dir'.
Since the cve.py file can be used as a library it's up to the caller to
decide whether or not to update the NVD database.
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 867017e736)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Given the following criteria: `cpe:2.3:a:oneidentitty:syslog-ng:*:*:*:*:-:*:*:*`.
The former `cpe_matches` implementation would match with the following
CPE: `cpe:2.3:a:oneidentitty:syslog-ng:4.71:*:*:*:premium:*:*:*`.
The 'hyphen' ('-') meaning is "Not Attributed" (NA) a criteria with no
attributed software edition shouldn't match with a CPE with an attributed
software edition:
https://csrc.nist.gov/pubs/ir/7695/final
This patch also create a distinct 'CPE' object that aggregate the
function specifics to CPEs like it's done for 'CVE'.
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 35f376d88e)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The `support/script/cve.py` file is used as a library. Depending on how
you use this library you might not want to write content to stdout when
calling its function.
This patch move the 'updating' log to the 'pkg-stats' script and write
the alert when LooseVersion doesn't have a version to stderr.
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 4b318dea17)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The each_cpe() method is careful that some CVEs have no
"configurations", but some CVEs such as
https://nvd.nist.gov/vuln/detail/CVE-2025-32915 apparently have a
"configurations" node, but no "nodes" inside the "configurations",
causing an exception:
Traceback (most recent call last):
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1382, in <module>
__main__()
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1371, in __main__
check_package_cves(args.nvd_path, packages)
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 679, in check_package_cves
check_package_cve_affects(cve, cpe_product_pkgs)
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 638, in check_package_cve_affects
for product in cve.affected_products:
^^^^^^^^^^^^^^^^^^^^^
File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 185, in affected_products
return set(cpe_product(p['id']) for p in self.each_cpe())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 185, in <genexpr>
return set(cpe_product(p['id']) for p in self.each_cpe())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 173, in each_cpe
for node in nodes['nodes']:
~~~~~^^^^^^^^^
KeyError: 'nodes'
Fixes:
54f8d97c91 ("support/scripts/pkg-stats: adapt to NVD v2 json format")
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Romain:
- add reference to buildroot commit introducing the issue
- a similar patch was sent by Daniel Lang (thanks!)
- needed on 2025.02.x to fix "Daily results" email]
(cherry picked from commit 67422b9d9c)
Signed-off-by: Romain Naour <romain.naour@smile.fr>
The NVD database contains some CPEs that are wrongly not associated
with any version number. They are for example sometimes associated
with very old CVEs.
Those CPEs are annoying, because they pollute our pkg-stat CVE results
with CVE entries which actually don't affect us.
The proper way to solve it is, and should remain, to fix the NVD
database by reporting these issues. Having to deal with a lot of
CVEs/CPEs, the NVD database is however slow to be updated.
To reduce the noise in our pkg-stats results in the meantime, one
possibility is to add <PKG_IGNORE_CVES> entries for those CVEs. This
however comes with the downside that even once the NVD database gets
fixed, those ignored entries risk remaining in Buildroot forever
because they are undetected.
This commit tries to address this downside by checking for and
reporting CVEs that are ignored in Buildroot, but where the
NVD reports our package version as unaffected. Those CVEs will appear
in the 'CVEs Ignored' column as '(stale)', and the cell will be
colored the same way warnings are. This should allow us to detect and
remove those entries.
It can be tested for example by adding the following variable to the
apache package (for a CVE that was recently fixed in the NVD database):
APACHE_IGNORE_CVES = CVE-1999-0236
Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 56ea5a0226)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The last usage of each_product() was removed in commit
52ae092046 ("support/scripts/cve.py: use
the JSON data in 1.1 schema").
Since it's now unused, remove it.
Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit d0a7a46813)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Annoyingly, using "--disable warning" does not disable the warnings
checks.
It turns out that we look for "warnings" (i.e. with an 's') to know if
we should disable the warnings check, so update the help text
accordingly.
Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit 92e7ab78d6)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The ARMV7-A toolchains are capable to compile binaries for ARMv8-A CPU
in AArch32 execution state.
This adds the BR2_ARM_CPU_ARMV8A option in the 'conditions' to allow
ARMV8-A CPU such as Cortex-A53 or Cortex-A72 to use ARMV7-A toolchains.
Signed-off-by: Gaël PORTAY <gael.portay+rtone@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 8fd537ae05)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Running pkg-stats is currently quite verbose, as it shows one line per
package when checking for the upstream URL, and another one line per
package when checking for the latest version on
release-monitoring.org.
This noisy output is a bit annoying when pkg-stats is run in a
cronjob, like we do to update https://autobuild.buildroot.net/stats/
every day. This commit adds a -v/--verbose option, off by default, to
have a less noisy output.
Suggested-by: Peter Korsgaard <peter@korsgaard.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 203e9def71)
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
SC2086 is now reported for auto_id since shellcheck 0.9.0:
In support/scripts/mkusers line 453:
add_one_group "${g}" ${auto_id}
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
So quote it to get rid of this error.
Signed-off-by: Romain Naour <romain.naour@smile.fr>
[Peter: quote variable instead of disabling check]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Rather than fork+exec:ing a whole bash process for every file of
interest, split up the work into ${PARALLEL_JOBS} separate file lists,
and then fork off a single worker per list.
Since workers now run in a fork, we also don't have to re-parse the
arguments on every call to patch_file().
In the worker, we also add a low-overhead check for the ELF magic,
which allows us to skip all non-ELFs without having to fork off
additional processes. In the event that we encounter an invalid ELF,
that is still taken care of by the existing exitcode-test in
patch_file().
On a 4-core/8-thread box, this cuts down the time it takes to fix the
RPATHS in the host directory from around 25s to 1.5s.
Signed-off-by: Tobias Waldekranz <tobias@waldekranz.com>
[Peter: fix shellcheck issues]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The commit 6889056f1e (support/scripts/genimage.sh) adds support for
creating bmap images.
Since then, the script genimage.sh looses the error returned by the
genimage command.
As a consequence, the target-finalize target looses the error as well,
and so on up to make.
This adds the shell option -e to exit immediately if a command ends with
an error, so the error is not ignored.
Fixes:
gportay@archlinux ~/src/rtone-br2-external $ make
(...)
>>> Executing post-image script support/scripts/genimage.sh
INFO: cmd: "mkdir -p "/home/gportay/src/rtone-br2-external/output/build/genimage.tmp"" (stderr):
INFO: cmd: "rm -rf "/home/gportay/src/rtone-br2-external/output/build/genimage.tmp"/*" (stderr):
INFO: cmd: "mkdir -p "/home/gportay/src/rtone-br2-external/output/build/genimage.tmp"" (stderr):
INFO: cmd: "cp -a "/tmp/tmp.EclKGwtA4R" "/home/gportay/src/rtone-br2-external/output/build/genimage.tmp/root"" (stderr):
INFO: cmd: "mkdir -p "/home/gportay/src/rtone-br2-external/output/images"" (stderr):
INFO: vfat(efi-part.vfat): cmd: "mkdosfs '/home/gportay/src/rtone-br2-external/output/images/efi-part.vfat'" (stderr):
INFO: vfat(efi-part.vfat): adding file 'efi-part/EFI' as 'EFI' ...
INFO: vfat(efi-part.vfat): cmd: "MTOOLS_SKIP_CHECK=1 mcopy -sp -i '/home/gportay/src/rtone-br2-external/output/images/efi-part.vfat' '/home/gportay/src/rtone-br2-external/output/images/efi-part/EFI' '::EFI'" (stderr):
Disk full
INFO: vfat(efi-part.vfat): cmd: "rm -f "/home/gportay/src/rtone-br2-external/output/images/efi-part.vfat"" (stderr):
ERROR: vfat(efi-part.vfat): failed to generate efi-part.vfat
INFO: cmd: "rm -rf "/home/gportay/src/rtone-br2-external/output/build/genimage.tmp/"" (stderr):
make[1]: Leaving directory '/home/gportay/src/rtone-br2-external/buildroot'
gportay@archlinux ~/src/rtone-br2-external $ echo $?
0
Signed-off-by: Gaël PORTAY <gael.portay+rtone@gmail.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
Nios2 will be removed from Buildroot, so Bootlin toolchains will
no longer provide nios2 toolchain in further releases.
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
When I initially made the CVE cells collapsible, I neglected to
count the unsure CVEs. This patch adds unsure CVEs to the cell collapsing
calcualation to ensure that cells with lots of unsure CVEs actually get collapsed.
This patch also removes the "+ 1" from the cve_total calculation,
which fixes the cve_total being off-by-one.
I'm not sure *why* I did that in the first place.
demo:
https://sen-h.codeberg.page/pkg-stats-demos/@pages/add-unsure_cves-to-cve_total-calc.html
Signed-off-by: Sen Hastings <sen@hastings.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
With the introduction of ARM FDPIC support, we need to make sure that
the existing ARM FLAT external toolchains are not made
visible/available when ARM FDPIC is selected. This commit updates the
gen-bootlin-toolchains script to take this into account.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit a87abcf6da (Makefile: run PPD and RPATH fixup in host-fialize)
(sic) moved the fixups peviously done in prepare-sdk, to host-finalize.
This exposed a bug in fix-rpath, when RPATH contains multiple entries,
like: /PPD/host-foo/host/lib:/PPD/host-foo/host/lib/foo
In that situation, we want to get rid of /PPD/host-foo and replace it
with the finale HOST_DIR, so we mangle the RPATH with a sed expression.
However, that sed expression only ever replaces the first match, as it
is missing the 'g' option. Thus, the second (and following) parts of
RPATH are still referring to the PPD, and thus patchelf does not find it
relative to the final HOST_DIR, amd rops it. This eventually lead to a
final RPATH set as $ORIGIN/../lib instead of the expected
$ORIGIN/../lib:$ORIGIN/../lib/foo
This is the case for host-systemd, which installs some of its libraries
in $PREFIX/lib/systemd/ and adds an RPATH set appropriately to
/PPD/host-systemd/host/lib:/PPD/host-systemd/host/lib/systemd and that
gets incorrectly mangled.
Fixes: https://gitlab.com/buildroot.org/buildroot/-/issues/39
Also fix a typo in the comment just above.
Reported-by: José Luis Salvador Rufo @jlsalvador
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: José Luis Salvador Rufo <salvador.joseluis@gmail.com>
Reviewed-by: José Luis Salvador Rufo <salvador.joseluis@gmail.com>
Tested-by: Adam Duskett <adam.duskett@amarulasolutions.com>
Reviewed-by: Adam Duskett <adam.duskett@amarulasolutions.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This reverts commit e0ed05cc00.
Since this commit has been applied, the update of the NVD data fails:
Checking packages CVEs
Updating from https://github.com/fkie-cad/nvd-json-data-feeds/
Traceback (most recent call last):
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1346, in <module>
__main__()
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1335, in __main__
check_package_cves(args.nvd_path, packages)
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 660, in check_package_cves
for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 105, in read_nvd_dir
CVE.download_nvd(nvd_git_dir)
File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 74, in download_nvd
subprocess.check_call(
File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'pull', '--depth', '1']' returned non-zero exit status 128.
Since we couldn't immediately figure out what's going on, let's revert
for now until this get investigated and fixed.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
As of the 2024.05 toolchain release, the toolchain tarballs are now
xz-compressed instead of bz2-compressed. Since this script is only
meant to update to the latest toolchains, we don't really bother
supporting the older toolchains that are bz2-compressed, and we simply
assume the tarballs are xz-compressed.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Just a simple clone and pull with --depth 1 should be enough to parse
the CVE data and generate the pkg-stats report.
From a full clone and a depth-1 clone, and the size delta is 2.9GiB
vs. 2.2GiB.
The download size does change: from 983.55MiB down to 270.78MiB.
It's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit 6889056f1e (support/scripts/genimage.sh: support creating a bmap
image) added unconditional gzip compression of the genimage outputs if
bmap-tools is enabled, which is problematic for a number of reasons:
- The gzip invocation is not reproducable (E.G. does not use -n)
- The original file is removed, dropping the sparse info
- It hardcodes policy, E.G. bmap supports a number of different compression
algorithms / compression may not be desired for all images / different
compression levels may be desired
So drop the compression step. If desired, compression can be done with a
post-image script.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Acked-by: TIAN Yuanhao <tianyuanhao3@163.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The patch adds the possibility to create, in addition to the usual
image, an image of type bmap that drastically reduces the amount of
data that needs to be written to an SD card, resulting in time
savings.
It looks at whether BR2_PACKAGE_HOST_BMAP_TOOLS is enabled to decide
whether it should or not generate bmap tool images. It generates bmap
images for all images referenced in the genimage configuration file,
as long as they exist in $(BINARIES_DIR).
Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The current implementation of buildroot depe dependency graphing
either does forward- or reverse-dependency traversal.
This patch enables buildroot to graph forward and reverse dependencies
on the graph for the same package: (Diagram Credit: Yann E. MORIN)
$ make pkg-d-graph-both-depends
pkg A -. .-> pkg E
\ /
pkg B ----> pkg D ----> pkg F
/ \
pkg C -' '-> pkg G
In the above example a single graph shows pkg {A,B,C} are needed
by pkg D, and pkg D is a dependency of pkg {E,F,G}.
Makefile help and manual are also updated.
Signed-off-by: Steve Hay <me@stevenhay.com>
[Arnout:
- remove DEPTH and RDEPTH, their functionality is already covered by
BR2_GRAPH_DEPS_OPTS;
- remove --rdepth, it was felt to not add sufficient added value;
- add the new target to the manual;
- fix flake8 errors.
]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Instead of having done_deps as a global variable, pass it as an argument
to the print_pkg_deps function. This will allow the recursive search to
be started multiple times.
Signed-off-by: Steve Hay <me@stevenhay.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
This patch was created as a fix to a problem that occurred during the
compilation of QEMU:
>>> qemu 8.1.1 Patching
Applying 0001-tests-fp-disable-fp-bench-build-by-default.patch using patch:
patching file tests/fp/meson.build
Hunk #1 succeeded at 138 with fuzz 2 (offset -502 lines).
Applying 0002-softmmu-qemu-seccomp.c-add-missing-header-for-CLONE_.patch using patch:
patching file softmmu/qemu-seccomp.c
Applying 0004-tracing-install-trace-events-file-only-if-necessary.patch using patch:
patching file trace/meson.build
With the bump to version 8.1.1, the patch that disabled the compilation
of the fp-bench test does not report any errors, even though the patch
itself is no longer applicable. The only noticeable message is:
"Hunk #1 succeeded at 138 with fuzz 2 (offset -502 lines)."
As reported by the patch man page:
"With context diffs, and to a lesser extent with normal diffs, patch can
detect when the line numbers mentioned in the patch are incorrect, and
attempts to find the correct place to apply each hunk of the patch.
As a first guess, it takes the line number mentioned for the hunk, plus
or minus any offset used in applying the previous hunk. If that is not
the correct place, patch scans both forwards and backwards for a set of
lines matching the context given in the hunk. First patch looks for a
place where all lines of the context match. If no such place is found,
and it's a context diff, and the maximum fuzz factor is set to 1 or more,
then another scan takes place ignoring the first and last line of
context. If that fails, and the maximum fuzz factor is set to 2 or more,
the first two and last two lines of context are ignored, and another
scan is made. The default maximum fuzz factor is 2.
If the hunk is installed at a different line from the line number
specified in the diff, you are told the offset. A single large offset
may indicate that a hunk was installed in the wrong place. You are also
told if a fuzz factor was used to make the match, in which case you
should also be slightly suspicious."
By setting the maximum fuzz factor to 0, we avoid that patches which
cannot be applied are incorrectly reported as valid, with positive
side-effects on version bumps.
Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Commit f77d698f83 changed apply-patches to
use $TAR instead of `tar`, but did not define a fallback if $TAR is not
defined. This results in an error when calling apply-patches.sh from
outside Buildroot's Makefile. Our team uses this script to setup local
checkouts of Buildroot package's with patches for development.
Signed-off-by: Brandon Maier <brandon.maier@collins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Note that here, it is in fact not really relevant. We only extract a
tarball, and we don't use any "modern" or GNU-only options like
--strip-components. However, for consistency it's better to use the same
tar everywhere.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: "Maier, Brandon L Collins" <Brandon.Maier@collins.com>
Reviewed-by: brandon.maier@collins.com
[Arnout: quote TAR="..."]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
sortGrid()'s handling of git hashes and other large hex numbers
has been inconsistent, they can be detected as strings or numbers
depending on what type of character they start with.
This patch fixes the behaviour by using a regex to capture everything
that looks like a big hex number and treat it as a string.
This means when you sort by current version ascending all the version
strings with big hex numbers should show up first, sorted 0-9,a-f.
First we check for a string length >= 39, and then apply a regex
to return an array with every char from that string that matched
the regex. If the length of this array is still >= 39 we can assume
we are looking at something containing a git hash.
The reason why the length is defined as ">= 39" and not "40" or
"39 or 40" is twofold:
Firstly, 39 was chosen as a minimum to match stuff with 39 char git
hashes, like the rockchip-mali package.
Secondly, there is no max because we actually want to catch not
just explicitly git hashes, but any verson string with big gnarly
hex numbers in it.
Stuff like: "1.4.2-168-ged3039cdbeeb28fc0011c3585d8f7dfb91038292"
Why? Well, the idea is less about git hashes and sorting
and more about grouping similarly formatted version strings.
It would be impossble (or at least annoyingly complicated) and of
dubious utility to get a real sequential sort out of the
current version column, so the attempt here is to at the very
least collect all the similarly formatted things together.
This isn't perfect, but it's a (arguably) more useful sorted
output than before.
A demo is available here:
https://sen-h.codeberg.page/pkg-stats-demos/@pages/fix-improve-git-hash-sorting.html
Signed-off-by: Sen Hastings <sen@hastings.org>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Because the div_class variable was not reassigned a value,
cells in the latest_version column were still being assigned
hash_file classes and so were being picked up as elements in
the hash_file sort. This lead to execessive grid gap elements
stacking up and creating huge blank spaces at the top of the page.
This is very noticable on pages with a large number of packages,
like the ones the autobuilder creates.
original behaviour(click the "Hash file" column label twice):
http://autobuild.buildroot.org/stats/master.html
demo of fixed behaviour:
https://sen-h.codeberg.page/pkg-stats-demos/@pages/fix-bug-when-sorting-by-hash-file.html
Signed-off-by: Sen Hastings <sen@hastings.org>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
the user tables do not use trailing backslash \ to continue lines,
so we don't want them to be interpreted thusly, so we use 'read -r'
(SC2162).
Integer variables need not be quoted (SC2086). In any case, should
there be an actual issue and they be set empty, that would cause a
runtime issue, wether they be quoted or not.
The binary -o and -a ar perfectly defined in bash's test (SC2166).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 41ea61d59c (support/scripts/mkusers: allow option for system
uid/gid) confused GID and UID variables: the GID limits were used to
create UIDs.
Fix that.
Note that this fixes a shellcheck error; although there are many more
shellcheck errors, these fixes are semantically a bug that need to be
fixed separately from the coding style issues reported by shellcheck.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Norbert Lange <nolange79@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, we grab the per-year CVE feeds, in two passes: first, we grab
the meta files, and check whether something has changed since last we
downloaded it; second, we download the feed proper, unless the meta file
has not changed, in which case we use the locally cached feed.
However, it has appeared that the FKIE releases no longer provide the
meta files, which means that (once again), our daily reports are broken.
The obvious fix would be to drop the use of the meta file, and always
and unconditionally download the feeds. That's relatively trivial to do,
but the feeds are relatively big (even as xz-xompressed).
However, the CVE database from FKIE is available as a git tree. Git is
pretty good at only sending delta when updating a local copy. In
addition, the git tree, contains each CVE as an individual file, so it
is relatively easier to scan and parse.
Switch to using a local git clone.
Slightly surprisingly (but not so much either), parsing the CVE files is
much faster when using the git working copy, than it is when parsing the
per-year feeds: indeed, the per-year feeds are xz-compressed, and even
if python is slow-ish to scan a directory and opening files therein, it
is still much faster than to decompress xz files. The timing delta [0]
is ~100s before and ~10s now, about a ten time improvement, over the
whole package set.
The drawback, however, is that the git tree is much bigger on-disk, from
~55MiB for the per-year compressed feeds, to 2.1GiB for the git tree
(~366MiB) and a working copy (~1.8GiB)... Given very few people are
going to use that, that's considered acceptable...
Eventually, with a bit of hacking [1], the two pkg-stats, before and
after this change, yield the same data (except for the date and commit
hash).
[0] hacking support/scripts/pkg-stats to display the time before/after
the CVE scan, and hacking support/scripts/cve.py to do no download so
that only the CVE scan happens (and also because the meta files are no
longer available).
[1] sorting the CVE lists in json, sorting the json keys, and using the
commit from the FKIE git tree that was used for the current per-year
feeds.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Commit 22b6945552 (support/scripts/cve.py: switch from NVD to FKIE for
the JSON files) had to change the decompressor from gz to xz, as the new
location is using xz compression.
That commit mentioned that it was spawning an external xz process to do
the decompression, on the pretence that "there is no xz decompressor in
Python stdlib."
Before version 3.1, ijson.items() only accepted a file-like object as
input (that file-like object could yield bytes() or str(), both were
supported). Starting with version 3.1, ijson.items() also accepts that
it be directly passed bytes() or str() directly. subprocess.check_output()
means we are now passing bytes() to ijson.items(), so it fails on ijson
versions before 3.1, with failures such as:
[...]
File "/usr/lib/python3/dist-packages/ijson/backends/python.py", line 25, in Lexer
if type(f.read(0)) == bytetype:
AttributeError: 'bytes' object has no attribute 'read'
Ubuntu 20.04, on which the pkg-stats run to generate the daily report,
only has ijson 2.3. More recent distros have more recent versions of
ijson, like Fedora 39 that has 3.2.3, recent enough to support being fed
bytes(). Commit 22b6945552 was tested on Fedora 39, so did not catch
the issue.
However, the reasoning in 22b6945552 is wrong: there *is* the lzma
module, at least since python 3.3 (that is, aeons ago), which is able to
read xz-compressed files; it also has an API similar to the gzip module,
and can provide a file-like object that exposes the decompressed data.
So, do just that: provide an lzma-wrapped file-like object to ijson, so
that we can eventually recover our daily reports that everything is
broken! :-]
Note that this construct still works on recent versions!
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
