The __name__ == '__main__' guard allows importing pkg-stats as a
module using importlib, circumventing the normal module filename
requirements. This in turn makes it possible to test/debug individual
functions.
Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit 8cde69e101)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The file handle can be iterated over directly and each line is used
exactly once, so the only effect of reading all lines into a list
first was higher memory use and complexity.
Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit 07f7ad9898)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The filter is supposed to exclude host/target infra from output if the
respective package is not built with the current
configuration.
However, excluding host packages did not work correctly: If keep_host
is False because the host package is not built, the next branch was
checked and included the host infra in output with "target" type if
the target package is built. For a package that support host and
target build, but gets built only for the target, this leads to output
like (Meson example):
meson (target)
host-meson (target)
Skip host infra in the target branch instead. Also include
Package.infra in Package.__str__() result, which was needed for
debugging this bug.
Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit e8dcebf459)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Use only one of the classes for "error" or "warning" status so they
look different, and format the error/warning text for both. Do not
make the text a link if the URL is None.
Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit 7961bd10b9)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The previous Config.* glob also caught linux/Config.ext.in and
package/php/Config.ext, as well as some backup files created by
editors (e.g. Config.in~ after editing a Config.in file in Emacs),
leading to wrong results depending on directory listing order.
Also use "with" to automatically close the file when the block is
left, even on error.
Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit 77a464969c)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
This commit adds the -N/--needs-update option, disabled by default,
to list only packages with newer upstream versions. All other packages
will be excluded from the HTML or JSON output.
Signed-off-by: Kadambini Nema <kadambini.nema@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit ed9466e7f9)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
When running "make pkg-stats" on a host with Python 3.14 (e.g.
Fedora 43 for example), the execution fails with the error:
Checking URL status
Traceback (most recent call last):
File "/buildroot/support/scripts/pkg-stats", line 1387, in <module>
__main__()
~~~~~~~~^^
File "/buildroot/support/scripts/pkg-stats", line 1368, in __main__
loop = asyncio.get_event_loop()
File "/usr/lib64/python3.14/asyncio/events.py", line 715, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
% threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'MainThread'.
This is due to a breaking change introduced in Python 3.14
asyncio.get_event_loop(). See [1]. Before Python 3.14, this call was
creating and setting an event loop if there was none. This situation
is now a runtime error.
In order to fix this issue with newer Python version, while keeping
backward compatibility, this commit replaces the code:
loop = asyncio.get_event_loop()
by an explicit event loop creation:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
This commit was tested on a Fedora 43 host with Python-3.14.2, and
with the Buildroot Docker image plus the python3-aiohttp package
which is a Debian 12 with Python-3.11.2.
[1] https://docs.python.org/3.14/library/asyncio-eventloop.html#asyncio.get_event_loop
Signed-off-by: Julien Olivain <ju.o@free.fr>
Tested-by: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit e9f426aa52)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
As requested by the gnu.org admins:
"
Since August 2024, we've been under DDoS attacks from common command line
tools.
To fix this, we would need to change the user-agent from "Python/3.11
aiohttp/3.8.4" to "buildroot.org pkg-stats" instead.
"
It indeed probably makes sense to use an unique user-agent string, so rework
the script to do that.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit 0981b4117d)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The gnu.org admins have been blocking the IP address of machines running
pkg-stats as the GET requests for the (many) packages with gnu.org URLs are
seen as abusive.
The resource body is not used, so use a HTTP HEAD request instead of a GET
to limit server load and bandwidth use.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit f9f3e6ccc6)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
This patch move the 'download_nvd' call to the 'pkg-stats' script
instead of automatically calling 'read_nvd_dir'.
Since the cve.py file can be used as a library it's up to the caller to
decide whether or not to update the NVD database.
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 867017e736)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Given the following criteria: `cpe:2.3:a:oneidentitty:syslog-ng:*:*:*:*:-:*:*:*`.
The former `cpe_matches` implementation would match with the following
CPE: `cpe:2.3:a:oneidentitty:syslog-ng:4.71:*:*:*:premium:*:*:*`.
The 'hyphen' ('-') meaning is "Not Attributed" (NA) a criteria with no
attributed software edition shouldn't match with a CPE with an attributed
software edition:
https://csrc.nist.gov/pubs/ir/7695/final
This patch also create a distinct 'CPE' object that aggregate the
function specifics to CPEs like it's done for 'CVE'.
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 35f376d88e)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The `support/script/cve.py` file is used as a library. Depending on how
you use this library you might not want to write content to stdout when
calling its function.
This patch move the 'updating' log to the 'pkg-stats' script and write
the alert when LooseVersion doesn't have a version to stderr.
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 4b318dea17)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
The NVD database contains some CPEs that are wrongly not associated
with any version number. They are for example sometimes associated
with very old CVEs.
Those CPEs are annoying, because they pollute our pkg-stat CVE results
with CVE entries which actually don't affect us.
The proper way to solve it is, and should remain, to fix the NVD
database by reporting these issues. Having to deal with a lot of
CVEs/CPEs, the NVD database is however slow to be updated.
To reduce the noise in our pkg-stats results in the meantime, one
possibility is to add <PKG_IGNORE_CVES> entries for those CVEs. This
however comes with the downside that even once the NVD database gets
fixed, those ignored entries risk remaining in Buildroot forever
because they are undetected.
This commit tries to address this downside by checking for and
reporting CVEs that are ignored in Buildroot, but where the
NVD reports our package version as unaffected. Those CVEs will appear
in the 'CVEs Ignored' column as '(stale)', and the cell will be
colored the same way warnings are. This should allow us to detect and
remove those entries.
It can be tested for example by adding the following variable to the
apache package (for a CVE that was recently fixed in the NVD database):
APACHE_IGNORE_CVES = CVE-1999-0236
Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 56ea5a0226)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Annoyingly, using "--disable warning" does not disable the warnings
checks.
It turns out that we look for "warnings" (i.e. with an 's') to know if
we should disable the warnings check, so update the help text
accordingly.
Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit 92e7ab78d6)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Running pkg-stats is currently quite verbose, as it shows one line per
package when checking for the upstream URL, and another one line per
package when checking for the latest version on
release-monitoring.org.
This noisy output is a bit annoying when pkg-stats is run in a
cronjob, like we do to update https://autobuild.buildroot.net/stats/
every day. This commit adds a -v/--verbose option, off by default, to
have a less noisy output.
Suggested-by: Peter Korsgaard <peter@korsgaard.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 203e9def71)
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
When I initially made the CVE cells collapsible, I neglected to
count the unsure CVEs. This patch adds unsure CVEs to the cell collapsing
calcualation to ensure that cells with lots of unsure CVEs actually get collapsed.
This patch also removes the "+ 1" from the cve_total calculation,
which fixes the cve_total being off-by-one.
I'm not sure *why* I did that in the first place.
demo:
https://sen-h.codeberg.page/pkg-stats-demos/@pages/add-unsure_cves-to-cve_total-calc.html
Signed-off-by: Sen Hastings <sen@hastings.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
sortGrid()'s handling of git hashes and other large hex numbers
has been inconsistent, they can be detected as strings or numbers
depending on what type of character they start with.
This patch fixes the behaviour by using a regex to capture everything
that looks like a big hex number and treat it as a string.
This means when you sort by current version ascending all the version
strings with big hex numbers should show up first, sorted 0-9,a-f.
First we check for a string length >= 39, and then apply a regex
to return an array with every char from that string that matched
the regex. If the length of this array is still >= 39 we can assume
we are looking at something containing a git hash.
The reason why the length is defined as ">= 39" and not "40" or
"39 or 40" is twofold:
Firstly, 39 was chosen as a minimum to match stuff with 39 char git
hashes, like the rockchip-mali package.
Secondly, there is no max because we actually want to catch not
just explicitly git hashes, but any verson string with big gnarly
hex numbers in it.
Stuff like: "1.4.2-168-ged3039cdbeeb28fc0011c3585d8f7dfb91038292"
Why? Well, the idea is less about git hashes and sorting
and more about grouping similarly formatted version strings.
It would be impossble (or at least annoyingly complicated) and of
dubious utility to get a real sequential sort out of the
current version column, so the attempt here is to at the very
least collect all the similarly formatted things together.
This isn't perfect, but it's a (arguably) more useful sorted
output than before.
A demo is available here:
https://sen-h.codeberg.page/pkg-stats-demos/@pages/fix-improve-git-hash-sorting.html
Signed-off-by: Sen Hastings <sen@hastings.org>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Because the div_class variable was not reassigned a value,
cells in the latest_version column were still being assigned
hash_file classes and so were being picked up as elements in
the hash_file sort. This lead to execessive grid gap elements
stacking up and creating huge blank spaces at the top of the page.
This is very noticable on pages with a large number of packages,
like the ones the autobuilder creates.
original behaviour(click the "Hash file" column label twice):
http://autobuild.buildroot.org/stats/master.html
demo of fixed behaviour:
https://sen-h.codeberg.page/pkg-stats-demos/@pages/fix-bug-when-sorting-by-hash-file.html
Signed-off-by: Sen Hastings <sen@hastings.org>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Commit f71d9f49e5 (support/scripts/pkg-stats: fix datetime deprecation
warning) forgot to consider that the datetime.UTC suggested by python
3.12, was only introduced with python 3.11.
However, we are still generating the daily report on a python 3.8
version, which fails at runtime:
AttributeError: module 'datetime' has no attribute 'UTC'
It turns out that datetime.UTC is just an alias for datetime.timezone.utc,
which seems to have existed since before python3...
Use datetime.timezone.utc instead of its alias.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
While the old NVD JSON feed provided data files where the CVEs were
sorted by ID, the new feed from FKIE does not have sorted CVEs.
Add a method to sort a list of CVE IDs (i.e. CVE ID strings, not CVE
objects!), and use that when emiting the HTML output.
The JSON output need not be sorted, because it is supposed to be used
for post-processing, and we do not care about the ordering there; a
consumer interested in sorting should sort on their side.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Abide by the warning reported with python 3.12:
.../support/scripts/pkg-stats:1289: DeprecationWarning:
datetime.datetime.utcnow() is deprecated and scheduled for removal
in a future version. Use timezone-aware objects to represent
datetimes in UTC: datetime.datetime.now(datetime.UTC).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Currently, when the version string is "too long", it is arbitrarily
truncated.
This works well for commit hashes, because usually the truncation is
long enough to provide a short hash that is still unique in the
upstream VCS.
However, there are non-hash-like versions strings that get truncated
and wihch the discriminant part is toward the end.
Yet, adapting the version cell to the widest versions string (most
probably a git hash) is not very interesting; the table is already very
large.
Make the cell with the version string scrollable: we get to keep the
best of both worlds: a narrow version cell, and a full-length version
string that can be copy-pasted if needed.
Signed-off-by: Sen Hastings <sen@hastings.org>
[yann.morin.1998@free.fr: reword commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
We currently check the CPE database to see if the CPE ID we use
(including the version) is already in the database.
However, the version part of the CPE ID is not actually used for CVE
matching. Instead, the CVEs have a range of versions associated with
them and we match against those ranges.
In addition, NVD is moving to a new API for accessing the CPE database.
It will not longer be possible to simply download all the CPE IDs, and
due to rate limiting, the download will have to be done in several
queries.
Since all of this is anyway of limited use, drop the CPE database lookup
entirely. Instead, as long as a CPE ID is defined in a package, it is
considered OK, without any checks.
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Search the external trees for package files and add them to the list.
The list of directories walked and excluded are the same as for the main
tree, and should work out of the box if the user sticks to the directory
structure suggested in the manual.
Two additional properties were added to the Package class, the tree name and
the path. For consistency and to simplify the code, packages in the main tree
are marked as coming from "BR2".
The HTML output has a new column listing the external name (or "BR2") and the
json output has a new property "tree".
Signed-off-by: Juan Carrano <juan.carrano@ebee.berlin>
[Arnout:
- fix flake8 error "'itertools' imported but unused";
- use str.split instead of str.partition;
- use BR2_EXTERNAL_BUILDROOT_PATH instead of BR2_EXTERNAL_BR2_PATH;
- remove pkgdir variable, instead use self.pkgdir.
]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Instead of only checking .mk and Config.in{,.host}, check
all files in a package directory.
.checkpackageignore isn't considered here, therefore the shown number
includes ignored warnings as well.
Add another css class to signal some warning, compared to a lot (>5),
similar to patches.
Signed-off-by: Daniel Lang <dalang@gmx.at>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some packages are grouped and have a general makefile that defines
reusable variables. These makefiles have no relevant information for
pkg-stats and should be excluded.
Signed-off-by: Daniel Lang <dalang@gmx.at>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The intention of this script is to generate the XML that can be sent to
NVD to request a new CPE identifier.
As discussed on the mailing list [0] keeping up with version numbers of
all registered CPE ID won't work.
In addition the feed used to generated the XML files will be retired
[1]. In the future an API needs to be used for fetching the data in
connection with a local database.
All of this works against keeping this script and porting it to the new
API.
As a last blow Matthew, the original author concluded [2]:
> Makes sense to drop it. There never got to be enough momentum in the overall
> software community to make CVE or even the new identifier really accurate.
The intention is to ignore the version part of CPE IDs in the future,
and only look at the version range specified on a CVE. Therefore, a tool
to add new CPE ID versions isn't useful to us. It might still be useful
to have a tool to create the vendor and project parts of a CPE ID.
However, the current gen-missing-cpe tool doesn't support that, and the
API is anyway going to be retired. So there is no reason at all to keep
this around.
Remove gen-missing-cpe and the cpedb module. Remove the Makefile target
to call the script.
Since the cpedb module is removed, the CPEDB_URL definition must be
moved to the place where it is still used, in pkg-stats.
[0]: https://lists.buildroot.org/pipermail/buildroot/2023-August/672620.html
[1]: https://nvd.nist.gov/General/News/change-timeline
[2]: https://lists.buildroot.org/pipermail/buildroot/2023-August/672651.html
Signed-off-by: Daniel Lang <dalang@gmx.at>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Fixes flake8 errors:
support/scripts/pkg-stats:1013:133: E501 line too long (164 > 132 characters)
support/scripts/pkg-stats:1018:36: F541 f-string is missing placeholders
support/scripts/pkg-stats:1110:199: E261 at least two spaces before inline comment
For the first and the last one, we chose to split the long lines rather
than adding noqa: 501. Indeed, the long lines make it very unreadable,
and there are relatively natural places where the line can be broken.
Also split a line just below the second one in a similar way.
The f-string on 1018 doesn't need to be an f-string.
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
This is done either by switching to single quoted f-strings, triple
double quoted f-strings when needed, or simply single-quoted strings.
The renderer HTML is exactly identical before/after this commit.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-By: Sen Hastings <sen@phobosdpl.com>
Acked-By: Sen Hastings <sen@phobosdpl.com>
Within single-quoted f-strings, and within triple double quoted
strings, escaping all the double quotes is completely useless and
makes the code more difficult to read. Get rid of all this useless
escaping.
The renderer HTML is exactly identical before/after this commit.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-By: Sen Hastings <sen@phobosdpl.com>
Acked-By: Sen Hastings <sen@phobosdpl.com>
This attempts to re-implement the "sortable_hint" feature without
relying on words. The column headers and CVE expand/contract buttons
change color and cursor style on hover.
If Javascript is enabled:
Just like [PATCH 1/3] more rules are applied to the
generated stylesheet before content is loaded.
If Javascript is disabled:
The headers stay pearly white. :-)
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Sometimes a package can have a lot of CVEs.
Rather than have the CVE cell make a really tall row
(that means you have to scroll a bunch) this collapses the CVE
cell to a fixed size scrollable element with a
sticky button that lets you expand and collapse it.
If Javascript is enabled:
A stylesheet is generated and appended before content rendering,
amending the cells style to have a fixed height and overflow.
Also, the expand/contract button is unhidden.
This means the CVE cells are rendered in a collapsed state
instead of being rendered in an expanded state and then
slamming shut.
This avoids a "flash" and *helps* (vertically at least) manage CLS
(cumulative layout shift).
see: https://web.dev/cls/
If Javascript is disabled:
The cells stay fully open and the expand/contract button stays hidden.
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
sortGrid() has been rewritten to dynamically generate stylesheets with
explicit grid-row properties to re-order the rows, instead of removing
and reinserting the cells.
Performance *should* now be comperable to sorttable.js.
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Like all good problems, disparate pieces work together to create
a "synergistically" hairy mess.
The sortGrid() overhaul highlighted a flaw in pkg-stats allowing
for duplicate package class names across rows.
As an example,
boot/barebox/barebox.mk and boot/barebox/barebox/barebox.mk
both get the classname ._barebox and so sortGrid() sticks them on
the same line giving a table with a vestigal row sticking out
of the right side like some kind of appendage.
Also I neglected to add a "_" to the current version column's cells
pkgname class so instead of "._pkgname" we had ".pkgname" and so
the cells were not collected properly as part of the row.
These issues explain the formatting weirdness.
package classnames are now ".path_to_package_makefile" without suffix
(.mk) (so ._boot_barebox_barebox and ._boot_barebox_barebox_barebox
instead of ._barebox) in order to guarantee uniqueness.
and what was *accidentally*
class="centered current_version data .barebox" is now
class="centered current_version data ._boot_barebox_barebox"
just like *all* the other cells in the row. :p
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The 'rtl8723ds' name, when queried from release-monitoring.org at
https://release-monitoring.org/api/projects/?pattern=rtl8723ds returns
one project, with one "stable_versions" array, which is empty. This
was not expected by the pkg-stats code, causing an exception:
Task exception was never retrieved
future: <Task finished name='Task-764' coro=<check_package_latest_version_get() done, defined at /home/thomas/projets/buildroot/./support/scripts/pkg-stats:558> exception=IndexError('list index out of range')>
Traceback (most recent call last):
File "/home/thomas/projets/buildroot/./support/scripts/pkg-stats", line 566, in check_package_latest_version_get
if await check_package_get_latest_version_by_guess(session, pkg):
File "/home/thomas/projets/buildroot/./support/scripts/pkg-stats", line 544, in check_package_get_latest_version_by_guess
projects[0]['stable_versions'][0],
IndexError: list index out of range
This commit therefore improves the checks done on the results received
from release-monitoring.org to avoid this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
CSS classes are generated for each package name for sorting purposes,
However some package names start with a number and this is not allowed.
(see https://www.w3.org/TR/CSS2/syndata.html#value-def-identifier)
Fix is to prepend a character to every class name such as "_".
so every ".package" is now "._package".
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This fixes the .version-needs-update class being overridden by .correct class.
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When we use the statistics output to generate a CVE/CPE customer
report showing whether a product is affected by CVEs, we are primarily
interested in whether they are relevant to the target
system. Currently we cannot see if the package is configured for the
build (infra==host) and/or the target system (infra==target).
Therefore this commit extends the pkg-stats script to leverage the
information available in "make show-info" output to tweak the list of
package infrastructures for each package. Thanks to this commit, the
script now has a more consistent behavior:
* When pkg-stats is run without -c, i.e without a defined Buildroot
configuration, it continues to operate as it did, i.e it lists all
package infrastructures supported by the package (such as autotools
host+target, or kconfig target, etc.)
* When pkg-stats is run with -c, i.e with a defined Buildroot
configuration which defines the list of packages that should be
considered, then for each package it now lists only the package
infrastructures used by the package in that current
configuration. For example if you have a package with a host and
target variant, but only the host variant is used in your
configuration, now the pkg-stats output will only say that the host
variant of this package is used;
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[Thomas: pretty much rework the entire implementation and how the
result is presented.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit f1bcb2a45c introduced a number of
flake8 errors. Fix these by:
- adding noqa to the multi-line string containing tabs;
- replacing other tabs with spaces;
- removing space after opening parenthesis;
- splitting the long lines.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This migrates pkg-stats.html from html tables to CSS grid, allowing
the use of newer, simpler javascript that is short enough to be
inlined, instead of relying on externally hosted javascript.
Javascript sorting function was rewritten from scratch in ~55 lines,
short enough to be inlined directly in the html.
Tables were redone in CSS grid, but with care taken to mimic existing
"look and feel" of prevous implementation, albeit with slightly
better responsive behavior and default styling characteristics.
Column labels are now "sticky" and stay stuck to the top of the
viewport as you scroll down the page.
Also, css was rewritten in fewer lines and table elements were changed
to divs (for grid support).
Other small misc fixes include quoted hrefs and document language
declarations to make the w3c html validator happy.
Signed-off-by: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
pkg-stats currently uses the services from support/scripts/cpedb.py to
match the CPE identifiers of packages with the official CPE database.
Unfortunately, the cpedb.py code uses regular ElementTree parsing,
which involves loading the full XML tree into memory. This causes the
pkg-stats process to consume a huge amount of memory:
thomas 1310458 85.2 21.4 3708952 3450164 pts/5 R+ 16:04 0:33 | | \_ python3 ./support/scripts/pkg-stats
So, 3.7 GB of VSZ and 3.4 GB of RSS are used by the pkg-stats
process. This is causing the OOM killer to kick-in on machines with
relatively low memory.
This commit reimplements the XML parsing needed to do the CPE matching
directly in pkg-stats, using the XmlParser functionality of
ElementTree, also called "streaming parsing". Thanks to this, we never
load the entire XML tree in RAM, but only stream it through the
parser, and construct a very simple list of all CPE identifiers. The
max memory consumption of pkg-stats is now:
thomas 1317511 74.2 0.9 381104 152224 pts/5 R+ 16:08 0:17 | | \_ python3 ./support/scripts/pkg-stats
So, 381 MB of VSZ and 152 MB of RSS, which is obviously much better.
The JSON output of pkg-stats for the full package set, before and after
this commit, is exactly identical.
Now, one will probably wonder why this isn't directly changed in
cpedb.py. The reason is simple: cpedb.py is also used by
support/scripts/missing-cpe, which (for now) heavily relies on having
in memory the ElementTree objects, to re-generate a snippet of XML
that allows us to submit to NIST new CPE entries.
So, future work could include one of those two options:
(1) Re-integrate cpedb.py into missing-cpe directly, and live with
two different ways of processing the CPE database.
(2) Rewrite the missing-cpe logic to also be compatible with a
streaming parsing, which would allow this logic to be again
shared between pkg-stats and missing-cpe.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- add missing import of requests
- import CPEDB_URL from cpedb, instead of duplicating it
- fix flake8 errors
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Some upstream sites are very slow to respond, and the default timeout
of 300 seconds of the aiohttp.ClientSession() is too long. Let's
reduce it to 15 seconds.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The check_package_get_latest_version_by_distro() function analyzes the
data returned by release-monitoring.org. For two of our
packages (bento4 and qextserialport), release-monitoring.org returns
something that is a bit odd: it returns an entry with a
"stable_versions" field that contains an empty array. Our code was
ready to have or not have a "stable_versions" entry, but when it is
present, we assumed it was not an empty array. These two packages, for
some reason, break this assumption.
In order to solve this problem, this commit is more careful, and uses
the stable_versions field only if it exists and it has at least one
entry. The code is also reworked as a sequence of "if...elif...else"
to be more readable.
This fixes the following exception when running pkg-stats on the full
package set:
Task exception was never retrieved
future: <Task finished name='Task-10772' coro=<check_package_latest_version_get() done, defined at ./support/scripts/pkg-stats:532> exception=IndexError('list index out of range')>
Traceback (most recent call last):
File "./support/scripts/pkg-stats", line 535, in check_package_latest_version_get
if await check_package_get_latest_version_by_distro(session, pkg):
File "./support/scripts/pkg-stats", line 489, in check_package_get_latest_version_by_distro
version = data['stable_versions'][0] if 'stable_versions' in data else data['version'] if 'version' in data else None
IndexError: list index out of range
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: non-sequence tests as True]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit 471ecea5ee (core/show-info: 'name' only applies to packages)
removed the 'name' field for rootfs (really, for non-package) entries,
thus breaking the pkg-stats processing.
We fix that by excluding any entry that has no 'name', on the assumption
that if it has no name, it is not a package.
Reported-by: Xogium on IRC
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
