From aacca7f0639bac6bb3fb262d0b1738c4387edeb0 Mon Sep 17 00:00:00 2001 From: Nicolas Carrier Date: Tue, 17 Jan 2023 12:04:40 +0100 Subject: [PATCH] package/pkg-generic.mk: local and override support in legal-info Sources of packages which are using the 'local' _METHOD_SITE or which declare an _OVERRIDE_SRC_DIR are not archived when generating the legal-info target. This is problematic since those packages are the most likely to require being published, because it's highly probable that their sources have been modified. A typical use case is a package (e.g. linux) that is maintained as a git submodule of the BR2_EXTERNAL and with the LINUX_OVERRIDE_SRCDIR set appropriately, or with a tool like repo putting it in a well-known location. This patch generates a tar.gz archive on the fly, containing the result of the rsync of the package's source directory, using the same arguments as during the preparation of a build. Note that is not possible to archive the builddir, since it may contain artifacts of previous compilations. In order to create a tarball that is reproducible, we reuse the mk_tar_gz function from the download helpers. We have to provide it with the TAR variable. We also have to cd to the build directory first, because the script generates a temporary file in the current directory. That script needs a date. We unforuntately don't have a good reproducible data. The timestamps of the files in the override srcdir itself aren't useful either: if they come from git, they typically have the time of checkout. Therefore, we fix the date at Jan 1, 1970 (which we can do with @0). The mk_tar_gz function could actually work directly in the override srcdir without the need for an rsync - except for one limitation: it doesn't have the equivalent of RSYNC_VCS_EXCLUSIONS. It would be nice to refactor the script so it could accept a list of exclusions instead of a single one and also work at any level instead of just the toplevel. But that is for future improvement. It is not possible to consider rsync as a "normal" download method either (which would have solved the initial problem), because if we do that: * the DL dir's content would be altered improperly * the rsync + tar gz overhead would be paid on compilation and is significant, for example, for a linux kernel * a hash file would be necessary, which would be a burden to maintain, for sources frequently modified Signed-off-by: Nicolas Carrier [Arnout: - Drop the auxiliary variables: their use is contained within one code block and it's easier to read with being explicit. - Put it in an additional subdirectory .legal-info-rsync of the build directory, to reduce the chance of conflict with an existing file/directory. - Archive the directory (e.g. linux-custom) rather than its contents, so that it matches a normal tarball. - Use the mk_tar_gz function from download helpers to create a reproducible tarball. - Split long lines. ] Signed-off-by: Arnout Vandecappelle --- package/pkg-generic.mk | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 365d6c748d..171163dcb4 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -1155,13 +1155,20 @@ else endif # license files ifeq ($$($(2)_REDISTRIBUTE),YES) -ifeq ($$($(2)_SITE_METHOD),local) -# Packages without a tarball: don't save and warn - @$$(call legal-warning-nosource,$$($(2)_RAWNAME),local) - -else ifneq ($$($(2)_OVERRIDE_SRCDIR),) - @$$(call legal-warning-nosource,$$($(2)_RAWNAME),override) - +ifneq ($$($(2)_OVERRIDE_SRCDIR),) +# local/override packages: copy it and archive the copy + @echo "Package is of type local or override, archive sources" + $$(Q)rm -rf $$($(2)_BUILDDIR)/.legal-info-rsync + $$(Q)mkdir -p $$($(2)_BUILDDIR)/.legal-info-rsync + $$(Q)rsync -au --chmod=u=rwX,go=rX $$(RSYNC_VCS_EXCLUSIONS) \ + $(call qstrip,$$($(2)_OVERRIDE_SRCDIR))/ \ + $$($(2)_BUILDDIR)/.legal-info-rsync/ + $$(Q). support/download/helpers; set -x; cd $$($(2)_BUILDDIR); TAR=$$(TAR) mk_tar_gz \ + $$($(2)_BUILDDIR)/.legal-info-rsync/ \ + $$($(2)_BASENAME_RAW) \ + @0 \ + $$($(2)_REDIST_SOURCES_DIR)/$$($(2)_BASENAME_RAW).tar.gz + $$(Q)rm -rf $$($(2)_BUILDDIR)/.legal-info-rsync else # Other packages