Add a per-suite boolean to enable by-hash; store the by-hash files in
the db, and record when they stopped being referenced, so that
clean-suites can delete them after the archive's stayofexecution time.
In generate-release, where we have checksums for all the things,
hardlink files to the by-hash dir for each of the suite's configured
hash methods.
Signed-off-by: Julien Cristau <jcristau@debian.org>
---
changes in v2:
- use archive.stayofexecution as delay before removing files from
by-hash
- don't assume any particular ordering for suite.checksums
changes in v3:
- rebase on latest master, update115 is now update116
- handle missing files in clean_byhash
changes in v4:
- use hardlinks instead of symlinks
- don't initialize `unreferenced` to its default value
- in clean_byhash, remove useless fetchall, and handle ENOENT from
unlink instead of checking for existence beforehand
Luke Faraone [Sat, 30 Apr 2016 21:59:52 +0000 (21:59 +0000)]
Add python-debianbts as a dependency in setup/README (Closes: #823118)
Needed to run the test suite:
======================================================================
ERROR: test_importing_rm (test_imports.ImportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/daksrc/tests/test_imports.py", line 15, in test_fn
__import__('dak', fromlist=[cmd])
File "/srv/daksrc/dak/rm.py", line 54, in <module>
from daklib.rm import remove
File "/srv/daksrc/dak/daklib/rm.py", line 52, in <module>
import debianbts as bts
ImportError: No module named debianbts
Before Debian LTS started to use the security archive, all uploads that
would trigger buildds would be sourceful uploads to a policy queue. So
new packages entering the build queues and pushing the public mirrors
would likely not happen in the same cron.unchecked run.
With Debian LTS, uploads would go to both the build queues and trigger a
public mirror push. If the mirror push takes more than three minutes,
cron.buildd would however not notify w-b of changes.
This patch moves the w-b notification right behind the update of the
build queues and before the mirror push to avoid the problem. Ideally
we would not rely on anything happening within the last three minutes
though.
Let the python interpreter handle it and actually exit with the right
status, otherwise when a sub-command calls sys.exit(1) we end up exiting
0 instead, which is confusing if not dangerous.
With "wheezy" being used for Debian LTS, the archive needs to be
published not only when policy queue uploads are accepted, but also
when direct uploads to "wheezy" arrive.
Newer SQLAlchemy versions seem to track the `binary` attribute of
`BinaryMetadata`. This means the association proxy in `DBBinary`
would first create a `BinaryMetadata` with `binary` set to `None`,
flush this to the database and then set the actual value of `binary`
we want. However the flush fails, as `NULL` is not allowed for the
`binary_id` column.
The problem can be avoided by not setting `binary` to `None`. This is
treated as an "undefined value" by SQLAlchemy and changing it later
means the database row never sees `NULL`.
The same applies to the `source` attribute of `SourceMetadata`.
Move more specific source file regexes before general ones
The orig tarball regexes match also the orig tarball detached
signatures because they are not anchored at the end, and the loop
short-circuits on first match. So we have to move them before their
more general counterparts.
Guillem Jover [Thu, 11 Feb 2016 09:27:16 +0000 (10:27 +0100)]
Fix support for detached upstream signatures
We should only allow upstream tarballs with detached signatures, but not
native tarballs, «<source>_<version>.orig.asc» or «<source>_<version>.asc»
filenames. We also need to allow this on all relevant regexes. This commit
fixes and refactors the regexes to implement the above, and to make it
easier to modify a single place in the future.
Guillem Jover [Thu, 11 Feb 2016 09:27:13 +0000 (10:27 +0100)]
Stop using undefined reject function in check_dsc_files
When there's an unknown file in a source package, the function spits
this:
,---
Processing raised an exception: global name 'reject' is not defined.
Traceback (most recent call last):
File "/srv/ftp-master.debian.org/dak/dak/daklib/archive.py", line 966, in check
chk().check(self)
File "/srv/ftp-master.debian.org/dak/dak/daklib/checks.py", line 497, in check
rejects = utils.check_dsc_files(dsc_fn, control, source.files.keys())
File "/srv/ftp-master.debian.org/dak/dak/daklib/utils.py", line 343, in check_dsc_files
reject("%s: unexpected source file '%s'" % (dsc_filename, f))
NameError: global name 'reject' is not defined
`---
Just set the reject message and break from the loop.
Ansgar Burchardt [Wed, 30 Mar 2016 21:23:13 +0000 (23:23 +0200)]
Remove files that are (no longer) generated
This makes it less error-prone to switch compression methods: the files
that are no longer generated are automatically removed. Before they had
to be removed manually.
Ansgar Burchardt [Sat, 23 Jan 2016 11:30:30 +0000 (12:30 +0100)]
Add new "mirror" cronscript
This just prepares the mirror tree and pushes the changes to the
mirrors. It is intended to be used for point releases or in other
situations where a manual mirror push is needed.
Guillem Jover [Sun, 17 Jan 2016 01:08:12 +0000 (02:08 +0100)]
Add support for detached upstream signatures
Upstream tarballs usually come with detached signatures, which would be
useful to have in the source package, as an additional check that could
be performed to verify its integrity and provenance.
stable-proposed-updates is called "proposed-updates" in projectb, so
that's what we pass to the byhand script. It shouldn't be turned into
"proposed-updates-proposed-updates".
Ansgar Burchardt [Sat, 16 Jan 2016 14:23:39 +0000 (15:23 +0100)]
Add option to forbid source and/or binary uploads to a suite
This is intended to be used to forbid uploads to policy queues and build
queues which would result in confusing results. While we already have a
"reject" option in the dak.conf's SuiteMappings, having the option in
the database means we don't have to edit dak.conf when configuring new
suites.
Another usecase will be the "unstable" suite for Debian Ports: here we
want to import the source packages (and arch:all) from the main archive
and only accept binary uploads. Additional source packages must go into
the "unreleased-*" suites. This was not possible via SuiteMappings.
Ansgar Burchardt [Sat, 16 Jan 2016 14:07:47 +0000 (15:07 +0100)]
Check for debug suite for upload suite, not override suite
Debug packages are exempt from NEW if the target suite has a seperate
debug suite. However the code checked if the target's override suite
had a debug suite... For the Debian archive this went unnoticed as
experimental's override suite is unstable and both suites have a
seperate debug suite.
projects / dak.git / log