Merge remote-tracking branch 'ansgar/package-set' into merge

* ansgar/package-set:
  process-new: allow overrides to work for Package-Set
  Make use of Package-Set in determine_new
  queue.Upload: separate code to load .dsc from check_dsc
  utils.py: Add parser for Package-Set field in .dsc
  process-upload: Dinstall::LintianTags may not be set

Signed-off-by: Joerg Jaspert <joerg@debian.org>

process-new: allow overrides to work for Package-Set

New packages from Package-Set do not always have files assocciated with
them. In that case edited overrides would be lost once we call
override_new again. We pass the old new dictionary so we can preserve
this information.

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

Merge branch 'contents' into pkgsrc

Merge branch 'pkgsrc' of ftp-master.debian.org:public_html/dak into pkgsrc

Make use of Package-Set in determine_new

Make use of the Package-Set field of an included .dsc file when looking
for NEW packages.  See http://bugs.debian.org/619131 for more
information.

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

Merge branch 'master' into contents

Conflicts:
daklib/dbconn.py

queue.Upload: separate code to load .dsc from check_dsc

Move the code to load the .dsc file into the self.pkg.dsc dictionary
into a separate function so it can be called without the side effects in
check_dsc.

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

utils.py: Add parser for Package-Set field in .dsc

See http://bugs.debian.org/619131 for more information.

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

process-upload: Dinstall::LintianTags may not be set

Handle the case when Dinstall::LintianTags is not set correctly.

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

Refactor functions getSource() and getBinaries().

Both functions are moved from dak/generate_filelist.py to
daklib/lists.py. The generator now yields a tuple with the id as the
first element.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' of ftp-master.debian.org:/srv/ftp.debian.org/git/dak

ORMObject: make validation more robust.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'merge'

* merge:
  Fix dak update-db

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak:
  Proper fix for validate problem
  validate is somehow causing a double flush; comment for now

Merge remote-tracking branch 'ansgar/fix-update-db' into merge

* ansgar/fix-update-db:
  Fix dak update-db

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Proper fix for validate problem

Signed-off-by: Mark Hymers <mhy@debian.org>

validate is somehow causing a double flush; comment for now

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak:
  Fix up permissions
  metadata generation work
  Use UnpackedSource in queue.py.
  Improve exception handling in make-changelog.
  Work with service mode
  Use UnpackedSource in make-changelog.
  Add new class UnpackedSource and a new test.
  Start work on Packages import

Optimize file rename in dak generate.

Signed-off-by: Torsten Werner <twerner@debian.org>

Handle exceptions in dak contents scan-source.

Signed-off-by: Torsten Werner <twerner@debian.org>

Rename contents subcommands to be consistent with metadata.

Signed-off-by: Torsten Werner <twerner@debian.org>

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

Fix up permission

Signed-off-by: Mark Hymers <mhy@debian.org>

Fix up permissions

Signed-off-by: Mark Hymers <mhy@debian.org>

Add subcommand source-scan to 'dak contents'.

Signed-off-by: Torsten Werner <twerner@debian.org>

Add class SourceContentsScanner.

Signed-off-by: Torsten Werner <twerner@debian.org>

move to service usage

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Fix dak update-db

Merge remote branch 'mhy/psimport' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

metadata generation work

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge remote branch 'ftpmaster/master' into psimport

Conflicts:
daklib/dbconn.py

Signed-off-by: Mark Hymers <mhy@debian.org>

Rename ContentsScanner to BinaryContentsScanner.

Signed-off-by: Torsten Werner <twerner@debian.org>

Use UnpackedSource in queue.py.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'contents'

Improve exception handling in make-changelog.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

Work with service mode

Signed-off-by: Mark Hymers <mhy@debian.org>

allow null versions

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Use UnpackedSource in make-changelog.

Signed-off-by: Torsten Werner <twerner@debian.org>

Add new class UnpackedSource and a new test.

Signed-off-by: Torsten Werner <twerner@debian.org>

use  service name

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'pkgsrc'

Metadata: add cascading to mapper configuration.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak:
  Configure and test more mapper relationships.

no longer pgdatabase

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Configure and test more mapper relationships.

- DBBinary and DBSource have relationships to BinaryMetadata and SourceMetadata.
- Both have an associationproxy to hide the details.

Signed-off-by: Torsten Werner <twerner@debian.org>

Start work on Packages import

Signed-off-by: Mark Hymers <mhy@debian.org>

Add Metadata classes + tests.

+ some cleanup of dbconn.py

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'contents' of ftp-master.debian.org:public_html/dak into contents

Replace DBConn.reset() by something more robust.

Signed-off-by: Torsten Werner <twerner@debian.org>

Fix database tests.

Signed-off-by: Torsten Werner <twerner@debian.org>

gps

move stable/oldstable into gps too, so we only generate packages/sources
from there, ever

Signed-off-by: Joerg Jaspert <joerg@debian.org>

license

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Contents: replace multithreading by multiprocessing.

That avoids the global interpreter lock of CPython.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'contents'

Add more logging to contents generate.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

Add Built-Using support

Signed-off-by: Mark Hymers <mhy@debian.org>

Contents: ignore errors when removing destination file.

Signed-off-by: Torsten Werner <twerner@debian.org>

Do no longer re-exec dak contents.

We just call DBConn().reset() in the child process now.

Signed-off-by: Torsten Werner <twerner@debian.org>

ries sync

no longer push or name anything merkel
sync ries differently depending on if its called with sync or pool
that way cron.hourly does a small sync while cron.dinstall a full one triggers

Signed-off-by: Joerg Jaspert <joerg@debian.org>

shutup

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Contents: use the new string_agg() function of Pg9.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak:
  Fix typo
  Improve db conf handling
  Commit uncommitted fix
  Patch in support for service= DSN syntax

add the syncscript for ries. also call it hourly

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Don't remove sources which are referenced by other binaries

Signed-off-by: Mark Hymers <mhy@debian.org>

Fix typo

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

Improve db conf handling

Signed-off-by: Mark Hymers <mhy@debian.org>

Commit uncommitted fix

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

Patch in support for service= DSN syntax

Signed-off-by: Mark Hymers <mhy@debian.org>

add ries

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

Use PK at twerners suggestion

Signed-off-by: Mark Hymers <mhy@debian.org>

set db schema version

Signed-off-by: Mark Hymers <mhy@debian.org>

Add method reset() to class DBConn() and test it.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge remote branch 'ftpmaster/master'

First pass at P/S database schema

Signed-off-by: Mark Hymers <mhy@debian.org>

First pass at extra_src_references table

Signed-off-by: Mark Hymers <mhy@debian.org>

Add table src_contents.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' of ftp-master.debian.org:/srv/ftp.debian.org/git/dak

Add logging to update-db including transaction id.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

Really update to DB version #44.

Signed-off-by: Torsten Werner <twerner@debian.org>

Merge branch 'master' into dbtests

Conflicts:
dak/update_db.py

Signed-off-by: Torsten Werner <twerner@debian.org>

Rename DB upgrade #43 to #44.

bad, bad mhy...

Signed-off-by: Torsten Werner <twerner@debian.org>

make check_overrides not delete stuff thats not at least 14 days old

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Don't contradict ourselves within 2 lines

Signed-off-by: Mark Hymers <mhy@debian.org>

add mvo quote

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak:
  Prepare for pg 9.0

gps

change the way we run the pool once more, hopefully being less bad now. also, this tool is EOL somehwat, so who cares how bad it looks

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Prepare for pg 9.0

Signed-off-by: Mark Hymers <mhy@debian.org>

gps timeout/error handling

return the result of the a-f call.
additionally get it using a timeout.
when either a TimeoutError occurs or the result is non-zero, complain about it
and then exit non-zero, so dinstall could stop.

Signed-off-by: Joerg Jaspert <joerg@debian.org>

mhy, das kruemelmonster

Signed-off-by: Joerg Jaspert <joerg@debian.org>

database backup foo

no longer dump the full database on each dinstall run. instead we just
save a transaction id. DSA is doing the backup dumps in whatever cycle,
and keeping the WAL stuff around, so we can tell them to get us to
whatever point in our db which we might need, ie. the specific transaction
id.

as those are just 9 bytes per file, and not 4gigabytes, we now also save
wth we have been before and after unchecked runs, which makes our backup
be nearer to actual filesystem state than ever.

Signed-off-by: Joerg Jaspert <joerg@debian.org>
Signed-off-by: Joerg Jaspert <joerg@debian.org>

remove pgdakdev

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Remove old contents code and tables.

Signed-off-by: Torsten Werner <twerner@debian.org>

parse scriptvars, not . vars

Signed-off-by: Joerg Jaspert <joerg@debian.org>