show-new: limit to five threads to avoid using too many resources

Signed-off-by: Luca Falavigna <dktrkranz@debian.org>

Do not fail on missing DB connection

Signed-off-by: Luca Falavigna <dktrkranz@debian.org>

Create session when there is need to

Signed-off-by: Luca Falavigna <dktrkranz@debian.org>

Merge remote branch 'drkranz/show-new' into merge

* drkranz/show-new:
  show-new: implement a timeout to prevent it hanging (Closes: #629271)

Signed-off-by: Joerg Jaspert <joerg@debian.org>

show-new: implement a timeout to prevent it hanging (Closes: #629271)

Signed-off-by: Luca Falavigna <dktrkranz@debian.org>

Merge remote branch 'tolimar2/lintian-2.5.0' into merge

* tolimar2/lintian-2.5.0:
  Add missing-pre-dependency-on-multiarch-support to the lintian autorjects as requested by Steve Langasek
  Adding install-info-used-in-maintainer-script to the non-fatal lintian-reject-tags; closes #607293
  Some lintian tags got renamed with 2.5.0
  Fix typo in templated reported by Mohammed Adnène Trojette

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote-tracking branch 'master/master' into signatures-2

gpg.py: set O_NONBLOCK for writing

We need O_NONBLOCK as otherwise os.write() would block for large input.

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

Add missing-pre-dependency-on-multiarch-support to the lintian autorjects
as requested by Steve Langasek

Adding install-info-used-in-maintainer-script to the non-fatal
lintian-reject-tags; closes #607293

Some lintian tags got renamed with 2.5.0

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

a newline is ONE char, so allow one char long lines

Signed-off-by: Joerg Jaspert <joerg@debian.org>

allow empty lines but nothing else

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'ansgar/signatures-2' into merge

* ansgar/signatures-2:
  do not check for a valid signature in parse_deb822, parse_changes

Signed-off-by: Joerg Jaspert <joerg@debian.org>

do not check for a valid signature in parse_deb822, parse_changes

Not all files need to have a valid signature: old .dsc's can be signed
by a key no longer in the keyrings.

Merge remote branch 'ansgar/signatures-2' into merge

* ansgar/signatures-2:
  daklib/gpg.py: use os._exit() instead of sys.exit() in child process
  daklib/utils.py (parse_deb822, parse_changes): use session_wrapper

Signed-off-by: Joerg Jaspert <joerg@debian.org>

daklib/gpg.py: use os._exit() instead of sys.exit() in child process

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

daklib/utils.py (parse_deb822, parse_changes): use session_wrapper

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

Merge remote branch 'ansgar/signatures-2' into merge

* ansgar/signatures-2:
  tests/test_parse_changes.py: add another testcase
  daklib/gpg.py: capture stderr
  tests/test_parse_changes.py: do not require signatures
  daklib/utils.py (parse_deb822): include PGP armor in filecontents
  tests/test_parse_changes.py: update for new SignedFile exceptions
  daklib/utils.py (parse_deb822): use SignedFile to handle PGP signature
  tests: dsc/4.dsc is invalid
  examine-package: use SignedFile to remove PGP armor
  daklib/gpg.py: small library for PGP-signed files

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'youam/master' into merge

* youam/master:
  debianqueued: check for unsigned .changes content

Signed-off-by: Joerg Jaspert <joerg@debian.org>

debianqueued: check for unsigned .changes content

If signed .changes files contain clear text contents outside of the
signed part, remove the .changes file.

tests/test_parse_changes.py: add another testcase

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

Merge remote branch 'drkranz/examine-package' into merge

* drkranz/examine-package:
  Also check for codename when looking for Provides

Signed-off-by: Joerg Jaspert <joerg@debian.org>

daklib/gpg.py: capture stderr

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

tests/test_parse_changes.py: do not require signatures

We cannot require a valid signature in the tests as we don't have a
keyring here.

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

Also check for codename when looking for Provides

Signed-off-by: Luca Falavigna <dktrkranz@debian.org>

Merge remote branch 'drkranz/examine-package' into merge

* drkranz/examine-package:
  examine-package: display known virtual packages

Signed-off-by: Joerg Jaspert <joerg@debian.org>

daklib/utils.py (parse_deb822): include PGP armor in filecontents

We need to include the PGP armor in filecontents as this is later passed
to us again and we now always require a signature (even if it does not
need to be a valid one).

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

tests/test_parse_changes.py: update for new SignedFile exceptions

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

daklib/utils.py (parse_deb822): use SignedFile to handle PGP signature

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

tests: dsc/4.dsc is invalid

The test for dsc/4.dsc should fail as it is not a valid armored OpenPGP
container.

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

examine-package: use SignedFile to remove PGP armor

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

daklib/gpg.py: small library for PGP-signed files

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

examine-package: display known virtual packages

Dedicated to our beloved Haskell and OCaml folks,
now we can process their packages quicker.
It's their turn to find a more complicated way
to define package dependencies, now :)

Signed-off-by: Luca Falavigna <dktrkranz@debian.org>

Merge remote branch 'drkranz/master' into merge

* drkranz/master:
  Fix show-new to avoid generating zombie processes

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Fix show-new to avoid generating zombie processes

Signed-off-by: Luca Falavigna <dktrkranz@debian.org>

more useful logging of added overrides

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'ansgar/daklog' into merge

* ansgar/daklog:
  restore program name to log messages

Signed-off-by: Joerg Jaspert <joerg@debian.org>

restore program name to log messages

also use -9 for bzip2

Signed-off-by: Joerg Jaspert <joerg@debian.org>

don't include filename or timestamp in gzipped files

(optional: also compress at level 9)

Omitting the timestamp makes it possible to reconstruct an exact copy of the .gz
file given only the uncompressed contents.  The approx caching proxy relies on this
to apply pdiffs and generate compressed index files that correctly match their checksums.

Signed-off-by: Eric Cooper <ecc@cmu.edu>

Merge remote branch 'ansgar/p-s-from-db' into merge

* ansgar/p-s-from-db:
  Add missing import statements

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Add missing import statements

also import Suite

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'ansgar/p-s-from-db' into merge

* ansgar/p-s-from-db:
  gps2: Initialize process pool before connecting to database

Signed-off-by: Joerg Jaspert <joerg@debian.org>

gps2: Initialize process pool before connecting to database

import_dataset

we also dont need gps called in here

Signed-off-by: Joerg Jaspert <joerg@debian.org>

kill do_dists

no need to write packages files in unchecked anymore. gps2 doesnt have
braindead berkeley db shit to update

Signed-off-by: Joerg Jaspert <joerg@debian.org>

s/base/basen/

Signed-off-by: Joerg Jaspert <joerg@debian.org>

s/foo/base

Signed-off-by: Joerg Jaspert <joerg@debian.org>

pdiff shit

add a function to fill up a git tree for the possible replacement of the shit script thats currently creating pdiff files

Signed-off-by: Joerg Jaspert <joerg@debian.org>

check mirror

we check the mirror before we push anything. We currently parse
the SHA1 field, checking for existence, size and checksum.
this means we should catch the HASH SUM MISMATCH crap.

of course this means the ones using ftp-master.d.o directly still lose, but we cant help
that.

Signed-off-by: Joerg Jaspert <joerg@debian.org>
Signed-off-by: Joerg Jaspert <joerg@debian.org>

import a bit from from processpool

Signed-off-by: Joerg Jaspert <joerg@debian.org>

no longer run a-f cleanup in weekly

Signed-off-by: Joerg Jaspert <joerg@debian.org>

dont output tag lines from metadata table, only external overrides

Signed-off-by: Joerg Jaspert <joerg@debian.org>

use gps2 in unchecked. also disable gen-filelists in dinstall

Signed-off-by: Joerg Jaspert <joerg@debian.org>

also do it for imports

Signed-off-by: Joerg Jaspert <joerg@debian.org>

switch of a-f

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'mhy/multiproc' into merge

* mhy/multiproc:
  Get syntax right
  Deal with args being a tuple
  Add backwards compatibility code
  Make daklog be a Singleton
  Bugfixes for usage of DakProcessPool API
  remove unused variable
  Fix variable name
  Fix function name
  Fix up pool usage
  Enhance process pool implementation

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'ftpmaster/master' into multiproc

Conflicts:
dak/generate_packages_sources2.py

Signed-off-by: Mark Hymers <mhy@debian.org>

kick out useless print stuff that only annoys in logs

Signed-off-by: Joerg Jaspert <joerg@debian.org>

Merge remote branch 'ansgar/p-s-from-db' into merge

* ansgar/p-s-from-db:
  do not generate binary-source/Packages
  look for already existing files in build queues
  fix typo in email address
  sort Packages by source name first

Signed-off-by: Joerg Jaspert <joerg@debian.org>

do not generate binary-source/Packages

look for already existing files in build queues

fix typo in email address

sort Packages by source name first

Merge remote branch 'ftpmaster/master' into multiproc

cron.unchecked: call generate-filelist with arguments.

Signed-off-by: Torsten Werner <twerner@debian.org>

Fix exception handling in process-new.

closes: #620361

Signed-off-by: Torsten Werner <twerner@debian.org>

Fall back to Maintainer if Changed-By missing

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge remote branch 'ftpmaster/master' into multiproc

Conflicts:
dak/show_new.py

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge remote branch 'ansgar/p-s-from-db' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge remote-tracking branch 'origin/master' into p-s-from-db

Fix function add_dsc_to_db().

Signed-off-by: Torsten Werner <twerner@debian.org>

Update metadata for test DB.

Signed-off-by: Torsten Werner <twerner@debian.org>

Do not sync any lock files.

Signed-off-by: Torsten Werner <twerner@debian.org>

Install default SIGPIPE handler before calling dpkg-deb.

Signed-off-by: Torsten Werner <twerner@debian.org>

Don't remove files we previously generated

Signed-off-by: Mark Hymers <mhy@debian.org>

Short circuit already processed packages much more quickly

Signed-off-by: Mark Hymers <mhy@debian.org>

Pass session

Signed-off-by: Mark Hymers <mhy@debian.org>

Don't end up with multiple database sessions in check_sig call

Signed-off-by: Mark Hymers <mhy@debian.org>

Linearise show_new for now

Signed-off-by: Mark Hymers <mhy@debian.org>

Map squeeze-updates and stable-updates

Signed-off-by: Mark Hymers <mhy@debian.org>

Limit show-new to one process for now

Signed-off-by: Mark Hymers <mhy@debian.org>

Re-enable queue_rss after deleting the old pickle file

Signed-off-by: Mark Hymers <mhy@debian.org>

use new debian module name

Signed-off-by: Mark Hymers <mhy@debian.org>

Python behaves differently if you're redirecting to a file or not...

Signed-off-by: Mark Hymers <mhy@debian.org>

Encode using utf-8 and hope for the best

Signed-off-by: Mark Hymers <mhy@debian.org>

dak graph seems ok too

Signed-off-by: Mark Hymers <mhy@debian.org>

Re-enable show-new, but don't kill cron.hourly if it dies

Signed-off-by: Mark Hymers <mhy@debian.org>

Fix typo

Signed-off-by: Mark Hymers <mhy@debian.org>

temporary hack to fix squeeze-updates / stable-updates suite name

Signed-off-by: Mark Hymers <mhy@debian.org>

Comment out some things in cron.hourly which are broken

Signed-off-by: Mark Hymers <mhy@debian.org>

Merge remote branch 'ftpmaster/master'

Make cron.daily shell script more robust.

... and reindent it.

Signed-off-by: Torsten Werner <twerner@debian.org>

Excluding more directories from syncing.

Signed-off-by: Torsten Werner <twerner@debian.org>

copy external overrides from unstable to testing

Signed-off-by: Ansgar Burchardt <ansgar@debian.org>

Merge remote-tracking branch 'origin/master' into p-s-from-db

sick

Signed-off-by: Joerg Jaspert <joerg@debian.org>