git.decadent.org.uk Git

Contents: replace multithreading by multiprocessing.

That avoids the global interpreter lock of CPython.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 22 Mar 2011 15:50:39 +0000 (16:50 +0100)]

Merge branch 'contents'

commit | commitdiff | tree

Torsten Werner [Tue, 22 Mar 2011 15:42:58 +0000 (15:42 +0000)]

Add more logging to contents generate.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Mark Hymers [Tue, 22 Mar 2011 15:07:29 +0000 (15:07 +0000)]

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Tue, 22 Mar 2011 15:06:04 +0000 (15:06 +0000)]

Add Built-Using support

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 22 Mar 2011 14:54:11 +0000 (14:54 +0000)]

Contents: ignore errors when removing destination file.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 22 Mar 2011 14:47:09 +0000 (14:47 +0000)]

Do no longer re-exec dak contents.

We just call DBConn().reset() in the child process now.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Tue, 22 Mar 2011 14:31:15 +0000 (15:31 +0100)]

ries sync

no longer push or name anything merkel
sync ries differently depending on if its called with sync or pool
that way cron.hourly does a small sync while cron.dinstall a full one triggers

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Tue, 22 Mar 2011 13:50:12 +0000 (14:50 +0100)]

shutup

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 22 Mar 2011 13:18:55 +0000 (13:18 +0000)]

Contents: use the new string_agg() function of Pg9.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Tue, 22 Mar 2011 13:09:55 +0000 (14:09 +0100)]

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak:
  Fix typo
  Improve db conf handling
  Commit uncommitted fix
  Patch in support for service= DSN syntax

commit | commitdiff | tree

Joerg Jaspert [Tue, 22 Mar 2011 13:09:49 +0000 (14:09 +0100)]

add the syncscript for ries. also call it hourly

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Mark Hymers [Tue, 22 Mar 2011 10:56:55 +0000 (10:56 +0000)]

Don't remove sources which are referenced by other binaries

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Tue, 22 Mar 2011 10:42:39 +0000 (10:42 +0000)]

Fix typo

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Tue, 22 Mar 2011 10:41:51 +0000 (10:41 +0000)]

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Tue, 22 Mar 2011 10:41:17 +0000 (10:41 +0000)]

Improve db conf handling

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Tue, 22 Mar 2011 10:21:58 +0000 (10:21 +0000)]

Commit uncommitted fix

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Tue, 22 Mar 2011 10:20:18 +0000 (10:20 +0000)]

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Tue, 22 Mar 2011 10:18:57 +0000 (10:18 +0000)]

Patch in support for service= DSN syntax

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Tue, 22 Mar 2011 10:14:10 +0000 (11:14 +0100)]

add ries

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 21:45:58 +0000 (21:45 +0000)]

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 21:45:43 +0000 (21:45 +0000)]

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 21:38:29 +0000 (21:38 +0000)]

Use PK at twerners suggestion

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 21:35:47 +0000 (21:35 +0000)]

set db schema version

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 21 Mar 2011 21:26:21 +0000 (22:26 +0100)]

Add method reset() to class DBConn() and test it.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 20:55:49 +0000 (20:55 +0000)]

Merge remote branch 'ftpmaster/master'

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 20:26:30 +0000 (20:26 +0000)]

First pass at P/S database schema

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 20:18:26 +0000 (20:18 +0000)]

First pass at extra_src_references table

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 21 Mar 2011 20:07:33 +0000 (21:07 +0100)]

Add table src_contents.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 21 Mar 2011 19:44:39 +0000 (20:44 +0100)]

Merge branch 'master' of ftp-master.debian.org:/srv/ftp.debian.org/git/dak

commit | commitdiff | tree

Torsten Werner [Mon, 21 Mar 2011 19:43:29 +0000 (20:43 +0100)]

Add logging to update-db including transaction id.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 19:23:58 +0000 (19:23 +0000)]

Merge remote branch 'mhy/master' into merge

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 21 Mar 2011 19:18:53 +0000 (20:18 +0100)]

Really update to DB version #44.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 21 Mar 2011 17:21:16 +0000 (18:21 +0100)]

Merge branch 'master' into dbtests

Conflicts:
dak/update_db.py

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 21 Mar 2011 17:14:26 +0000 (18:14 +0100)]

Rename DB upgrade #43 to #44.

bad, bad mhy...

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 21 Mar 2011 15:56:26 +0000 (16:56 +0100)]

make check_overrides not delete stuff thats not at least 14 days old

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 15:32:10 +0000 (15:32 +0000)]

Don't contradict ourselves within 2 lines

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 15:06:30 +0000 (15:06 +0000)]

add mvo quote

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 21 Mar 2011 13:17:57 +0000 (14:17 +0100)]

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak:
Prepare for pg 9.0

commit | commitdiff | tree

Joerg Jaspert [Mon, 21 Mar 2011 13:17:45 +0000 (14:17 +0100)]

gps

change the way we run the pool once more, hopefully being less bad now. also, this tool is EOL somehwat, so who cares how bad it looks

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Mark Hymers [Mon, 21 Mar 2011 11:44:59 +0000 (11:44 +0000)]

Prepare for pg 9.0

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 21 Mar 2011 11:32:16 +0000 (12:32 +0100)]

gps timeout/error handling

return the result of the a-f call.
additionally get it using a timeout.
when either a TimeoutError occurs or the result is non-zero, complain about it
and then exit non-zero, so dinstall could stop.

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 21 Mar 2011 11:28:45 +0000 (12:28 +0100)]

mhy, das kruemelmonster

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 21 Mar 2011 09:39:05 +0000 (10:39 +0100)]

database backup foo

no longer dump the full database on each dinstall run. instead we just
save a transaction id. DSA is doing the backup dumps in whatever cycle,
and keeping the WAL stuff around, so we can tell them to get us to
whatever point in our db which we might need, ie. the specific transaction
id.

as those are just 9 bytes per file, and not 4gigabytes, we now also save
wth we have been before and after unchecked runs, which makes our backup
be nearer to actual filesystem state than ever.

Signed-off-by: Joerg Jaspert <joerg@debian.org>
Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 21 Mar 2011 08:43:58 +0000 (09:43 +0100)]

remove pgdakdev

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 21 Mar 2011 08:24:12 +0000 (09:24 +0100)]

Remove old contents code and tables.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Sun, 20 Mar 2011 23:46:00 +0000 (00:46 +0100)]

parse scriptvars, not . vars

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Torsten Werner [Sun, 20 Mar 2011 23:01:40 +0000 (00:01 +0100)]

Fix a comment in dbconn.py.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sun, 20 Mar 2011 22:30:46 +0000 (23:30 +0100)]

Fix copy-paste error.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sun, 20 Mar 2011 22:03:30 +0000 (23:03 +0100)]

Merge branch 'master' of ftp-master.debian.org:/srv/ftp.debian.org/git/dak

commit | commitdiff | tree

Torsten Werner [Sun, 20 Mar 2011 22:02:34 +0000 (23:02 +0100)]

Ignore more warnings from python-apt.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Sat, 19 Mar 2011 14:05:06 +0000 (15:05 +0100)]

Merge branch 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak:
Add filename de-duplication to contents scanning.

commit | commitdiff | tree

Joerg Jaspert [Sat, 19 Mar 2011 14:04:59 +0000 (15:04 +0100)]

updates

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 15 Mar 2011 18:14:47 +0000 (19:14 +0100)]

Merge branch 'dbtests'

commit | commitdiff | tree

Torsten Werner [Tue, 15 Mar 2011 17:58:39 +0000 (18:58 +0100)]

Add filename de-duplication to contents scanning.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Julien Cristau [Mon, 14 Mar 2011 21:48:57 +0000 (22:48 +0100)]

byhand-di: allow YYYYMMDD+<suite>x version numbers

To: debian-dak@lists.debian.org
Cc: Julien Cristau <jcristau@debian.org>, Otavio Salvador <otavio@ossystems.com.br>
Date: Mon, 14 Mar 2011 14:11:39 +0100
X-Spam-Status: No, score=-1.775 tagged_above=-99 required=4.6 tests=[BAYES_00=-1.9, SARE_SUB_OBFU_OTHER=0.135, T_RP_MATCHES_RCVD=-0.01] autolearn=no
X-Mailer: git-send-email 1.7.2.5
Resent-From: debian-dak@lists.debian.org

stable updates need to have a higher version than binNMUs, but
YYYYMMDD<suite>x << YYYYMMDD+b1.

Signed-off-by: Julien Cristau <jcristau@debian.org>
Cc: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 14 Mar 2011 21:07:40 +0000 (22:07 +0100)]

Break hardlink when writing Contents files.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Sat, 12 Mar 2011 14:05:44 +0000 (15:05 +0100)]

we do want -n1, not -n16. we want as many parallel xz as we can

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Sat, 12 Mar 2011 09:51:42 +0000 (10:51 +0100)]

dinstall - mirror

exclude the temporary files from apt-ftparchive (*.new), and delete excluded
files too.
Should keep the mirrors happy, cos those files arent readable for others.

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Torsten Werner [Fri, 11 Mar 2011 22:13:28 +0000 (23:13 +0100)]

Handle empty packages in 'dak contents scan'.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Mark Hymers [Fri, 11 Mar 2011 20:27:40 +0000 (20:27 +0000)]

allow squeeze-updates to be reset

Signed-off-by: Mark Hymers <mhy@debian.org>

commit | commitdiff | tree

Torsten Werner [Fri, 11 Mar 2011 19:30:22 +0000 (20:30 +0100)]

Switch to 'dak contents generate'.

It replaces APT::FTPArchive::Contents.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Thu, 10 Mar 2011 22:07:45 +0000 (23:07 +0100)]

Enable the new contents generation for experimental.

... as a first test

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Thu, 10 Mar 2011 21:52:57 +0000 (22:52 +0100)]

Write full section to Contents files.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Wed, 9 Mar 2011 08:15:00 +0000 (09:15 +0100)]

Merge branch 'master' into dbtests

commit | commitdiff | tree

Torsten Werner [Wed, 9 Mar 2011 08:14:05 +0000 (09:14 +0100)]

Add workaround for duplicate filenames in contents.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Tue, 8 Mar 2011 21:57:15 +0000 (22:57 +0100)]

Merge branch 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak:
bugfixes
Contents: support override suites.

commit | commitdiff | tree

Joerg Jaspert [Tue, 8 Mar 2011 21:57:08 +0000 (22:57 +0100)]

dinstall - use xz for pg dumps, and use a | xargs instead of while

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 8 Mar 2011 21:47:19 +0000 (22:47 +0100)]

Merge branch 'master' of ssh://ftp-master.debian.org/srv/ftp.debian.org/git/dak

commit | commitdiff | tree

Torsten Werner [Tue, 8 Mar 2011 21:44:26 +0000 (21:44 +0000)]

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Tue, 8 Mar 2011 21:35:50 +0000 (22:35 +0100)]

Merge branch 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak

* 'master' of ssh://franck.debian.org/srv/ftp.debian.org/git/dak:
  Use 'dak contents generate_helper'.
  bugfixes
  Add a new subcommand 'generate_helper' to 'dak contents'.
  Switch to multiprocessing module and reap child gzips.

commit | commitdiff | tree

Joerg Jaspert [Tue, 8 Mar 2011 21:35:32 +0000 (22:35 +0100)]

dinstall - dont recompress .xz files

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 8 Mar 2011 21:29:43 +0000 (22:29 +0100)]

Contents: support override suites.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 8 Mar 2011 20:57:17 +0000 (21:57 +0100)]

Merge branch 'master' into dbtests

commit | commitdiff | tree

Torsten Werner [Tue, 8 Mar 2011 20:55:38 +0000 (20:55 +0000)]

Use 'dak contents generate_helper'.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 8 Mar 2011 20:13:06 +0000 (20:13 +0000)]

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 8 Mar 2011 19:59:26 +0000 (20:59 +0100)]

Add a new subcommand 'generate_helper' to 'dak contents'.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Tue, 8 Mar 2011 18:39:59 +0000 (19:39 +0100)]

Switch to multiprocessing module and reap child gzips.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 7 Mar 2011 22:31:03 +0000 (23:31 +0100)]

And make the whitespace something useful

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 7 Mar 2011 22:30:33 +0000 (23:30 +0100)]

Fixup the regex

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 7 Mar 2011 21:47:49 +0000 (22:47 +0100)]

goodbye comments

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Joerg Jaspert [Mon, 7 Mar 2011 21:47:10 +0000 (22:47 +0100)]

during unchecked no longer recreate testing indices, only do them when importing new data there

Signed-off-by: Joerg Jaspert <joerg@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 7 Mar 2011 20:47:02 +0000 (21:47 +0100)]

Scan 10.000 files per unchecked run.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Mon, 7 Mar 2011 20:46:00 +0000 (21:46 +0100)]

Include all files (except directories) in bin_contents table.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sun, 6 Mar 2011 22:31:46 +0000 (23:31 +0100)]

Merge branch 'dbtests'

commit | commitdiff | tree

Torsten Werner [Sun, 6 Mar 2011 21:25:42 +0000 (22:25 +0100)]

Add missing 'dists' directory to method output_filename().

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sat, 5 Mar 2011 18:55:01 +0000 (19:55 +0100)]

Fully rename bootstrap_bin to scan.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sat, 5 Mar 2011 17:55:00 +0000 (17:55 +0000)]

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sat, 5 Mar 2011 17:26:52 +0000 (18:26 +0100)]

Rewrite most of dak/contents.py.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sat, 5 Mar 2011 15:23:25 +0000 (16:23 +0100)]

Update ContentsTestCase to the current formatting.

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sat, 5 Mar 2011 11:21:55 +0000 (11:21 +0000)]

some bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sat, 5 Mar 2011 11:19:18 +0000 (12:19 +0100)]

Implement class method ContentsWriter.write_all().

Signed-off-by: Torsten Werner <twerner@debian.org>

commit | commitdiff | tree

Torsten Werner [Sat, 5 Mar 2011 10:50:37 +0000 (10:50 +0000)]

bugfixes

Signed-off-by: Torsten Werner <twerner@debian.org>

dak changes to support and test code signing

RSS Atom