TODO
====
-[NB: I use this as a thought record/scribble, not everything on here
- makes sense and/or is actually ever going to get done, so IIWY I
- wouldn't use it as gospel for the future of dak or as a TODO
- list for random hacking.]
+* Check TODO.old and move still-valid/useful entries over here.
-================================================================================
-Others
-------
+* database table "binaries" contains a column 'type TEXT NOT
+ NULL'. This should be made a FK on override_type, as it only contains
+ deb/udeb strings.
- o 'dak check-overrides' should remove the src-only override when a
- binary+source override exists
+- sql query to do the db work for it:
+ ALTER TABLE binaries ADD COLUMN new_type INT4 REFERENCES override_type(id);
+ UPDATE BINARIES SET new_type = 7 WHERE type = 'deb';
+ UPDATE BINARIES SET new_type = 8 WHERE type = 'udeb';
+ ALTER TABLE binaries DROP COLUMN type;
+ ALTER TABLE binaries RENAME COLUMN new_type TO type;
- o reject on > or < in a version constraint
+- needs updateX.py written and then the rest of the code changed to deal
+ with it.
- o 'dak reject-proposed-updates' should only start an editor once to
- capture a message; it will usually be the same message for all
- files on the same command line.
+* Checkout SQL Alchemy and probably use that for our database layer.
-23:07 < aba> elmo: and, how about enhancing 'dak cruft-report' to spot half-dropped
- binaries on one arch (i.e. package used to build A and B, but B is
- no longer built on some archs)?
- o tabnanny the source
-
- o drop map-unreleased
-
- o check email only portions of addresses match too, iff the names
- don't, helps with the "James Troup <james@nocrew.org>"
- vs. "<james@nocrew.org>" case.
-
- o ensure .dsc section/prio match .changes section/prio
-
- o 'dak clean-suites' performance is kind of crap when asked to
- remove a lot of files (e.g. 2k or so).
-
- o we don't handle the case where an identical orig.tar.gz is
- mentioned in the .changes, but not in unchecked; but should we
- care?
-
- o 'dak ls' could do better sanity checking for -g/-G (e.g. not more
- than one suite, etc.)
-
- o use python2.2-tarfile (once it's in stable?) to check orig.tar.gz
- timestamps too.
-
- o need to decide on whether we're tying for most errors at once.. if
- so (probably) then make sure code doesn't assume variables exist and
- either way do something about checking error code of check_dsc and
- later functions so we skip later checks if they're bailing.
-
- o the .dak stuff is fundamentally braindamaged, it's not versioned
- so there's no way to change the format, yay me. need to fix.
- probably by putting a version var as the first thing and checking
- that.. auto-upgrade at least from original format would be good.
- might also be a good idea to put everything in one big dict after
- that?
-
- o [?, wishlist, distant future] RFC2047-ing should be extended to
- all headers of mails sent out.
-
- o reject sparc64 binaries in a non '*64*' package.
-
- o queue.py(source_exists): a) we take arguments as parameters that
- we could figure out for ourselves (we're part of the Upload class
- after all), b) we have this 3rd argument which defaults to "any"
- but could in fact be dropped since no one uses it like that.
-
- o 'dak process-unchecked': doesn't handle bin-only NMUs of stuff
- still in NEW, BYHAND or ACCEPTED (but not the pool) - not a big
- deal, upload can be retried once the source is in the archive, but
- still.
-
- o security global mail overrides should special case buildd stuff so
- that buildds get ACCEPTED mails (or maybe 'dak security-install' (?)), that way
- upload-security doesn't grow boundlessly.
-
- o 'dak security-install' should upload sourceful packages first,
- otherwise with big packages (e.g. X) and esp. when source is !i386,
- half the arches can be uploaded without source, get copied into
- queue/unaccepted and promptly rejected.
-
- o 'dak cruft-report's NVIU check doesn't catch cases where source
- package changed name, should check binaries
- too. [debian-devel@l.d.o, 2004-02-03]
-
- o cnf[Rm::logfile] is misnamed...
-
-<aj> i'd be kinda inclined to go with insisting the .changes file take
- the form ---- BEGIN PGP MESSAGE --- <non -- BEGIN/END lines> --
- BEGIN PGP SIG -- END PGP MESSAGE -- with no lines before or after,
- and rejecting .changes that didn't match that
-
- o 'dak cruft-report' should check for source packages not building any binaries
-
- o 'dak control-suite' should have a diff mode that accepts diff output!
-
- o 'dak clean-proposed-updates' doesn't deal with 'dak rm'-d
- packages, partial replacements etc. and more.
-
- o 'dak reject-proposed-updates' blindly deletes with no check that
- the delete failed which it might well given we only look for
- package/version, not package/version _in p-u_. duh.
-
- o 'dak rm' should remove obsolete changes when removing from p-u, or
- at least warn. or 'dak reject-proposed-updates' should handle it.
-
- o need a testsuite _badly_
-
- o 'dak process-unchecked' crashes if run as a user in -n mode when
- orig.tar.gz is in queue/new...
-
-<elmo_home> [<random>maybe I should reject debian packages with a non-Debian origin or bugs field</>]
-<Kamion> [<random>agreed; dunno what origin does but non-Debian bugs fields would be bad]
-
- o 'dak clean-suites' should make use of select..except select, temporary tables
- etc. rather than looping and calling SQL every time so we can do
- suite removal sanely (see potato-removal document)
-
- o 'dak rm' will happily include packages in the Cc list that aren't
- being removed...
-
- o 'dak rm' doesn't remove udebs when removing the source they build from
-
- o check_dsc_against_db's "delete an entry from files while you're
- not looking" habit is Evil and Bad.
-
- o 'dak process-new' allows you to edit the section and change the
- component, but really shouldn't.
-
- o 'dak rm' needs to, when not sending bug close mails, promote Cc: to
- To: and send the mail anyways.
-
- o the lockfile (Archive_Maintenance_In_Progress) should probably be in a conf file
-
- o 'dak ls' should cross-check the b.source field and if it's not
- null and s.name linked from it != the source given in
- -S/--source-and-binary ignore.
-
- o 'dak reject-proposed-updates' sucks; it should a) only spam d-i
- for sourceful rejections, b) sort stuff so it rejects sourceful
- stuff first. the non-sourceful should probably get a form mail, c)
- automate the non-sourceful stuff (see b).
-
- o 'dak process-unchecked' should do q-d stuff for faster AA [ryan]
-
- o split the morgue into source and binary so binaries can be purged first!
-
- o per-architecture priorities for things like different arch'es
- gcc's, silly BSD libftw, palo, etc.
-
- o use postgres 7.2's built-in stat features to figure out how indices are used etc.
-
- o 'dak init-archive' shouldn't be using location, it should run down suites instead
-
- o 'dak clean-proposed-updates' needs to know about udebs
-
- o by default hamstring dak's mail sending so that it won't send
- anything until someone edits a script; it's been used far too
- much to send spam atm :(
-
- o $ftpdir/indices isn't created by 'dak init-dir' because it's not in dak.conf
-
- o sanity check depends/recommends/suggests too? in fact for any
- empty field?
-
-[minor] 'dak process-accepted's copychanges, copydotdak handling
- sucks, the per-suite thing is static for all packages, so work out
- in advance dummy.
-
-[dak ls] # filenames ?
-[dak ls] # maintainer, component, install date (source only?), fingerprint?
-
- o UrgencyLog stuff should minimize it's bombing out(?)
- o Log stuff should open the log file
-
- o 'dak queue-report' should footnote the actual notes, and also *
- the versions with notes so we can see new versions since being
- noted...
-
- o 'dak queue-report' should have alternative sorting options, including reverse
- and without or without differentiaion.
-
- o 'dak import-users-from-passwd' should sync debadmin and ftpmaster (?)
-
- o <drow> Can't read file.:
- /org/security.debian.org/queue/accepted/accepted/apache-perl_1.3.9-14.1-1.21.20000309-1_sparc.dak.
- You assume that the filenames are relative to accepted/, might want
- to doc or fix that.
-
- o <neuro> the orig was in NEW, the changes that caused it to be NEW
- were pulled out in -2, and we end up with no orig in the archive
- :(
-
- o SecurityQueueBuild doesn't handle the case of foo_3.3woody1 with a
- new .orig.tar.gz followed by a foo_3.3potato1 with the same
- .orig.tar.gz; 'dak process-unchecked' sees it and copes, but the AA
- code doesn't and can't really easily know so the potato AA dir is
- left with no .orig.tar.gz copy. doh.
-
- o orig.tar.gz in accepted not handled properly (?)
-
- o 'dak security-install' doesn't include .orig.tar.gz but it should
-
- o permissions (paranoia, group write, etc.) configurability and overhaul
-
- o remember duplicate copyrights in 'dak process-new' and skip them, per package
-
- o <M>ove option for 'dak process-new' byhand proecessing
-
- o 'dak cruft-report' could do with overrides
-
- o database.get_location_id should handle the lack of archive_id properly
-
- o the whole versioncmp thing should be documented
-
- o 'dak process-new' doesn't do the right thing with -2 and -1 uploads, as you can
- end up with the .orig.tar.gz not in the pool
-
- o 'dak process-new' exits if you check twice (aj)
-
- o 'dak process-new' doesn't trap signals from 'dak examine-package' properly
-
- o queued and/or perl on sparc stable sucks - reimplement it.
-
- o aj's bin nmu changes
-
- o 'dak process-new':
- * priority >> optional
- * arch != {any,all}
- * build-depends wrong (via 'dak compare-suites')
- * suid
- * conflicts
- * notification/stats to admin daily
- o trap 'dak examine-package' exiting
- o distinguish binary only versus others (neuro)
-
- o cache changes parsed from ordering (careful tho: would be caching
- .changes from world writable incoming, not holding)
-
- o dak doesn't recognise binonlyNMUs correctly in terms of telling
- who their source is; source-must-exist does, but the info is not
- propogated down.
-
- o Fix BTS vs. dak sync issues by queueing(via BSMTP) BTS mail so
- that it can be released on deman (e.g. ETRN to exim).
-
- o maintainers file needs overrides
-
- [ change override.maintainer to override.maintainer-from +
- override.maintainer-to and have them reference the maintainers
- table. Then fix 'dak make-maintainers' to use them and write some scripting
- to handle the Santiago situation. ]
-
- o Validate Depends (et al.) [it should match \(\s*(<<|<|<=|=|>=|>|>>)\s*<VERSIONREGEXP>\)]
-
- o Clean up DONE; archive to tar file every 2 weeks, update tar tvzf INDEX file.
-
- o testing-updates suite: if binary-only and version << version in
- unstable and source-ver ~= source-ver in testing; then map
- unstable -> testing-updates ?
-
- o hooks or configurability for debian specific checks (e.g. check_urgency, auto-building support)
-
- o morgue needs auto-cleaning (?)
-
- o dak stats: two modes, all included, seperate
- o dak stats: add non-US
- o dak stats: add ability to control components, architectures, archives, suites
- o dak stats: add key to expand header
-
-================================================================================
-
-queue/approved
---------------
-
- o What to do with multi-suite uploads? Presumably hold in unapproved
- and warn? Or what? Can't accept just for unstable or reject just
- from stable.
-
- o Whenever we check for anything in accepted we also need to check in
- unapproved.
-
- o non-sourceful uploads should go straight through if they have
- source in accepted or the archive.
-
- o security uploads on auric should be pre-approved.
-
-================================================================================
-
-Less Urgent
------------
-
- o change utils.copy to try rename() first
-
- o [hard, long term] unchecked -> accepted should go into the db, not
- a suite, but similar. this would allow dak to get even faster,
- make 'dak ls' more useful, decomplexify specialacceptedautobuild
- and generally be more sane. may even be helpful to have e.g. new
- in the DB, so that we avoid corner cases like the .orig.tar.gz
- disappearing 'cos the package has been entirely removed but was
- still on stayofexecution when it entered new.
-
- o Logging [mostly done] (todo: 'dak clean-suites' (hard), .. ?)
-
- o 'dak process-unchecked': the tar extractor class doesn't need to be redone for each package
-
- o reverse of source-must-exist; i.e. binary-for-source-must-not-exist
- o REJECT reminders in 'dak clean-queues'.
- o 'dak examine-package' should check for conflicts and warn about them visavis priority [rmurray]
- o store a list of removed/files versions; also compare against them.
- [but be careful about scalability]
-
- o dak examine-package: print_copyright should be a lot more intelligent
- @ handle copyright.gz
- @ handle copyright.ja and copyright
- @ handle (detect at least) symlinks to another package's doc directory
- @ handle and/or fall back on source files (?)
-
- o To incorporate from utils:
- @ unreject
-
- o auto-purge out-of-date stuff from non-free/contrib so that testing and stuff works
- o doogie's binary -> source index
- o jt's web stuff, matt's changelog stuff (overlap)
-
- o [Hard] Need to merge non-non-US and non-US DBs.
-
- o experimental needs to auto clean (relative to unstable) [partial:
- 'dak cruft-report' warns about this]
-
- o Do a checkpc(1)-a-like which sanitizes a config files.
- o fix parse_changes()/build_file_list() to sanity check filenames
- o saftey check and/or rename debs so they match what they should be
-
- o Improve 'dak compare-suites'.
- o Need to optimize all the queries by using EXAMINE and building some INDEXs.
- [postgresql 7.2 will help here]
- o Need to enclose all the setting SQL stuff in transactions (mostly done).
- o Need to finish 'dak init-db' (a way to sync dak.conf and the DB)
- o Need the ability to rebuild all other tables from dists _or_ pools (in the event of disaster) (?)
- o Make the --help and --version options do stuff for all scripts
-
- o 'dak make-maintainers' can't handle whitespace-only lines (for the moment, this is feature)
-
- o generic way of saying isabinary and isadsc. (?)
-
- o s/distribution/suite/g
-
- o cron.weekly:
- @ weekly postins to d-c (?)
- @ backup of report (?)
- @ backup of changes.tgz (?)
-
- o --help doesn't work without /etc/dak/dak.conf (or similar) at
- least existing.
-
- o rename 'dak compare-suites' (clashes with existing 'dak compare-suites')...
-
- * Harder:
-
- o interrupting of stracing 'dak process-unchecked' causes exceptions errors from apt_inst calls
- o dependency checking (esp. stable) (partially done)
- o override checks sucks; it needs to track changes made by the
- maintainer and pass them onto ftpmaster instead of warning the
- maintainer.
- o need to do proper rfc822 escaping of from lines (as opposed to s/\.//g)
- o Revisit linking of binary->source in install() in dak.
- o Fix component handling in overrides (aj)
- o Fix lack of entires in source overrides (aj)
- o direport misreports things as section 'devel' (? we don't use direport)
- o vrfy check of every Maintainer+Changed-By address; valid for 3 months.
- o binary-all should be done on a per-source, per-architecture package
- basis to avoid, e.g. the perl-modules problem.
- o a source-missing-diff check: if the version has a - in it, and it
- is sourceful, it needs orig and diff, e.g. if someone uploads
- esound_0.2.22-6, and it is sourceful, and there is no diff ->
- REJECT (version has a dash, therefore not debian native.)
- o check linking of .tar.gz's to .dsc's.. see proftpd 1.2.1 as an example
- o archive needs md5sum'ed regularly, but takes too long to do all
- in one go; make progressive or weekly.
- o something needs to clear out .changes files from p-u when
- removing stuff superseded by newer versions. [but for now we have
- 'dak clean-proposed-updates']
- o test sig checking stuff in test/ (stupid thing is not modularized due to global abuse)
- o when encountering suspicous things (e.g. file tainting) do something more drastic
-
- * Easy:
-
- o suite mapping and component mapping are parsed per changes file,
- they should probably be stored in a dictionary created at startup.
- o don't stat/md5sum files you have entries for in the DB, moron
- boy (Dak.check_source_blah_blah)
- o promote changes["changes"] to mandatory in dak.py(dump_vars)
- after a month or so (or all .dak files contain in the queue
- contain it).
- o 'dak rm' should behave better with -a and without -b; see
- gcc-defaults removal for an example.
- o Reject on misconfigured kernel-package uploads
- o utils.extract_component_from_section: main/utils -> main/utils, main rather than utils, main
- o Fix 'dak process-unchecked' to warn if run when not in incoming or p-u
- o dak should validate multi-suite uploads; only possible valid one
- is "stable unstable"
- o cron.daily* should change umask (aj sucks)
- o 'dak cruft-report' doesn't look at debian-installer but should.
- o 'dak cruft-report' needs to check for binary-less source packages.
- o 'dak cruft-report' could accept a suite argument (?)
- o byhand stuff should send notification
- o 'dak poolize' should udpate db; move files, not the other way around [neuro]
- o 'dak rm' should update the stable changelog [joey]
- o update tagdb.dia
-
- * Bizzare/uncertain:
-
- o drop rather dubious currval stuff (?)
- o rationalize os.path.join() usage
- o 'dak cruft-report' also doesn't seem to warn about missing binary packages (??)
- o logging: hostname + pid ?
- o ANAIS should be done in dak (?)
- o Add an 'add' ability to 'dak rm' (? separate prog maybe)
- o Replicate old dinstall report stuff (? needed ?)
- o Handle the case of 1:1.1 which would overwrite 1.1 (?)
- o maybe drop -r/--regex in 'dak ls', make it the default and
- implement -e/--exact (a la joey's "elmo")
- o dsc files are not checked for existence/perms (only an issue if
- they're in the .dsc, but not the .changes.. possible?)
-
- * Cleanups & misc:
-
- o db_access' get_files needs to use exceptions not this None, > 0, < 0 return val BS (?)
- o The untouchable flag doesn't stop new packages being added to ``untouchable'' suites
-
-================================================================================
-
-Packaging
----------
-
- o Fix stuff to look in sensible places for libs and config file in debian package (?)
-
-================================================================================
-
- --help manpage
------------------------------------------------------------------------------
-check-archive X
-check-overrides X X
-check-proposed-updates X
-clean-proposed-updates X
-clean-queues X
-clean-suites X X
-compare-suites X
-control-overrides X X
-control-suite X X
-cruft-report X
-decode-dot-dak X
-examine-package X
-generate-releases X
-import-archive X
-import-users-from-passwd X X
-init-db X
-init-dirs X
-ls X X
-make-maintainers X X
-make-overrides X
-make-suite-file-list X
-poolize X X
-process-accepted X X
-process-new X X
-process-unchecked X
-queue-report X
-rm X X
-security-install X
-stats X
-symlink-dists X
-
-
-================================================================================
-
-Random useful-at-some-point SQL
--------------------------------
-
-UPDATE files SET last_used = '1980-01-01'
- FROM binaries WHERE binaries.architecture = <x>
- AND binaries.file = files.id;
-
-DELETE FROM bin_associations
- WHERE EXISTS (SELECT id FROM binaries
- WHERE architecture = <x>
- AND id = bin_associations.bin);
-
-================================================================================
--- /dev/null
+ TODO
+ ====
+
+[NB: I use this as a thought record/scribble, not everything on here
+ makes sense and/or is actually ever going to get done, so IIWY I
+ wouldn't use it as gospel for the future of dak or as a TODO
+ list for random hacking.]
+
+================================================================================
+
+Others
+------
+
+ o 'dak check-overrides' should remove the src-only override when a
+ binary+source override exists
+
+ o reject on > or < in a version constraint
+
+ o 'dak reject-proposed-updates' should only start an editor once to
+ capture a message; it will usually be the same message for all
+ files on the same command line.
+
+23:07 < aba> elmo: and, how about enhancing 'dak cruft-report' to spot half-dropped
+ binaries on one arch (i.e. package used to build A and B, but B is
+ no longer built on some archs)?
+
+ o tabnanny the source
+
+ o drop map-unreleased
+
+ o check email only portions of addresses match too, iff the names
+ don't, helps with the "James Troup <james@nocrew.org>"
+ vs. "<james@nocrew.org>" case.
+
+ o ensure .dsc section/prio match .changes section/prio
+
+ o 'dak clean-suites' performance is kind of crap when asked to
+ remove a lot of files (e.g. 2k or so).
+
+ o we don't handle the case where an identical orig.tar.gz is
+ mentioned in the .changes, but not in unchecked; but should we
+ care?
+
+ o 'dak ls' could do better sanity checking for -g/-G (e.g. not more
+ than one suite, etc.)
+
+ o use python2.2-tarfile (once it's in stable?) to check orig.tar.gz
+ timestamps too.
+
+ o need to decide on whether we're tying for most errors at once.. if
+ so (probably) then make sure code doesn't assume variables exist and
+ either way do something about checking error code of check_dsc and
+ later functions so we skip later checks if they're bailing.
+
+ o the .dak stuff is fundamentally braindamaged, it's not versioned
+ so there's no way to change the format, yay me. need to fix.
+ probably by putting a version var as the first thing and checking
+ that.. auto-upgrade at least from original format would be good.
+ might also be a good idea to put everything in one big dict after
+ that?
+
+ o [?, wishlist, distant future] RFC2047-ing should be extended to
+ all headers of mails sent out.
+
+ o reject sparc64 binaries in a non '*64*' package.
+
+ o queue.py(source_exists): a) we take arguments as parameters that
+ we could figure out for ourselves (we're part of the Upload class
+ after all), b) we have this 3rd argument which defaults to "any"
+ but could in fact be dropped since no one uses it like that.
+
+ o 'dak process-unchecked': doesn't handle bin-only NMUs of stuff
+ still in NEW, BYHAND or ACCEPTED (but not the pool) - not a big
+ deal, upload can be retried once the source is in the archive, but
+ still.
+
+ o security global mail overrides should special case buildd stuff so
+ that buildds get ACCEPTED mails (or maybe 'dak security-install' (?)), that way
+ upload-security doesn't grow boundlessly.
+
+ o 'dak security-install' should upload sourceful packages first,
+ otherwise with big packages (e.g. X) and esp. when source is !i386,
+ half the arches can be uploaded without source, get copied into
+ queue/unaccepted and promptly rejected.
+
+ o 'dak cruft-report's NVIU check doesn't catch cases where source
+ package changed name, should check binaries
+ too. [debian-devel@l.d.o, 2004-02-03]
+
+ o cnf[Rm::logfile] is misnamed...
+
+<aj> i'd be kinda inclined to go with insisting the .changes file take
+ the form ---- BEGIN PGP MESSAGE --- <non -- BEGIN/END lines> --
+ BEGIN PGP SIG -- END PGP MESSAGE -- with no lines before or after,
+ and rejecting .changes that didn't match that
+
+ o 'dak cruft-report' should check for source packages not building any binaries
+
+ o 'dak control-suite' should have a diff mode that accepts diff output!
+
+ o 'dak clean-proposed-updates' doesn't deal with 'dak rm'-d
+ packages, partial replacements etc. and more.
+
+ o 'dak reject-proposed-updates' blindly deletes with no check that
+ the delete failed which it might well given we only look for
+ package/version, not package/version _in p-u_. duh.
+
+ o 'dak rm' should remove obsolete changes when removing from p-u, or
+ at least warn. or 'dak reject-proposed-updates' should handle it.
+
+ o need a testsuite _badly_
+
+ o 'dak process-unchecked' crashes if run as a user in -n mode when
+ orig.tar.gz is in queue/new...
+
+<elmo_home> [<random>maybe I should reject debian packages with a non-Debian origin or bugs field</>]
+<Kamion> [<random>agreed; dunno what origin does but non-Debian bugs fields would be bad]
+
+ o 'dak clean-suites' should make use of select..except select, temporary tables
+ etc. rather than looping and calling SQL every time so we can do
+ suite removal sanely (see potato-removal document)
+
+ o 'dak rm' will happily include packages in the Cc list that aren't
+ being removed...
+
+ o 'dak rm' doesn't remove udebs when removing the source they build from
+
+ o check_dsc_against_db's "delete an entry from files while you're
+ not looking" habit is Evil and Bad.
+
+ o 'dak process-new' allows you to edit the section and change the
+ component, but really shouldn't.
+
+ o 'dak rm' needs to, when not sending bug close mails, promote Cc: to
+ To: and send the mail anyways.
+
+ o the lockfile (Archive_Maintenance_In_Progress) should probably be in a conf file
+
+ o 'dak ls' should cross-check the b.source field and if it's not
+ null and s.name linked from it != the source given in
+ -S/--source-and-binary ignore.
+
+ o 'dak reject-proposed-updates' sucks; it should a) only spam d-i
+ for sourceful rejections, b) sort stuff so it rejects sourceful
+ stuff first. the non-sourceful should probably get a form mail, c)
+ automate the non-sourceful stuff (see b).
+
+ o 'dak process-unchecked' should do q-d stuff for faster AA [ryan]
+
+ o split the morgue into source and binary so binaries can be purged first!
+
+ o per-architecture priorities for things like different arch'es
+ gcc's, silly BSD libftw, palo, etc.
+
+ o use postgres 7.2's built-in stat features to figure out how indices are used etc.
+
+ o 'dak init-archive' shouldn't be using location, it should run down suites instead
+
+ o 'dak clean-proposed-updates' needs to know about udebs
+
+ o by default hamstring dak's mail sending so that it won't send
+ anything until someone edits a script; it's been used far too
+ much to send spam atm :(
+
+ o $ftpdir/indices isn't created by 'dak init-dir' because it's not in dak.conf
+
+ o sanity check depends/recommends/suggests too? in fact for any
+ empty field?
+
+[minor] 'dak process-accepted's copychanges, copydotdak handling
+ sucks, the per-suite thing is static for all packages, so work out
+ in advance dummy.
+
+[dak ls] # filenames ?
+[dak ls] # maintainer, component, install date (source only?), fingerprint?
+
+ o UrgencyLog stuff should minimize it's bombing out(?)
+ o Log stuff should open the log file
+
+ o 'dak queue-report' should footnote the actual notes, and also *
+ the versions with notes so we can see new versions since being
+ noted...
+
+ o 'dak queue-report' should have alternative sorting options, including reverse
+ and without or without differentiaion.
+
+ o 'dak import-users-from-passwd' should sync debadmin and ftpmaster (?)
+
+ o <drow> Can't read file.:
+ /org/security.debian.org/queue/accepted/accepted/apache-perl_1.3.9-14.1-1.21.20000309-1_sparc.dak.
+ You assume that the filenames are relative to accepted/, might want
+ to doc or fix that.
+
+ o <neuro> the orig was in NEW, the changes that caused it to be NEW
+ were pulled out in -2, and we end up with no orig in the archive
+ :(
+
+ o SecurityQueueBuild doesn't handle the case of foo_3.3woody1 with a
+ new .orig.tar.gz followed by a foo_3.3potato1 with the same
+ .orig.tar.gz; 'dak process-unchecked' sees it and copes, but the AA
+ code doesn't and can't really easily know so the potato AA dir is
+ left with no .orig.tar.gz copy. doh.
+
+ o orig.tar.gz in accepted not handled properly (?)
+
+ o 'dak security-install' doesn't include .orig.tar.gz but it should
+
+ o permissions (paranoia, group write, etc.) configurability and overhaul
+
+ o remember duplicate copyrights in 'dak process-new' and skip them, per package
+
+ o <M>ove option for 'dak process-new' byhand proecessing
+
+ o 'dak cruft-report' could do with overrides
+
+ o database.get_location_id should handle the lack of archive_id properly
+
+ o the whole versioncmp thing should be documented
+
+ o 'dak process-new' doesn't do the right thing with -2 and -1 uploads, as you can
+ end up with the .orig.tar.gz not in the pool
+
+ o 'dak process-new' exits if you check twice (aj)
+
+ o 'dak process-new' doesn't trap signals from 'dak examine-package' properly
+
+ o queued and/or perl on sparc stable sucks - reimplement it.
+
+ o aj's bin nmu changes
+
+ o 'dak process-new':
+ * priority >> optional
+ * arch != {any,all}
+ * build-depends wrong (via 'dak compare-suites')
+ * suid
+ * conflicts
+ * notification/stats to admin daily
+ o trap 'dak examine-package' exiting
+ o distinguish binary only versus others (neuro)
+
+ o cache changes parsed from ordering (careful tho: would be caching
+ .changes from world writable incoming, not holding)
+
+ o dak doesn't recognise binonlyNMUs correctly in terms of telling
+ who their source is; source-must-exist does, but the info is not
+ propogated down.
+
+ o Fix BTS vs. dak sync issues by queueing(via BSMTP) BTS mail so
+ that it can be released on deman (e.g. ETRN to exim).
+
+ o maintainers file needs overrides
+
+ [ change override.maintainer to override.maintainer-from +
+ override.maintainer-to and have them reference the maintainers
+ table. Then fix 'dak make-maintainers' to use them and write some scripting
+ to handle the Santiago situation. ]
+
+ o Validate Depends (et al.) [it should match \(\s*(<<|<|<=|=|>=|>|>>)\s*<VERSIONREGEXP>\)]
+
+ o Clean up DONE; archive to tar file every 2 weeks, update tar tvzf INDEX file.
+
+ o testing-updates suite: if binary-only and version << version in
+ unstable and source-ver ~= source-ver in testing; then map
+ unstable -> testing-updates ?
+
+ o hooks or configurability for debian specific checks (e.g. check_urgency, auto-building support)
+
+ o morgue needs auto-cleaning (?)
+
+ o dak stats: two modes, all included, seperate
+ o dak stats: add non-US
+ o dak stats: add ability to control components, architectures, archives, suites
+ o dak stats: add key to expand header
+
+================================================================================
+
+queue/approved
+--------------
+
+ o What to do with multi-suite uploads? Presumably hold in unapproved
+ and warn? Or what? Can't accept just for unstable or reject just
+ from stable.
+
+ o Whenever we check for anything in accepted we also need to check in
+ unapproved.
+
+ o non-sourceful uploads should go straight through if they have
+ source in accepted or the archive.
+
+ o security uploads on auric should be pre-approved.
+
+================================================================================
+
+Less Urgent
+-----------
+
+ o change utils.copy to try rename() first
+
+ o [hard, long term] unchecked -> accepted should go into the db, not
+ a suite, but similar. this would allow dak to get even faster,
+ make 'dak ls' more useful, decomplexify specialacceptedautobuild
+ and generally be more sane. may even be helpful to have e.g. new
+ in the DB, so that we avoid corner cases like the .orig.tar.gz
+ disappearing 'cos the package has been entirely removed but was
+ still on stayofexecution when it entered new.
+
+ o Logging [mostly done] (todo: 'dak clean-suites' (hard), .. ?)
+
+ o 'dak process-unchecked': the tar extractor class doesn't need to be redone for each package
+
+ o reverse of source-must-exist; i.e. binary-for-source-must-not-exist
+ o REJECT reminders in 'dak clean-queues'.
+ o 'dak examine-package' should check for conflicts and warn about them visavis priority [rmurray]
+ o store a list of removed/files versions; also compare against them.
+ [but be careful about scalability]
+
+ o dak examine-package: print_copyright should be a lot more intelligent
+ @ handle copyright.gz
+ @ handle copyright.ja and copyright
+ @ handle (detect at least) symlinks to another package's doc directory
+ @ handle and/or fall back on source files (?)
+
+ o To incorporate from utils:
+ @ unreject
+
+ o auto-purge out-of-date stuff from non-free/contrib so that testing and stuff works
+ o doogie's binary -> source index
+ o jt's web stuff, matt's changelog stuff (overlap)
+
+ o [Hard] Need to merge non-non-US and non-US DBs.
+
+ o experimental needs to auto clean (relative to unstable) [partial:
+ 'dak cruft-report' warns about this]
+
+ o Do a checkpc(1)-a-like which sanitizes a config files.
+ o fix parse_changes()/build_file_list() to sanity check filenames
+ o saftey check and/or rename debs so they match what they should be
+
+ o Improve 'dak compare-suites'.
+ o Need to optimize all the queries by using EXAMINE and building some INDEXs.
+ [postgresql 7.2 will help here]
+ o Need to enclose all the setting SQL stuff in transactions (mostly done).
+ o Need to finish 'dak init-db' (a way to sync dak.conf and the DB)
+ o Need the ability to rebuild all other tables from dists _or_ pools (in the event of disaster) (?)
+ o Make the --help and --version options do stuff for all scripts
+
+ o 'dak make-maintainers' can't handle whitespace-only lines (for the moment, this is feature)
+
+ o generic way of saying isabinary and isadsc. (?)
+
+ o s/distribution/suite/g
+
+ o cron.weekly:
+ @ weekly postins to d-c (?)
+ @ backup of report (?)
+ @ backup of changes.tgz (?)
+
+ o --help doesn't work without /etc/dak/dak.conf (or similar) at
+ least existing.
+
+ o rename 'dak compare-suites' (clashes with existing 'dak compare-suites')...
+
+ * Harder:
+
+ o interrupting of stracing 'dak process-unchecked' causes exceptions errors from apt_inst calls
+ o dependency checking (esp. stable) (partially done)
+ o override checks sucks; it needs to track changes made by the
+ maintainer and pass them onto ftpmaster instead of warning the
+ maintainer.
+ o need to do proper rfc822 escaping of from lines (as opposed to s/\.//g)
+ o Revisit linking of binary->source in install() in dak.
+ o Fix component handling in overrides (aj)
+ o Fix lack of entires in source overrides (aj)
+ o direport misreports things as section 'devel' (? we don't use direport)
+ o vrfy check of every Maintainer+Changed-By address; valid for 3 months.
+ o binary-all should be done on a per-source, per-architecture package
+ basis to avoid, e.g. the perl-modules problem.
+ o a source-missing-diff check: if the version has a - in it, and it
+ is sourceful, it needs orig and diff, e.g. if someone uploads
+ esound_0.2.22-6, and it is sourceful, and there is no diff ->
+ REJECT (version has a dash, therefore not debian native.)
+ o check linking of .tar.gz's to .dsc's.. see proftpd 1.2.1 as an example
+ o archive needs md5sum'ed regularly, but takes too long to do all
+ in one go; make progressive or weekly.
+ o something needs to clear out .changes files from p-u when
+ removing stuff superseded by newer versions. [but for now we have
+ 'dak clean-proposed-updates']
+ o test sig checking stuff in test/ (stupid thing is not modularized due to global abuse)
+ o when encountering suspicous things (e.g. file tainting) do something more drastic
+
+ * Easy:
+
+ o suite mapping and component mapping are parsed per changes file,
+ they should probably be stored in a dictionary created at startup.
+ o don't stat/md5sum files you have entries for in the DB, moron
+ boy (Dak.check_source_blah_blah)
+ o promote changes["changes"] to mandatory in dak.py(dump_vars)
+ after a month or so (or all .dak files contain in the queue
+ contain it).
+ o 'dak rm' should behave better with -a and without -b; see
+ gcc-defaults removal for an example.
+ o Reject on misconfigured kernel-package uploads
+ o utils.extract_component_from_section: main/utils -> main/utils, main rather than utils, main
+ o Fix 'dak process-unchecked' to warn if run when not in incoming or p-u
+ o dak should validate multi-suite uploads; only possible valid one
+ is "stable unstable"
+ o cron.daily* should change umask (aj sucks)
+ o 'dak cruft-report' doesn't look at debian-installer but should.
+ o 'dak cruft-report' needs to check for binary-less source packages.
+ o 'dak cruft-report' could accept a suite argument (?)
+ o byhand stuff should send notification
+ o 'dak poolize' should udpate db; move files, not the other way around [neuro]
+ o 'dak rm' should update the stable changelog [joey]
+ o update tagdb.dia
+
+ * Bizzare/uncertain:
+
+ o drop rather dubious currval stuff (?)
+ o rationalize os.path.join() usage
+ o 'dak cruft-report' also doesn't seem to warn about missing binary packages (??)
+ o logging: hostname + pid ?
+ o ANAIS should be done in dak (?)
+ o Add an 'add' ability to 'dak rm' (? separate prog maybe)
+ o Replicate old dinstall report stuff (? needed ?)
+ o Handle the case of 1:1.1 which would overwrite 1.1 (?)
+ o maybe drop -r/--regex in 'dak ls', make it the default and
+ implement -e/--exact (a la joey's "elmo")
+ o dsc files are not checked for existence/perms (only an issue if
+ they're in the .dsc, but not the .changes.. possible?)
+
+ * Cleanups & misc:
+
+ o db_access' get_files needs to use exceptions not this None, > 0, < 0 return val BS (?)
+ o The untouchable flag doesn't stop new packages being added to ``untouchable'' suites
+
+================================================================================
+
+Packaging
+---------
+
+ o Fix stuff to look in sensible places for libs and config file in debian package (?)
+
+================================================================================
+
+ --help manpage
+-----------------------------------------------------------------------------
+check-archive X
+check-overrides X X
+check-proposed-updates X
+clean-proposed-updates X
+clean-queues X
+clean-suites X X
+compare-suites X
+control-overrides X X
+control-suite X X
+cruft-report X
+decode-dot-dak X
+examine-package X
+generate-releases X
+import-archive X
+import-users-from-passwd X X
+init-db X
+init-dirs X
+ls X X
+make-maintainers X X
+make-overrides X
+make-suite-file-list X
+poolize X X
+process-accepted X X
+process-new X X
+process-unchecked X
+queue-report X
+rm X X
+security-install X
+stats X
+symlink-dists X
+
+
+================================================================================
+
+Random useful-at-some-point SQL
+-------------------------------
+
+UPDATE files SET last_used = '1980-01-01'
+ FROM binaries WHERE binaries.architecture = <x>
+ AND binaries.file = files.id;
+
+DELETE FROM bin_associations
+ WHERE EXISTS (SELECT id FROM binaries
+ WHERE architecture = <x>
+ AND id = bin_associations.bin);
+
+================================================================================
projects / dak.git / commitdiff