X-Git-Url: https://git.decadent.org.uk/gitweb/?p=dak.git;a=blobdiff_plain;f=docs%2FTODO;h=9fca17e9af378df246d4d654edba8f1f0484d21e;hp=778128cb2131cf58b2e5de2966efa55208bcba64;hb=026bdb9580e6d18bf504d2f44d46890df5d89f1a;hpb=ca1925d9291ec667f63dc6a895a3ac36fa9d6547 diff --git a/docs/TODO b/docs/TODO index 778128cb..9fca17e9 100644 --- a/docs/TODO +++ b/docs/TODO @@ -1,489 +1,89 @@ TODO ==== -[NB: I use this as a thought record/scribble, not everything on here - makes sense and/or is actually ever going to get done, so IIWY I - wouldn't use it as gospel for the future of dak or as a TODO - list for random hacking.] +Various +------- -================================================================================ +* Improve logging in dak control-suite. --add is silent, not nice. same + for --remove -Others ------- +* Throw away all DD uploaded .debs. (Depend on "Lintian based automated + rejects") + - Need a way to define a build-architecture for arch_all debs. Some of + them can only be build on certain architectures. + A control file header build-architecture: YXY should do it. + - its a suite option, not active for all at once. + - should have all buildd machines under dsa control - o 'dak check-overrides' should remove the src-only override when a - binary+source override exists +* Check TODO.old and move still-valid/useful entries over here. - o reject on > or < in a version constraint +* need a testsuite _badly_ -23:07 < aba> elmo: and, how about enhancing 'dak cruft-report' to spot half-dropped - binaries on one arch (i.e. package used to build A and B, but B is - no longer built on some archs)? +* database table "binaries" contains a column 'type TEXT NOT + NULL'. This should be made a FK on override_type, as it only contains + deb/udeb strings. - o tabnanny the source + - sql query to do the db work for it: + ALTER TABLE binaries ADD COLUMN new_type INT4 REFERENCES override_type(id); + UPDATE BINARIES SET new_type = 7 WHERE type = 'deb'; + UPDATE BINARIES SET new_type = 8 WHERE type = 'udeb'; + ALTER TABLE binaries DROP COLUMN type; + ALTER TABLE binaries RENAME COLUMN new_type TO type; - o drop map-unreleased + - needs updateX.py written and then the rest of the code changed to deal + with it. - o check email only portions of addresses match too, iff the names - don't, helps with the "James Troup " - vs. "" case. +* use pythonX.Y-tarfile to check orig.tar.gz timestamps too. - o ensure .dsc section/prio match .changes section/prio +* security global mail overrides should special case buildd stuff so + that buildds get ACCEPTED mails. Or maybe send them at + new-security-install time. That way upload-security doesn't grow + boundlessly. - o 'dak clean-suites' performance is kind of crap when asked to - remove a lot of files (e.g. 2k or so). +* debianqueued sucks. Reimplement in a sane way. - o we don't handle the case where an identical orig.tar.gz is - mentioned in the .changes, but not in unchecked; but should we - care? - - o 'dak ls' could do better sanity checking for -g/-G (e.g. not more - than one suite, etc.) - - o use python2.2-tarfile (once it's in stable?) to check orig.tar.gz - timestamps too. - - o need to decide on whether we're tying for most errors at once.. if - so (probably) then make sure code doesn't assume variables exist and - either way do something about checking error code of check_dsc and - later functions so we skip later checks if they're bailing. - - o the .dak stuff is fundamentally braindamaged, it's not versioned - so there's no way to change the format, yay me. need to fix. - probably by putting a version var as the first thing and checking - that.. auto-upgrade at least from original format would be good. - might also be a good idea to put everything in one big dict after - that? - - o [?, wishlist, distant future] RFC2047-ing should be extended to - all headers of mails sent out. - - o reject sparc64 binaries in a non '*64*' package. - - o queue.py(source_exists): a) we take arguments as parameters that - we could figure out for ourselves (we're part of the Upload class - after all), b) we have this 3rd argument which defaults to "any" - but could in fact be dropped since no one uses it like that. - - o 'dak process-unchecked': doesn't handle bin-only NMUs of stuff - still in NEW, BYHAND or ACCEPTED (but not the pool) - not a big - deal, upload can be retried once the source is in the archive, but - still. - - o security global mail overrides should special case buildd stuff so - that buildds get ACCEPTED mails (or maybe 'dak security-install' (?)), that way - upload-security doesn't grow boundlessly. - - o 'dak security-install' should upload sourceful packages first, - otherwise with big packages (e.g. X) and esp. when source is !i386, - half the arches can be uploaded without source, get copied into - queue/unaccepted and promptly rejected. - - o 'dak cruft-report's NVIU check doesn't catch cases where source - package changed name, should check binaries - too. [debian-devel@l.d.o, 2004-02-03] - - o cnf[Rm::logfile] is misnamed... - - i'd be kinda inclined to go with insisting the .changes file take - the form ---- BEGIN PGP MESSAGE --- -- - BEGIN PGP SIG -- END PGP MESSAGE -- with no lines before or after, - and rejecting .changes that didn't match that - - o 'dak cruft-report' should check for source packages not building any binaries - - o 'dak control-suite' should have a diff mode that accepts diff output! - - o 'dak clean-proposed-updates' doesn't deal with 'dak rm'-d - packages, partial replacements etc. and more. - - o 'dak reject-proposed-updates' blindly deletes with no check that - the delete failed which it might well given we only look for - package/version, not package/version _in p-u_. duh. - - o 'dak rm' should remove obsolete changes when removing from p-u, or - at least warn. or 'dak reject-proposed-updates' should handle it. - - o need a testsuite _badly_ - - o 'dak process-unchecked' crashes if run as a user in -n mode when - orig.tar.gz is in queue/new... - - [maybe I should reject debian packages with a non-Debian origin or bugs field] - [agreed; dunno what origin does but non-Debian bugs fields would be bad] - - o 'dak clean-suites' should make use of select..except select, temporary tables - etc. rather than looping and calling SQL every time so we can do - suite removal sanely (see potato-removal document) - - o 'dak rm' will happily include packages in the Cc list that aren't - being removed... - - o 'dak rm' doesn't remove udebs when removing the source they build from - - o check_dsc_against_db's "delete an entry from files while you're - not looking" habit is Evil and Bad. - - o 'dak process-new' allows you to edit the section and change the - component, but really shouldn't. - - o 'dak rm' needs to, when not sending bug close mails, promote Cc: to - To: and send the mail anyways. - - o the lockfile (Archive_Maintenance_In_Progress) should probably be in a conf file - - o 'dak ls' should cross-check the b.source field and if it's not - null and s.name linked from it != the source given in - -S/--source-and-binary ignore. - - o 'dak reject-proposed-updates' sucks; it should a) only spam d-i - for sourceful rejections, b) sort stuff so it rejects sourceful - stuff first. the non-sourceful should probably get a form mail, c) - automate the non-sourceful stuff (see b). - - o 'dak process-unchecked' should do q-d stuff for faster AA [ryan] - - o split the morgue into source and binary so binaries can be purged first! - - o per-architecture priorities for things like different arch'es - gcc's, silly BSD libftw, palo, etc. - - o use postgres 7.2's built-in stat features to figure out how indices are used etc. - - o 'dak init-archive' shouldn't be using location, it should run down suites instead - - o 'dak clean-proposed-updates' needs to know about udebs - - o by default hamstring dak's mail sending so that it won't send - anything until someone edits a script; it's been used far too - much to send spam atm :( - - o $ftpdir/indices isn't created by 'dak init-dir' because it's not in dak.conf - - o sanity check depends/recommends/suggests too? in fact for any - empty field? - -[minor] 'dak process-accepted's copychanges, copydotdak handling - sucks, the per-suite thing is static for all packages, so work out - in advance dummy. - -[dak ls] # filenames ? -[dak ls] # maintainer, component, install date (source only?), fingerprint? - - o UrgencyLog stuff should minimize it's bombing out(?) - o Log stuff should open the log file - - o 'dak queue-report' should footnote the actual notes, and also * - the versions with notes so we can see new versions since being - noted... - - o 'dak queue-report' should have alternative sorting options, including reverse - and without or without differentiaion. - - o 'dak import-users-from-passwd' should sync debadmin and ftpmaster (?) - - o Can't read file.: - /org/security.debian.org/queue/accepted/accepted/apache-perl_1.3.9-14.1-1.21.20000309-1_sparc.dak. - You assume that the filenames are relative to accepted/, might want - to doc or fix that. - - o the orig was in NEW, the changes that caused it to be NEW - were pulled out in -2, and we end up with no orig in the archive - :( - - o SecurityQueueBuild doesn't handle the case of foo_3.3woody1 with a - new .orig.tar.gz followed by a foo_3.3potato1 with the same - .orig.tar.gz; 'dak process-unchecked' sees it and copes, but the AA - code doesn't and can't really easily know so the potato AA dir is - left with no .orig.tar.gz copy. doh. - - o orig.tar.gz in accepted not handled properly (?) - - o 'dak security-install' doesn't include .orig.tar.gz but it should - - o permissions (paranoia, group write, etc.) configurability and overhaul - - o remember duplicate copyrights in 'dak process-new' and skip them, per package - - o ove option for 'dak process-new' byhand proecessing - - o 'dak cruft-report' could do with overrides - - o database.get_location_id should handle the lack of archive_id properly - - o the whole versioncmp thing should be documented - - o 'dak process-new' doesn't do the right thing with -2 and -1 uploads, as you can - end up with the .orig.tar.gz not in the pool - - o 'dak process-new' exits if you check twice (aj) - - o 'dak process-new' doesn't trap signals from 'dak examine-package' properly - - o queued and/or perl on sparc stable sucks - reimplement it. - - o aj's bin nmu changes - - o 'dak process-new': - * priority >> optional - * arch != {any,all} - * build-depends wrong (via 'dak compare-suites') - * suid - * conflicts - * notification/stats to admin daily - o trap 'dak examine-package' exiting - o distinguish binary only versus others (neuro) - - o cache changes parsed from ordering (careful tho: would be caching - .changes from world writable incoming, not holding) - - o dak doesn't recognise binonlyNMUs correctly in terms of telling - who their source is; source-must-exist does, but the info is not - propogated down. - - o Fix BTS vs. dak sync issues by queueing(via BSMTP) BTS mail so - that it can be released on deman (e.g. ETRN to exim). - - o maintainers file needs overrides - - [ change override.maintainer to override.maintainer-from + - override.maintainer-to and have them reference the maintainers - table. Then fix 'dak make-maintainers' to use them and write some scripting - to handle the Santiago situation. ] - - o Validate Depends (et al.) [it should match \(\s*(<<|<|<=|=|>=|>|>>)\s*\)] - - o Clean up DONE; archive to tar file every 2 weeks, update tar tvzf INDEX file. - - o testing-updates suite: if binary-only and version << version in - unstable and source-ver ~= source-ver in testing; then map - unstable -> testing-updates ? - - o hooks or configurability for debian specific checks (e.g. check_urgency, auto-building support) - - o morgue needs auto-cleaning (?) - - o dak stats: two modes, all included, seperate - o dak stats: add non-US - o dak stats: add ability to control components, architectures, archives, suites - o dak stats: add key to expand header - -================================================================================ - -queue/approved +NEW processing -------------- +* 'dak process-new' allows you to edit the section and change the + component, but really shouldn't allow the component change. - o What to do with multi-suite uploads? Presumably hold in unapproved - and warn? Or what? Can't accept just for unstable or reject just - from stable. - - o Whenever we check for anything in accepted we also need to check in - unapproved. - - o non-sourceful uploads should go straight through if they have - source in accepted or the archive. - - o security uploads on auric should be pre-approved. - -================================================================================ - -Less Urgent ------------ - - o change utils.copy to try rename() first - - o [hard, long term] unchecked -> accepted should go into the db, not - a suite, but similar. this would allow dak to get even faster, - make 'dak ls' more useful, decomplexify specialacceptedautobuild - and generally be more sane. may even be helpful to have e.g. new - in the DB, so that we avoid corner cases like the .orig.tar.gz - disappearing 'cos the package has been entirely removed but was - still on stayofexecution when it entered new. - - o Logging [mostly done] (todo: 'dak clean-suites' (hard), .. ?) - - o 'dak process-unchecked': the tar extractor class doesn't need to be redone for each package - - o reverse of source-must-exist; i.e. binary-for-source-must-not-exist - o REJECT reminders in 'dak clean-queues'. - o 'dak examine-package' should check for conflicts and warn about them visavis priority [rmurray] - o store a list of removed/files versions; also compare against them. - [but be careful about scalability] - - o dak examine-package: print_copyright should be a lot more intelligent - @ handle copyright.gz - @ handle copyright.ja and copyright - @ handle (detect at least) symlinks to another package's doc directory - @ handle and/or fall back on source files (?) - - o To incorporate from utils: - @ unreject - - o auto-purge out-of-date stuff from non-free/contrib so that testing and stuff works - o doogie's binary -> source index - o jt's web stuff, matt's changelog stuff (overlap) - - o [Hard] Need to merge non-non-US and non-US DBs. - - o experimental needs to auto clean (relative to unstable) [partial: - 'dak cruft-report' warns about this] - - o Do a checkpc(1)-a-like which sanitizes a config files. - o fix parse_changes()/build_file_list() to sanity check filenames - o saftey check and/or rename debs so they match what they should be - - o Improve 'dak compare-suites'. - o Need to optimize all the queries by using EXAMINE and building some INDEXs. - [postgresql 7.2 will help here] - o Need to enclose all the setting SQL stuff in transactions (mostly done). - o Need to finish 'dak init-db' (a way to sync dak.conf and the DB) - o Need the ability to rebuild all other tables from dists _or_ pools (in the event of disaster) (?) - o Make the --help and --version options do stuff for all scripts - - o 'dak make-maintainers' can't handle whitespace-only lines (for the moment, this is feature) - - o generic way of saying isabinary and isadsc. (?) - - o s/distribution/suite/g - - o cron.weekly: - @ weekly postins to d-c (?) - @ backup of report (?) - @ backup of changes.tgz (?) - - o --help doesn't work without /etc/dak/dak.conf (or similar) at - least existing. - - o rename 'dak compare-suites' (clashes with existing 'dak compare-suites')... - - * Harder: - - o interrupting of stracing 'dak process-unchecked' causes exceptions errors from apt_inst calls - o dependency checking (esp. stable) (partially done) - o override checks sucks; it needs to track changes made by the - maintainer and pass them onto ftpmaster instead of warning the - maintainer. - o need to do proper rfc822 escaping of from lines (as opposed to s/\.//g) - o Revisit linking of binary->source in install() in dak. - o Fix component handling in overrides (aj) - o Fix lack of entires in source overrides (aj) - o direport misreports things as section 'devel' (? we don't use direport) - o vrfy check of every Maintainer+Changed-By address; valid for 3 months. - o binary-all should be done on a per-source, per-architecture package - basis to avoid, e.g. the perl-modules problem. - o a source-missing-diff check: if the version has a - in it, and it - is sourceful, it needs orig and diff, e.g. if someone uploads - esound_0.2.22-6, and it is sourceful, and there is no diff -> - REJECT (version has a dash, therefore not debian native.) - o check linking of .tar.gz's to .dsc's.. see proftpd 1.2.1 as an example - o archive needs md5sum'ed regularly, but takes too long to do all - in one go; make progressive or weekly. - o something needs to clear out .changes files from p-u when - removing stuff superseded by newer versions. [but for now we have - 'dak clean-proposed-updates'] - o test sig checking stuff in test/ (stupid thing is not modularized due to global abuse) - o when encountering suspicous things (e.g. file tainting) do something more drastic - - * Easy: - - o suite mapping and component mapping are parsed per changes file, - they should probably be stored in a dictionary created at startup. - o don't stat/md5sum files you have entries for in the DB, moron - boy (Dak.check_source_blah_blah) - o promote changes["changes"] to mandatory in dak.py(dump_vars) - after a month or so (or all .dak files contain in the queue - contain it). - o 'dak rm' should behave better with -a and without -b; see - gcc-defaults removal for an example. - o Reject on misconfigured kernel-package uploads - o utils.extract_component_from_section: main/utils -> main/utils, main rather than utils, main - o Fix 'dak process-unchecked' to warn if run when not in incoming or p-u - o dak should validate multi-suite uploads; only possible valid one - is "stable unstable" - o cron.daily* should change umask (aj sucks) - o 'dak cruft-report' doesn't look at debian-installer but should. - o 'dak cruft-report' needs to check for binary-less source packages. - o 'dak cruft-report' could accept a suite argument (?) - o byhand stuff should send notification - o 'dak poolize' should udpate db; move files, not the other way around [neuro] - o 'dak rm' should update the stable changelog [joey] - o update tagdb.dia +* 'dak process-new' doesn't do the right thing with -2 and -1 uploads, + as you can end up with the .orig.tar.gz not in the pool or belonging + to a wrong suite. - * Bizzare/uncertain: +* 'dak process-new' doesn't trap signals from 'dak examine-package' properly - o drop rather dubious currval stuff (?) - o rationalize os.path.join() usage - o 'dak cruft-report' also doesn't seem to warn about missing binary packages (??) - o logging: hostname + pid ? - o ANAIS should be done in dak (?) - o Add an 'add' ability to 'dak rm' (? separate prog maybe) - o Replicate old dinstall report stuff (? needed ?) - o Handle the case of 1:1.1 which would overwrite 1.1 (?) - o maybe drop -r/--regex in 'dak ls', make it the default and - implement -e/--exact (a la joey's "elmo") - o dsc files are not checked for existence/perms (only an issue if - they're in the .dsc, but not the .changes.. possible?) +* 'dak queue-report' should footnote the actual notes, and also * the + versions with notes so we can see new versions since being noted... - * Cleanups & misc: +* the orig was in NEW, the changes that caused it to be NEW + were pulled out in -2, and we end up with no orig in the archive :( - o db_access' get_files needs to use exceptions not this None, > 0, < 0 return val BS (?) - o The untouchable flag doesn't stop new packages being added to ``untouchable'' suites -================================================================================ -Packaging ---------- +Override handling +----------------- +* 'dak check-overrides' should remove the src-only override when a + binary+source override exists - o Fix stuff to look in sensible places for libs and config file in debian package (?) +* override checks sucks; it needs to track changes made by the + maintainer and pass them onto ftpmaster instead of warning the maintainer. -================================================================================ +* Fix component handling in overrides - --help manpage ------------------------------------------------------------------------------ -check-archive X -check-overrides X X -check-proposed-updates X -clean-proposed-updates X -clean-queues X -clean-suites X X -compare-suites X -control-overrides X X -control-suite X X -cruft-report X -decode-dot-dak X -examine-package X -generate-releases X -import-archive X -import-users-from-passwd X X -init-db X -init-dirs X -ls X X -make-maintainers X X -make-overrides X -make-suite-file-list X -poolize X X -process-accepted X X -process-new X X -process-unchecked X -queue-report X -rm X X -security-install X -stats X -symlink-dists X +Cruft +----- +* 'dak cruft-report' could do with overrides -================================================================================ +* cruft-report could spot "half-dropped" binaries. Like if a package + used to build A and B, but B is no longer built for half the + architectures. -Random useful-at-some-point SQL -------------------------------- +* cruft-report's NVIU check doesn't catch cases where source package + changed name, should check binaries too. [debian-devel@l.d.o, + 2004-02-03] -UPDATE files SET last_used = '1980-01-01' - FROM binaries WHERE binaries.architecture = - AND binaries.file = files.id; +* 'dak cruft-report' doesn't look at debian-installer but should. -DELETE FROM bin_associations - WHERE EXISTS (SELECT id FROM binaries - WHERE architecture = - AND id = bin_associations.bin); -================================================================================