X-Git-Url: https://git.decadent.org.uk/gitweb/?a=blobdiff_plain;f=docs%2FTODO.old;fp=docs%2FTODO.old;h=c4dbb4dfdafed0a05bf1eabc91ea1217f3771cdc;hb=f12e9681132e32a922a4743260306b7b1e712d26;hp=0000000000000000000000000000000000000000;hpb=4aaacfa9e4285c14808266b511be5650f4acb22a;p=dak.git diff --git a/docs/TODO.old b/docs/TODO.old new file mode 100644 index 00000000..c4dbb4df --- /dev/null +++ b/docs/TODO.old @@ -0,0 +1,493 @@ + TODO + ==== + +[NB: I use this as a thought record/scribble, not everything on here + makes sense and/or is actually ever going to get done, so IIWY I + wouldn't use it as gospel for the future of dak or as a TODO + list for random hacking.] + +================================================================================ + +Others +------ + + o 'dak check-overrides' should remove the src-only override when a + binary+source override exists + + o reject on > or < in a version constraint + + o 'dak reject-proposed-updates' should only start an editor once to + capture a message; it will usually be the same message for all + files on the same command line. + +23:07 < aba> elmo: and, how about enhancing 'dak cruft-report' to spot half-dropped + binaries on one arch (i.e. package used to build A and B, but B is + no longer built on some archs)? + + o tabnanny the source + + o drop map-unreleased + + o check email only portions of addresses match too, iff the names + don't, helps with the "James Troup " + vs. "" case. + + o ensure .dsc section/prio match .changes section/prio + + o 'dak clean-suites' performance is kind of crap when asked to + remove a lot of files (e.g. 2k or so). + + o we don't handle the case where an identical orig.tar.gz is + mentioned in the .changes, but not in unchecked; but should we + care? + + o 'dak ls' could do better sanity checking for -g/-G (e.g. not more + than one suite, etc.) + + o use python2.2-tarfile (once it's in stable?) to check orig.tar.gz + timestamps too. + + o need to decide on whether we're tying for most errors at once.. if + so (probably) then make sure code doesn't assume variables exist and + either way do something about checking error code of check_dsc and + later functions so we skip later checks if they're bailing. + + o the .dak stuff is fundamentally braindamaged, it's not versioned + so there's no way to change the format, yay me. need to fix. + probably by putting a version var as the first thing and checking + that.. auto-upgrade at least from original format would be good. + might also be a good idea to put everything in one big dict after + that? + + o [?, wishlist, distant future] RFC2047-ing should be extended to + all headers of mails sent out. + + o reject sparc64 binaries in a non '*64*' package. + + o queue.py(source_exists): a) we take arguments as parameters that + we could figure out for ourselves (we're part of the Upload class + after all), b) we have this 3rd argument which defaults to "any" + but could in fact be dropped since no one uses it like that. + + o 'dak process-unchecked': doesn't handle bin-only NMUs of stuff + still in NEW, BYHAND or ACCEPTED (but not the pool) - not a big + deal, upload can be retried once the source is in the archive, but + still. + + o security global mail overrides should special case buildd stuff so + that buildds get ACCEPTED mails (or maybe 'dak security-install' (?)), that way + upload-security doesn't grow boundlessly. + + o 'dak security-install' should upload sourceful packages first, + otherwise with big packages (e.g. X) and esp. when source is !i386, + half the arches can be uploaded without source, get copied into + queue/unaccepted and promptly rejected. + + o 'dak cruft-report's NVIU check doesn't catch cases where source + package changed name, should check binaries + too. [debian-devel@l.d.o, 2004-02-03] + + o cnf[Rm::logfile] is misnamed... + + i'd be kinda inclined to go with insisting the .changes file take + the form ---- BEGIN PGP MESSAGE --- -- + BEGIN PGP SIG -- END PGP MESSAGE -- with no lines before or after, + and rejecting .changes that didn't match that + + o 'dak cruft-report' should check for source packages not building any binaries + + o 'dak control-suite' should have a diff mode that accepts diff output! + + o 'dak clean-proposed-updates' doesn't deal with 'dak rm'-d + packages, partial replacements etc. and more. + + o 'dak reject-proposed-updates' blindly deletes with no check that + the delete failed which it might well given we only look for + package/version, not package/version _in p-u_. duh. + + o 'dak rm' should remove obsolete changes when removing from p-u, or + at least warn. or 'dak reject-proposed-updates' should handle it. + + o need a testsuite _badly_ + + o 'dak process-unchecked' crashes if run as a user in -n mode when + orig.tar.gz is in queue/new... + + [maybe I should reject debian packages with a non-Debian origin or bugs field] + [agreed; dunno what origin does but non-Debian bugs fields would be bad] + + o 'dak clean-suites' should make use of select..except select, temporary tables + etc. rather than looping and calling SQL every time so we can do + suite removal sanely (see potato-removal document) + + o 'dak rm' will happily include packages in the Cc list that aren't + being removed... + + o 'dak rm' doesn't remove udebs when removing the source they build from + + o check_dsc_against_db's "delete an entry from files while you're + not looking" habit is Evil and Bad. + + o 'dak process-new' allows you to edit the section and change the + component, but really shouldn't. + + o 'dak rm' needs to, when not sending bug close mails, promote Cc: to + To: and send the mail anyways. + + o the lockfile (Archive_Maintenance_In_Progress) should probably be in a conf file + + o 'dak ls' should cross-check the b.source field and if it's not + null and s.name linked from it != the source given in + -S/--source-and-binary ignore. + + o 'dak reject-proposed-updates' sucks; it should a) only spam d-i + for sourceful rejections, b) sort stuff so it rejects sourceful + stuff first. the non-sourceful should probably get a form mail, c) + automate the non-sourceful stuff (see b). + + o 'dak process-unchecked' should do q-d stuff for faster AA [ryan] + + o split the morgue into source and binary so binaries can be purged first! + + o per-architecture priorities for things like different arch'es + gcc's, silly BSD libftw, palo, etc. + + o use postgres 7.2's built-in stat features to figure out how indices are used etc. + + o 'dak init-archive' shouldn't be using location, it should run down suites instead + + o 'dak clean-proposed-updates' needs to know about udebs + + o by default hamstring dak's mail sending so that it won't send + anything until someone edits a script; it's been used far too + much to send spam atm :( + + o $ftpdir/indices isn't created by 'dak init-dir' because it's not in dak.conf + + o sanity check depends/recommends/suggests too? in fact for any + empty field? + +[minor] 'dak process-accepted's copychanges, copydotdak handling + sucks, the per-suite thing is static for all packages, so work out + in advance dummy. + +[dak ls] # filenames ? +[dak ls] # maintainer, component, install date (source only?), fingerprint? + + o UrgencyLog stuff should minimize it's bombing out(?) + o Log stuff should open the log file + + o 'dak queue-report' should footnote the actual notes, and also * + the versions with notes so we can see new versions since being + noted... + + o 'dak queue-report' should have alternative sorting options, including reverse + and without or without differentiaion. + + o 'dak import-users-from-passwd' should sync debadmin and ftpmaster (?) + + o Can't read file.: + /org/security.debian.org/queue/accepted/accepted/apache-perl_1.3.9-14.1-1.21.20000309-1_sparc.dak. + You assume that the filenames are relative to accepted/, might want + to doc or fix that. + + o the orig was in NEW, the changes that caused it to be NEW + were pulled out in -2, and we end up with no orig in the archive + :( + + o SecurityQueueBuild doesn't handle the case of foo_3.3woody1 with a + new .orig.tar.gz followed by a foo_3.3potato1 with the same + .orig.tar.gz; 'dak process-unchecked' sees it and copes, but the AA + code doesn't and can't really easily know so the potato AA dir is + left with no .orig.tar.gz copy. doh. + + o orig.tar.gz in accepted not handled properly (?) + + o 'dak security-install' doesn't include .orig.tar.gz but it should + + o permissions (paranoia, group write, etc.) configurability and overhaul + + o remember duplicate copyrights in 'dak process-new' and skip them, per package + + o ove option for 'dak process-new' byhand proecessing + + o 'dak cruft-report' could do with overrides + + o database.get_location_id should handle the lack of archive_id properly + + o the whole versioncmp thing should be documented + + o 'dak process-new' doesn't do the right thing with -2 and -1 uploads, as you can + end up with the .orig.tar.gz not in the pool + + o 'dak process-new' exits if you check twice (aj) + + o 'dak process-new' doesn't trap signals from 'dak examine-package' properly + + o queued and/or perl on sparc stable sucks - reimplement it. + + o aj's bin nmu changes + + o 'dak process-new': + * priority >> optional + * arch != {any,all} + * build-depends wrong (via 'dak compare-suites') + * suid + * conflicts + * notification/stats to admin daily + o trap 'dak examine-package' exiting + o distinguish binary only versus others (neuro) + + o cache changes parsed from ordering (careful tho: would be caching + .changes from world writable incoming, not holding) + + o dak doesn't recognise binonlyNMUs correctly in terms of telling + who their source is; source-must-exist does, but the info is not + propogated down. + + o Fix BTS vs. dak sync issues by queueing(via BSMTP) BTS mail so + that it can be released on deman (e.g. ETRN to exim). + + o maintainers file needs overrides + + [ change override.maintainer to override.maintainer-from + + override.maintainer-to and have them reference the maintainers + table. Then fix 'dak make-maintainers' to use them and write some scripting + to handle the Santiago situation. ] + + o Validate Depends (et al.) [it should match \(\s*(<<|<|<=|=|>=|>|>>)\s*\)] + + o Clean up DONE; archive to tar file every 2 weeks, update tar tvzf INDEX file. + + o testing-updates suite: if binary-only and version << version in + unstable and source-ver ~= source-ver in testing; then map + unstable -> testing-updates ? + + o hooks or configurability for debian specific checks (e.g. check_urgency, auto-building support) + + o morgue needs auto-cleaning (?) + + o dak stats: two modes, all included, seperate + o dak stats: add non-US + o dak stats: add ability to control components, architectures, archives, suites + o dak stats: add key to expand header + +================================================================================ + +queue/approved +-------------- + + o What to do with multi-suite uploads? Presumably hold in unapproved + and warn? Or what? Can't accept just for unstable or reject just + from stable. + + o Whenever we check for anything in accepted we also need to check in + unapproved. + + o non-sourceful uploads should go straight through if they have + source in accepted or the archive. + + o security uploads on auric should be pre-approved. + +================================================================================ + +Less Urgent +----------- + + o change utils.copy to try rename() first + + o [hard, long term] unchecked -> accepted should go into the db, not + a suite, but similar. this would allow dak to get even faster, + make 'dak ls' more useful, decomplexify specialacceptedautobuild + and generally be more sane. may even be helpful to have e.g. new + in the DB, so that we avoid corner cases like the .orig.tar.gz + disappearing 'cos the package has been entirely removed but was + still on stayofexecution when it entered new. + + o Logging [mostly done] (todo: 'dak clean-suites' (hard), .. ?) + + o 'dak process-unchecked': the tar extractor class doesn't need to be redone for each package + + o reverse of source-must-exist; i.e. binary-for-source-must-not-exist + o REJECT reminders in 'dak clean-queues'. + o 'dak examine-package' should check for conflicts and warn about them visavis priority [rmurray] + o store a list of removed/files versions; also compare against them. + [but be careful about scalability] + + o dak examine-package: print_copyright should be a lot more intelligent + @ handle copyright.gz + @ handle copyright.ja and copyright + @ handle (detect at least) symlinks to another package's doc directory + @ handle and/or fall back on source files (?) + + o To incorporate from utils: + @ unreject + + o auto-purge out-of-date stuff from non-free/contrib so that testing and stuff works + o doogie's binary -> source index + o jt's web stuff, matt's changelog stuff (overlap) + + o [Hard] Need to merge non-non-US and non-US DBs. + + o experimental needs to auto clean (relative to unstable) [partial: + 'dak cruft-report' warns about this] + + o Do a checkpc(1)-a-like which sanitizes a config files. + o fix parse_changes()/build_file_list() to sanity check filenames + o saftey check and/or rename debs so they match what they should be + + o Improve 'dak compare-suites'. + o Need to optimize all the queries by using EXAMINE and building some INDEXs. + [postgresql 7.2 will help here] + o Need to enclose all the setting SQL stuff in transactions (mostly done). + o Need to finish 'dak init-db' (a way to sync dak.conf and the DB) + o Need the ability to rebuild all other tables from dists _or_ pools (in the event of disaster) (?) + o Make the --help and --version options do stuff for all scripts + + o 'dak make-maintainers' can't handle whitespace-only lines (for the moment, this is feature) + + o generic way of saying isabinary and isadsc. (?) + + o s/distribution/suite/g + + o cron.weekly: + @ weekly postins to d-c (?) + @ backup of report (?) + @ backup of changes.tgz (?) + + o --help doesn't work without /etc/dak/dak.conf (or similar) at + least existing. + + o rename 'dak compare-suites' (clashes with existing 'dak compare-suites')... + + * Harder: + + o interrupting of stracing 'dak process-unchecked' causes exceptions errors from apt_inst calls + o dependency checking (esp. stable) (partially done) + o override checks sucks; it needs to track changes made by the + maintainer and pass them onto ftpmaster instead of warning the + maintainer. + o need to do proper rfc822 escaping of from lines (as opposed to s/\.//g) + o Revisit linking of binary->source in install() in dak. + o Fix component handling in overrides (aj) + o Fix lack of entires in source overrides (aj) + o direport misreports things as section 'devel' (? we don't use direport) + o vrfy check of every Maintainer+Changed-By address; valid for 3 months. + o binary-all should be done on a per-source, per-architecture package + basis to avoid, e.g. the perl-modules problem. + o a source-missing-diff check: if the version has a - in it, and it + is sourceful, it needs orig and diff, e.g. if someone uploads + esound_0.2.22-6, and it is sourceful, and there is no diff -> + REJECT (version has a dash, therefore not debian native.) + o check linking of .tar.gz's to .dsc's.. see proftpd 1.2.1 as an example + o archive needs md5sum'ed regularly, but takes too long to do all + in one go; make progressive or weekly. + o something needs to clear out .changes files from p-u when + removing stuff superseded by newer versions. [but for now we have + 'dak clean-proposed-updates'] + o test sig checking stuff in test/ (stupid thing is not modularized due to global abuse) + o when encountering suspicous things (e.g. file tainting) do something more drastic + + * Easy: + + o suite mapping and component mapping are parsed per changes file, + they should probably be stored in a dictionary created at startup. + o don't stat/md5sum files you have entries for in the DB, moron + boy (Dak.check_source_blah_blah) + o promote changes["changes"] to mandatory in dak.py(dump_vars) + after a month or so (or all .dak files contain in the queue + contain it). + o 'dak rm' should behave better with -a and without -b; see + gcc-defaults removal for an example. + o Reject on misconfigured kernel-package uploads + o utils.extract_component_from_section: main/utils -> main/utils, main rather than utils, main + o Fix 'dak process-unchecked' to warn if run when not in incoming or p-u + o dak should validate multi-suite uploads; only possible valid one + is "stable unstable" + o cron.daily* should change umask (aj sucks) + o 'dak cruft-report' doesn't look at debian-installer but should. + o 'dak cruft-report' needs to check for binary-less source packages. + o 'dak cruft-report' could accept a suite argument (?) + o byhand stuff should send notification + o 'dak poolize' should udpate db; move files, not the other way around [neuro] + o 'dak rm' should update the stable changelog [joey] + o update tagdb.dia + + * Bizzare/uncertain: + + o drop rather dubious currval stuff (?) + o rationalize os.path.join() usage + o 'dak cruft-report' also doesn't seem to warn about missing binary packages (??) + o logging: hostname + pid ? + o ANAIS should be done in dak (?) + o Add an 'add' ability to 'dak rm' (? separate prog maybe) + o Replicate old dinstall report stuff (? needed ?) + o Handle the case of 1:1.1 which would overwrite 1.1 (?) + o maybe drop -r/--regex in 'dak ls', make it the default and + implement -e/--exact (a la joey's "elmo") + o dsc files are not checked for existence/perms (only an issue if + they're in the .dsc, but not the .changes.. possible?) + + * Cleanups & misc: + + o db_access' get_files needs to use exceptions not this None, > 0, < 0 return val BS (?) + o The untouchable flag doesn't stop new packages being added to ``untouchable'' suites + +================================================================================ + +Packaging +--------- + + o Fix stuff to look in sensible places for libs and config file in debian package (?) + +================================================================================ + + --help manpage +----------------------------------------------------------------------------- +check-archive X +check-overrides X X +check-proposed-updates X +clean-proposed-updates X +clean-queues X +clean-suites X X +compare-suites X +control-overrides X X +control-suite X X +cruft-report X +decode-dot-dak X +examine-package X +generate-releases X +import-archive X +import-users-from-passwd X X +init-db X +init-dirs X +ls X X +make-maintainers X X +make-overrides X +make-suite-file-list X +poolize X X +process-accepted X X +process-new X X +process-unchecked X +queue-report X +rm X X +security-install X +stats X +symlink-dists X + + +================================================================================ + +Random useful-at-some-point SQL +------------------------------- + +UPDATE files SET last_used = '1980-01-01' + FROM binaries WHERE binaries.architecture = + AND binaries.file = files.id; + +DELETE FROM bin_associations + WHERE EXISTS (SELECT id FROM binaries + WHERE architecture = + AND id = bin_associations.bin); + +================================================================================