New.

[dak.git] / docs / README.new-incoming

diff --git a/docs/README.new-incoming b/docs/README.new-incoming
new file mode 100644 (file)
index 0000000..1581bb7
--- /dev/null
+++ b/docs/README.new-incoming
@@ -0,0 +1,132 @@
+
+                     Proposed New Incoming System
+                    ============================
+
+This document outlines the proposed new system for handling the
+Incoming directories on ftp-master and non-US.
+
+The present:
+------------
+
+  o incoming is a world writable directory
+
+  o incoming is available to all through http://incoming.debian.org/
+
+  o incoming is processed once a day by dinstall
+
+  o uploads in incoming must have been there > 24 hours before they
+    are REJECTed.  If they are processed before that and have problems
+    they will be SKIPped (with no notification to the maintainer and/or
+    uploader).
+
+The proposed future:
+--------------------
+
+  o There will be 4 incoming directories:
+
+     @ "unchecked"  - where uploads from Queue Daemons and maintainers
+                     initially go
+
+     @ "install"    - where installable packages stay until the daily
+                      dinstall run
+
+     @ "new"       - where NEW packages (and their dependents[1]) requiring
+                     human processing go after being automatically
+                     checked by dinstall.
+
+     @ "byhand"            - where BYHAND packages (and their dependents[1])
+                      requiring human intervention go after being
+                      automatically checked by dinstall.
+
+    In addition there will be 3 support directories:
+
+     @ "reject"            - where rejected uploads go
+
+     @ "done"      - where the .changes files for packages that have been
+                     installed go.
+
+     @ "holding"    - a temporary working area for dinstall to hold
+                     packages while checking them.
+
+  o Packages in 'unchecked' are automatically checked every 15 minutes
+    and are either: REJECT, ACCEPT (i.e. -> 'install'), NEW or BYHAND.
+
+  o Only 'unchecked' is locally world-writeable.  The others are all,
+    of course, locally world-readable but only 'install' and 'byhand'
+    are publicly visible on http://incoming.debian.org/
+
+  o 'install' and 'byhand' are made available to the auto-builders so
+     they can build out of them.
+
+  o 'install' is processed once a day as before.
+
+  o list notification and bug closures are changed to be done for
+    ACCEPTs, not INSTALLs. Mail is sent only to the maintainer/uploader
+    on INSTALL.
+    [Rationale: this reduces the load both on our list server and our
+     BTS server; it also gives people better notice of uploads to
+     avoid duplication of work especially, for example, in the case of NMUs]
+    [NB: see [3] for clarifications of when ACCEPT/INSTALL mails are sent]
+
+Why:
+----
+
+  o Security (no more replaceable file races)
+  o Integrity (new http://i.d.o contains only signed (+installable) uploads[2])
+  o Needed for crypto-in-main integration
+  o Allows safe auto-building out of incoming
+  o Allows previously-prohibitively-expensive checks to be added to dinstall
+  o Much faster feedback on packages; no more 48 hour waits before
+    finding out your package has been REJECTed.
+
+What breaks:
+------------
+
+  o uploads of large packages directly to incoming over a slow link
+    can lead to bogus rejections.
+
+    * solution: Ensure the .changes file is the last file to be
+                uploaded (dput and dupload already do this) or upload
+                to a temporary directory then mv them into place with
+                ssh.
+
+  o people who upload packages but then want to retract or replace the
+    upload.
+
+    * solution: mostly "Don't do that then"; i.e. test your uploads
+      properly.  Uploads can still be replaced, simply by uploading a
+      higher versioned replacement.  Total retraction is harder but
+      usually only relevant for NEW packages.
+
+================================================================================
+
+[1] For versions of dependents meaning: binaries compiled from the
+    source of BYHAND or NEW uploads.  Due to katie's fascist
+    source-must-exist checking, these binaries must be held back until
+    the BYHAND/NEW uploads are processed.
+
+[2] When this was initially written there was still at least one
+    upload queue which will accept unsigned uploads from any
+    source. [I've since discover it's been deactivated, but not, AFAIK
+    because it allowed unsigned uploads.]
+
+[3]
+             --> reject
+            /
+           /
+unchecked  ---------------------------[*]-------> install ------[+]------> pool
+           \               ^   ^
+            |             /   /
+            |-->   new  --   /
+            |       |[4]    /
+            |       V      /
+            |--> byhand --/
+
+[*] is ACCEPT and when list notification and bug closure occurs
+[+] is INSTALL and when maintainer/uploader notification occurs
+
+[4] This is a corner case, included for completeness, ignore
+    it. [Boring details: NEW trumps BYHAND, so it's possible for a
+    upload with both BYHAND and NEW components to go from 'unchecked'
+    -> 'new' -> 'byhand' -> 'install']
+