[An updated version of the proposal sent to debian-devel-announce@l.d.o.  
 Debian-specific, but useful as a general overview of New Incoming.]

                     New Incoming System
		     ===================

This document outlines the new system for handling Incoming
directories on ftp-master and non-US.

The old system:
---------------

  o incoming was a world writable directory

  o incoming was available to everyone through http://incoming.debian.org/

  o incoming was processed once a day by dinstall

  o uploads in incoming had to have been there > 24 hours before they
    were REJECTed.  If they were processed before that and had
    problems they were SKIPped (with no notification to the maintainer
    and/or uploader).

The new system:
---------------

  o There's 4 incoming directories:

     @ "unchecked"  - where uploads from Queue Daemons and maintainers
		      initially go.

     @ "accepted"   - where accepted packages stay until the daily
                      dinstall run.

     @ "new"	    - where NEW packages (and their dependents[1]) requiring
		      human processing go after being automatically
		      checked by dinstall.

     @ "byhand"	    - where BYHAND packages (and their dependents[1])
                      requiring human intervention go after being
                      automatically checked by dinstall.

    In addition there's 3 support directories:

     @ "reject"	    - where rejected uploads go

     @ "done"	    - where the .changes files for packages that have been
		      installed go.

     @ "holding"    - a temporary working area for dinstall to hold
		      packages while checking them.

  o Packages in 'unchecked' are automatically checked every 15 minutes
    and are either: REJECT, ACCEPT, NEW or BYHAND.

  o Only 'unchecked' is locally world-writeable.  The others are all,
    of course, locally world-readable but only 'accepted' and 'byhand'
    are publicly visible on http://incoming.debian.org/

  o 'accepted' and 'byhand' are made available to the auto-builders so
     they can build out of them.

  o 'accepted' is processed once a day as before.

  o Maintainer/uploader & list notification and bug closures are
    changed to be done for ACCEPTs, not INSTALLs. 
    [Rationale: this reduces the load both on our list server and our
     BTS server; it also gives people better notice of uploads to
     avoid duplication of work especially, for example, in the case of
     NMUs.]
    [NB: see [3] for clarifications of when mails are sent.]

Why:
----

  o Security (no more replaceable file races)
  o Integrity (new http://i.d.o contains only signed (+installable) uploads[2])
  o Needed for crypto-in-main integration
  o Allows safe auto-building out of accepted
  o Allows previously-prohibitively-expensive checks to be added to dinstall
  o Much faster feedback on packages; no more 48 hour waits before
    finding out your package has been REJECTed.

What breaks:
------------

  o people who upload packages but then want to retract or replace the
    upload.

    * solution: mostly "Don't do that then"; i.e. test your uploads
      properly.  Uploads can still be replaced, simply by uploading a
      higher versioned replacement.  Total retraction is harder but
      usually only relevant for NEW packages.

================================================================================

[1] For versions of dependents meaning: binaries compiled from the
    source of BYHAND or NEW uploads.  Due to dak's fascist
    source-must-exist checking, these binaries must be held back until
    the BYHAND/NEW uploads are processed.

[2] When this mail was initially written there was still at least one
    upload queue which will accept unsigned uploads from any
    source. [I've since discovered it's been deactivated, but not,
    AFAIK because it allowed unsigned uploads.]

[3]
             --> reject
            /
           /
unchecked  -----------------------------[*]------> accepted ---------------> pool
           \               ^   ^
            |             /   /
            |-->   new  --   /
            |       |[4]    /
            |       V      /
            |--> byhand --/

[4] This is a corner case, included for completeness, ignore
    it. [Boring details: NEW trumps BYHAND, so it's possible for a
    upload with both BYHAND and NEW components to go from 'unchecked'
    -> 'new' -> 'byhand' -> 'accepted']