Proposed New Incoming System
		     ============================

This document outlines the proposed new system for handling the
Incoming directories on ftp-master and non-US.

The present:
------------

  o incoming is a world writable directory

  o incoming is available to all through http://incoming.debian.org/

  o incoming is processed once a day by dinstall

  o uploads in incoming must have been there > 24 hours before they
    are REJECTed.  If they are processed before that and have problems
    they will be SKIPped (with no notification to the maintainer and/or
    uploader).

The proposed future:
--------------------

  o There will be 4 incoming directories:

     @ "unchecked"  - where uploads from Queue Daemons and maintainers
		      initially go

     @ "install"    - where installable packages stay until the daily
                      dinstall run

     @ "new"	    - where NEW packages (and their dependents[1]) requiring
		      human processing go after being automatically
		      checked by dinstall.

     @ "byhand"	    - where BYHAND packages (and their dependents[1])
                      requiring human intervention go after being
                      automatically checked by dinstall.

    In addition there will be 3 support directories:

     @ "reject"	    - where rejected uploads go

     @ "done"	    - where the .changes files for packages that have been
		      installed go.

     @ "holding"    - a temporary working area for dinstall to hold
		      packages while checking them.

  o Packages in 'unchecked' are automatically checked every 15 minutes
    and are either: REJECT, ACCEPT (i.e. -> 'install'), NEW or BYHAND.

  o Only 'unchecked' is locally world-writeable.  The others are all,
    of course, locally world-readable but only 'install' and 'byhand'
    are publicly visible on http://incoming.debian.org/

  o 'install' and 'byhand' are made available to the auto-builders so
     they can build out of them.

  o 'install' is processed once a day as before.

  o list notification and bug closures are changed to be done for
    ACCEPTs, not INSTALLs. Mail is sent only to the maintainer/uploader
    on INSTALL.
    [Rationale: this reduces the load both on our list server and our
     BTS server; it also gives people better notice of uploads to
     avoid duplication of work especially, for example, in the case of NMUs]
    [NB: see [3] for clarifications of when ACCEPT/INSTALL mails are sent]

Why:
----

  o Security (no more replaceable file races)
  o Integrity (new http://i.d.o contains only signed (+installable) uploads[2])
  o Needed for crypto-in-main integration
  o Allows safe auto-building out of incoming
  o Allows previously-prohibitively-expensive checks to be added to dinstall
  o Much faster feedback on packages; no more 48 hour waits before
    finding out your package has been REJECTed.

What breaks:
------------

  o uploads of large packages directly to incoming over a slow link
    can lead to bogus rejections.

    * solution: Ensure the .changes file is the last file to be
                uploaded (dput and dupload already do this) or upload
                to a temporary directory then mv them into place with
                ssh.

  o people who upload packages but then want to retract or replace the
    upload.

    * solution: mostly "Don't do that then"; i.e. test your uploads
      properly.  Uploads can still be replaced, simply by uploading a
      higher versioned replacement.  Total retraction is harder but
      usually only relevant for NEW packages.

================================================================================

[1] For versions of dependents meaning: binaries compiled from the
    source of BYHAND or NEW uploads.  Due to katie's fascist
    source-must-exist checking, these binaries must be held back until
    the BYHAND/NEW uploads are processed.

[2] When this was initially written there was still at least one
    upload queue which will accept unsigned uploads from any
    source. [I've since discover it's been deactivated, but not, AFAIK
    because it allowed unsigned uploads.]

[3]
             --> reject
            /
           /
unchecked  ---------------------------[*]-------> install ------[+]------> pool
           \               ^   ^
            |             /   /
            |-->   new  --   /
            |       |[4]    /
            |       V      /
            |--> byhand --/

[*] is ACCEPT and when list notification and bug closure occurs
[+] is INSTALL and when maintainer/uploader notification occurs

[4] This is a corner case, included for completeness, ignore
    it. [Boring details: NEW trumps BYHAND, so it's possible for a
    upload with both BYHAND and NEW components to go from 'unchecked'
    -> 'new' -> 'byhand' -> 'install']