1 debianqueued -- daemon for managing Debian upload queues
2 ========================================================
4 Copyright (C) 1997 Roman Hodek <Roman.Hodek@informatik.uni-erlangen.de>
5 $Id: README,v 1.20 1999/07/08 09:35:37 ftplinux Exp $
8 Copyright and Disclaimer
9 ------------------------
11 This program is free software. You can redistribute it and/or
12 modify it under the terms of the GNU General Public License as
13 published by the Free Software Foundation: either version 2 or
14 (at your option) any later version.
16 This program comes with ABSOLUTELY NO WARRANTY!
18 You're free to modify this program at your will, according to the GPL,
19 and I don't object if you modify the program. But it would be nice if
20 you could send me back such changes if they could be of public
21 interest. I will try to integrate them into the mainstream version
28 debianqueued has been written for running a new Debian upload queue at
29 ftp.uni-erlangen.de, but I tried to keep it as general as possible and
30 it should be useable for other sites, too. If there should be
31 non-portabilities, tell me about them and we'll try to get them fixed!
33 Before installing debianqueued, you should have the following
36 - pgp (needed for checking signatures)
38 - ssh & Co. (but not necessarily sshd, only client programs used)
40 - md5sum (for checking file integrity)
42 - mkfifo (for creating the status FIFO)
48 - ar (for analyzing .deb files)
50 The daemon needs a directory of its own where the scripts reside and
51 where it can put certain files. This directory is called $queued_dir
52 in the Perl scripts and below. There are no special requirements where
53 in the filesystem hierarchy this directory should be.
55 All configurations are done in file 'config' in $queued_dir. For
56 security reasons, the $queued_dir should not be in a public FTP area,
57 and should be writeable (as the files in it) only for the user
58 maintaining the local debianqueued.
60 The file Queue.README and Queue.message in the distribution archive
61 are examples for README and .message files to put into the queue
62 directory. Modify them as you like, or don't install them if you
69 debianqueued is intended to run all time, not as a cron job.
70 Unfortunately, you can't start it at system boot time automatically,
71 because a human has to type in the pass phrase for the ssh key. So you
72 have to start the daemon manually.
74 The daemon can be stopped by simply killing it (with SIGTERM
75 preferrably). SIGTERM and SIGINT are blocked during some operations,
76 where it could leave files in a inconsistent state. So it make take
77 some time until the daemon really dies. If you have the urgent need
78 that it goes away immediately, use SIGQUIT. Please don't use SIGKILL
79 except unavoidable, because the daemon can't clean up after this
82 For your convenience, the daemon can kill and restart itself. If you
83 start debianqueued with a "-k" argument, it tries to kill a running
84 daemon (and it complains if none is running.) If "-r" is on the
85 command line, it tries to kill a running daemon first if there is one.
86 (If not, it starts anyway, but prints a little warning.) If a daemon
87 is running and a new one is started without "-r", you get an error
88 message about this. This is to protect you from restarting the daemon
91 The other script, dqueued-watcher, is intended as cron job, and it
92 watches that the daemon is running, in case that it should crash
93 sometimes. It also takes care of updating the Debian keyring files if
94 necessary. You should enter it e.g. like
96 0,30 * * * * .../dqueued-watcher
98 into your crontab. (Assuming you want to run it every 30 minutes,
99 which seems a good compromise.)
101 Both scripts (debianqueued and dqueued-watcher) need no special
102 priviledges and thus can be run as an ordinary user (not root). You
103 can create an own user for debianqueued (e.g. "dqueue"), but you need
104 not. The only difference could be which ssh key is used for connects
105 to the target host. But you can configure the file to take the ssh key
106 from in the config file.
112 The config file, $queued_dir/config, is plain Perl code and is
113 included by debianqueued and dqueued-watcher. You can set the
114 following variables there:
117 Non-zero values enable debugging output (to log file).
119 The following are all programs that debianqueued calls. You should
120 always use absolute pathnames!
122 - $pgp, $ssh, $scp, $ssh_agent, $ssh_add, $md5sum, $mail, $mkfifo,
127 o $mail should support the -s option for supplying a subject.
128 Therefore choose mailx if your mail doesn't know -s.
130 o $tar should be GNU tar, several GNU features are used (e.g.
131 --use-compress-program).
133 o $ar must be able to unpack *.deb files and must understand the
134 'p' command. Better check this first... If you don't define $ar
135 (or define it to be empty), debianqueued won't be able to
136 extract a maintainer address from .deb files. (Which isn't that
141 All binaries listed in this variable are tested to be present
142 before each queue run. If any is not available, the queue run is
143 delayed. This test can be useful if those binaries reside on NFS
144 filesystems which may be (auto-)mounted only slowly. It is
145 specially annoying for users if pgp can't be found and a .changes
149 Options passed to ssh and scp on every call. General ssh
150 configuration should be done here and not in ~/.ssh/config, to
151 avoid dependency on the user's settings. A good idea for
152 $ssh_options seems to be
154 -o'BatchMode yes' -o'FallBackToRsh no' -o'ForwardAgent no'
155 -o'ForwardX11 no' -o'PasswordAuthentication no'
156 -o'StrictHostKeyChecking yes'
159 The file containing the ssh key you want the daemon to use for
160 connects to the target host. If you leave this empty, the default
161 ~/.ssh/identity is used, which may or may not be what you want.
164 This names the queue directory itself. Probably it will be inside
165 the public FTP area. Don't forget to allow uploads to it in
166 ftpaccess if you're using wu-ftpd.
168 Maybe you should also allow anonymous users to rename files in that
169 directory, to fix upload problems (they can't delete files, so they
170 have to move the errorneous file out of the way). But this
171 introduces a denial-of-service security hole, that an attacker
172 renames files of other people and then a job won't be done. But at
173 least the data aren't lost, and the rename command probably was
174 logged by ftpd. Nevertheless, there's no urgent need to allow
175 renamings, because the queue daemon deletes all bad files
176 automatically, so they can be reuploaded under the same name.
177 Decide on your own...
180 This is a regular expression for files that never should be deleted
181 in the queue directory. The status file must be included here,
182 other probable candicates are .message and/or README files.
185 If this variable is true (i.e., not 0 or ""), all files belonging
186 to a job are changed to mode 644 only on the target host. The
187 alternative (if the variable is false, i.e. 0) is to change the
188 mode already locally, after the sizes and md5 sums have been
189 verified. The latter is the default.
191 The background for this is the following: The files must be
192 word-readable on master for dinstall to work, so they must be at
193 least mode 444, but 644 seems more useful. If the upload policy of
194 your site says that uploaded files shouldn't be readable for world,
195 the queue daemon has to change the permission at some point of
196 time. (scp copies a file's permissions just as the contents, so
197 after scp, the files on the target have the same mode as in the
198 queue directory.) If the files in the queue are mode 644 anyway,
199 you don't need to care about this option. The default --to give
200 word read permission in the queue already after some checks-- is
201 obviously less restrictive, but might be against the policy of your
202 site. The alternative keeps the files unreadable in the queue in
203 any case, and they'll be readable only on the target host.
206 This is the name of the status file or FIFO, through which users
207 can ask the daemon what it's currently doing. It should normally be
208 in the queue directory. If you change the name, please don't forget
209 to check $keep_files. See also the own section on the status file.
211 If you leave $statusfile empty, the daemon doesn't create and
212 manage a status file at all, if you don't want it. Unfortunately,
213 dqueued-watcher's algorithm to determine whether it already has
214 reported a missing daemon depends on the status file, so this
215 doesn't work anymore in this case. You'll get dead daemon mails on
216 every run of dqueued-watcher.
219 If this number is greater than 0, the status file is implemented as
220 a regular file, and updated at least every $statusdelay seconds. If
221 $statusdelay is 0, the FIFO implementation is used (see status file
225 The name of the PGP keyring the daemon uses to check PGP signatures
226 of .changes files. This is usually $queued_dir/debian-keyring.pgp.
227 It should contain exactly the keys of all Debian developers (i.e.
228 those and no other keys).
231 The name of the GnuPG keyring. The daemon now alternatively accepts
232 GnuPG signatures on .changes and .commands files. The value here is
233 usually $queued_dir/debian-keyring.gpg. It should contain only keys
234 of Debian developers (but not all developers have a GPG key
238 Path of the debian-keyring.tar.gz file inside a Debian mirror. The
239 file is "/debian/doc/debian-keyring.tar.gz" on ftp.debian.org,
240 don't know where you mirror it to... Leave it empty if you don't
241 have that file on your local machine. But then you'll have to
242 update the keyring manually from time to time.
244 - $keyring_archive_name:
245 Name of the PGP keyring file in the archive $keyring_archive. Currently
246 "debian-keyring*/debian-keyring.pgp".
248 - $gpg_keyring_archive_name:
249 Name of the GnuPG keyring file in the archive $keyring_archive. Currently
250 "debian-keyring*/debian-keyring.gpg".
253 The file debianqueued writes its logging data to. Usually "log" in
257 The file debianqueued writes its pid to. Usually "pid" in
261 Name of the target host, i.e. the host where the queue uploads to.
262 Usually "master.debian.org". (Ignored with "copy" upload method.)
265 The login on the target to use for uploads. (Ignored with "copy"
266 and "ftp" upload methods; "ftp" always does anonymous logins.)
269 The directory on the target to where files should be uploaded. On
270 master.debian.org this currently is
271 "/home/Debian/ftp/private/project/Incoming".
273 - $max_upload_retries:
274 This is the number how often the daemon tries to upload a job (a
275 .changes + the files belonging to it). After that number is
276 exhausted, all these files are deleted.
279 This is how many days are waited before logfiles are rotated. (The
280 age of the current log files is derived from the first date found
284 How many old log files to keep. The current logfile is what you
285 configured as $logfile above, older versions have ".0", ".1.gz",
286 ".2.gz", ... appended. I.e., all old versions except the first are
287 additionally gzipped. $log_keep is one higher than the max.
288 appended number that should exist.
291 If this is set to a true value (not 0 and not ""), dqueued-watcher
292 will send a mail with a summary of the daemon's acivities whenever
293 logfiles are rotated.
296 If that value is a file name (and not an empty string),
297 dqueued-watcher will write the same summary of daemon activities as
298 above to the named file. This can be in addition to sending a mail.
301 This is a (Perl) list of names of packages that must be uploaded to
302 nonus.debian.org and not to master. Since the queue daemon only can
303 deal with one target, it can't do that upload and thus must reject
304 the job. Generally you can treat this variable as a list of any
305 packages that should be rejected.
307 All the following timing variables are in seconds:
310 The time between the first (failed) upload try and the next one.
311 Usually shorter then $upload_delay_2 for quick retry after
315 The time between the following (except the first) upload retries.
318 The time between two queue runs. (May not be obeyed too exactly...
319 a few seconds deviation are normal).
321 - $stray_remove_timeout:
322 If a file not associated with any .changes file is found in the
323 queue directory, it is removed after this many seconds.
325 - $problem_report_timeout:
326 If there are problems with a job that could also be result of a
327 not-yet-complete upload (missing or too small files), the daemon
328 waits this long before reporting the problem to the uploader. This
329 avoids warning mails for slow but ongoing uploads.
331 - $no_changes_timeout:
333 If files are found in the queue directory that look like a Debian
334 upload (*.tar.gz, *.diff.gz, *.deb, or *.dsc files), but aren't
335 accompanied by a .changes file, then debianqueued tries to notify
336 the uploader after $no_changes_timeout seconds about this. This
337 value is somewhat similar to $problem_report_timeout, and the
340 Since there's no .changes, the daemon can't never be sure who
341 really uploaded the files, but it tries to extract the maintainer
342 address from all of the files mentioned above. If they're real
343 Debian files (except a .orig.tar.gz), this works in most cases.
345 - $bad_changes_timeout:
346 After this time, a job with persisting problems (missing files,
347 wrong size or md5 checksum) is removed.
350 This is the maximum time a remote command (ssh/scp) may take. It's
351 to protect against network unreliabilities and the like. Choose the
352 number sufficiently high, so that the timeout doesn't inadventedly
353 kill a longish upload. A few hours seems ok.
355 Contents of $queued_dir
356 -----------------------
358 $queued_dir contains usually the following files:
361 The configuration file, described above.
364 Log file of debianqueued. All interesting actions and errors are
365 logged there, in a format similar to syslog.
368 This file contains the pid of debianqueued, to detect double
369 daemons and for killing a running daemon.
371 - debian-keyring.pgp, debian-keyring.gpg:
372 These are the PGP and GnuPG key rings used by debianqueued to
373 verify the signatures of .changes files. It should contain the keys
374 of all Debian developers and no other keys. The current Debian key
375 ring can be obtained from
376 ftp.debian.org:/debian/doc/debian-keyring.tar.gz. dqueued-watcher
377 supports the facility to update this file automatically if you also
380 - debianqueued, dqueued-watcher:
383 All filenames except "config" can be changed in the config file. The
384 files are not really required to reside in $queued_dir, but it seems
385 practical to have them all together...
388 Details of Queue Processing
389 ---------------------------
391 The details of how the files in the queue are processed may be a bit
392 complicated. You can skip this section if you're not interested in
393 those details and everything is running fine... :-)
395 The first thing the daemon does on every queue run is determining all
396 the *.changes files present. All of them are subsequently read and
397 analyzed. The .changes MUST contain a Maintainer: field, and the
398 contents of that field should be the mail address of the uploader. The
399 address is used for sending back acknowledges and error messages.
400 (dinstall on master uses the same convention.)
402 Next, the PGP or GnuPG signature of the .changes is checked. The
403 signature must be valid and must belong to one of the keys in the
404 Debian keyring (see config variables $keyring and $gpg_keyring). This
405 ensures that only registered Debian developers can use the upload
406 queue to transfer files to master.
408 Then all files mentioned in the Files: field of the .changes are
409 checked. All of them must be present, and must have correct size and
410 md5 checksum. If any of this conditions is violated, the upload
411 doesn't happen and an error message is sent to the uploader. If the
412 error is a incorrect size/md5sum, the file is also deleted, because it
413 has to be reuploaded anyway, and it could be the case that the
414 uploader cannot easily overwrite a file in the queue dir (due to
415 upload permission restrictions). If the error is a missing file or a
416 too small file, the error message is hold back for some time
417 ($problems_report_timeout), because they can also be result of an
418 not-yet-complete upload.
420 The time baseline for when to send such a problem report is the
421 maximum modification time of the .changes itself and all files
422 mentioned in it. When such a report is sent, the setgid bit (show as
423 'S' in ls -l listing, in group x position) on the .changes is set to
424 note that fact, and to avoid the report being sent on every following
425 queue run. If any modification time becomes greater than the time the
426 setgid bit was set, a new problem report is sent, because obviously
427 something has changed to the files.
429 If a job is hanging around for too long with errors
430 ($bad_changes_timeout), the .changes and all its files are deleted.
431 The base for that timeout is again the maximum modification time as
434 If now the .changes itself and all its files are ok, an upload is
435 tried. The upload itself is done with scp. In that stage, various
436 errors from the net and/or ssh can occur. All these simply count as
437 upload failures, since it's not easy to distinguish transient and
438 permanent failures :-( If the scp goes ok, the md5sums of the files on
439 the target are compared with the local ones. This is to ensure that
440 the transfer didn't corrupt anything. On any error in the upload or in
441 the md5 check, the files written to the target host are deleted again
442 (they may be broken), and an error message is sent to the uploader.
444 The upload is retied $upload_delay_1 seconds later. If it fails again,
445 the next retries have a (longer) delay $upload_delay_2 between them.
446 At most $max_upload_retries retries are done. After all these failed,
447 all the files are deleted, since it seems we can't move them... For
448 remembering how many tries were alredy done (and when), debianqueued
449 uses a separate file. Its name is the .changes' filename with
450 ".failures" appended. It contains simply two integers, the retry count
451 and the last upload time (in Unix time format).
453 After a successfull upload, the daemon also checks for files that look
454 like they belonged to the same job, but weren't listed in the
455 .changes. Due to experience, this happens rather often with
456 .orig.tar.gz files, which people upload though they're aren't needed
457 nor mentioned in the .changes. The daemon uses the filename pattern
458 <pkg-name>_<version>* to find such unneeded files, where the Debian
459 revision is stripped from <version>. The latter is needed to include
460 .orig.tar.gz files, which don't have the Debian revision part. But
461 this also introduces the possibility that files of another upload for
462 the same package but with another revision are deleted though they
463 shouldn't. However, this case seems rather unlikely, so I didn't care
464 about it. If such files are deleted, that fact is mentioned in the
465 reply mail to the uploader.
467 If any files are found in the queue dir that don't belong to any
468 .changes, they are considered "stray". Such files are remove after
469 $stray_remove_timeout. This should be around 1 day or so, to avoid
470 files being removed that belong to a job, but whose .changes is still
471 to come. The daemon also tries to find out whether such stray files
472 could be part of an incomplete upload, where the .changes file is
473 still missing or has been forgotten. Files that match the patterns
474 *.deb, *.dsc, *.diff.gz, or *.tar.gz are analyzed whether a maintainer
475 address can be extracted from them. If yes, the maintainer is notified
476 about the incomplete upload after $no_changes_timeout seconds.
477 However, the maintainer needs not really be the uploader... It could
478 be a binary-only upload for another architecture, or a non-maintainer
479 upload. In these cases, the mail goes to the wrong wrong person :-(
480 But better than not writing at all, IMHO...
486 debianqueued provides a status file for the user in the queue
487 directory. By reading this file, the user can get an idea what the
488 daemon is currently doing.
490 There are two possible implementations of the status file: as a plain
491 file, or as a named pipe (FIFO). Both have their advantages and
494 If using the FIFO, the data printed (last ping time, next queue run)
495 are always up to date, because they're interrogated (by a signal) just
496 at the time the FIFO is opened for reading. Also, the daemon hasn't to
497 care about the status file if nobody accesses it. The bad things about
498 the FIFO: It is a potential portability problem, because not all
499 systems have FIFOs, or they behave different than I expect... But the
500 more severe problem: wu-ftpd refuses to send the contents of a FIFO on
501 a FTP GET request :-(( It does an explicit check whether a file to be
502 retrieved is a regular file. This can be easily patched [1], but not
503 everybody wants to do that or can do that (but I did it for
504 ftp.uni-erlangen.de). (BTW, there could still be problems (races) if
505 more than one process try to read the status file at the same time...)
507 The alternative is using a plain file, which is updated regularily by
508 the daemon. This works on every system, but causes more overhead (the
509 daemon has to wake up each $statusdelay seconds and write a file), and
510 the time figures in the file can't be exact. $statusdelay should be a
511 compromise between CPU wastage and desired accuracy of the times found
512 in the status file. I think 15 or 30 seconds should be ok, but your
515 If the status file is a FIFO, the queue daemon forks a second process
516 for watching the FIFO (so don't wonder if debianqueued shows up twice
517 in ps output :-), to avoid blocking a reading process too long until
518 the main daemon has time to watch the pipe. The status daemon requests
519 data from the main daemon by sending a signal (SIGUSR1). Nevertheless
520 it can happen that a process that opens the status file (for reading)
521 is blocked, because the daemon has crashed (or never has been started,
522 after reboot). To minimize chances for that situation, dqueued-watcher
523 replaces the FIFO by a plain file (telling that the daemon is down) if
524 it sees that no queue daemon is running.
527 [1]: This is such a patch, for wu-ftpd-2.4.2-BETA-13:
529 --- wu-ftpd/src/ftpd.c~ Wed Jul 9 13:18:44 1997
530 +++ wu-ftpd/src/ftpd.c Wed Jul 9 13:19:15 1997
531 @@ -1857,7 +1857,9 @@
535 - (fstat(fileno(fin), &st) < 0 || (st.st_mode & S_IFMT) != S_IFREG)) {
536 + (fstat(fileno(fin), &st) < 0 ||
537 + ((st.st_mode & S_IFMT) != S_IFREG &&
538 + (st.st_mode & S_IFMT) != S_IFIFO))) {
539 reply(550, "%s: not a plain file.", name);
547 The practical experiences with debianqueued showed that users
548 sometimes make errors with their uploads, resulting in misnamed or
549 corrupted files... Formerly they didn't have any chance to fix such
550 errors, because the ftpd usually doesn't allow deleting or renaming
551 files in the queue directory. (If you would allow this, *anybody* can
552 remove/rename files, which isn't desirable.) So users had to wait
553 until the daemon deleted the bad files (usually ~ 24 hours), before
554 they could start the next try.
556 To overcome this, I invented the *.command files. The daemon looks for
557 such files just as it tests for *.changes files on every queue run,
558 and processes them before the usual jobs. *.commands files must be PGP
559 or GnuPG signed by a known Debian developer (same test as for
560 *.changes), so only these people can give the daemon commands. Since
561 Debian developers can also delete files in master's incoming, the
562 *.commands feature doesn't give away any security.
564 The syntax of a *.commands file is much like a *.changes, but it
565 contains only two (mandatory) fields: Uploader: and Commands.
566 Uploader: contains the e-mail address of the uploader for reply mails,
567 and should have same contents as Maintainer: in a .changes. Commands:
568 is a multi-line field like e.g. Description: or Changes:. Every
569 continuation line must start with a space. Each line in Commands:
570 contains a command for the daemon that looks like a shell command (but
571 it isn't one, the daemon parses and executes it itself and doesn't use
572 sh or the respective binaries).
575 -----BEGIN PGP SIGNED MESSAGE-----
577 Uploader: Roman Hodek <Roman.Hodek@informatik.uni-erlangen.de>
579 rm hello_1.0-1_i386.deb
580 mv hello_1.0-1.dsx hello_1.0-1.dsc
582 -----BEGIN PGP SIGNATURE-----
585 iQCVAwUBNFiQSXVhJ0HiWnvJAQG58AP+IDJVeSWmDvzMUphScg1EK0mvChgnuD7h
586 BRiVQubXkB2DphLJW5UUSRnjw1iuFcYwH/lFpNpl7XP95LkLX3iFza9qItw4k2/q
587 tvylZkmIA9jxCyv/YB6zZCbHmbvUnL473eLRoxlnYZd3JFaCZMJ86B0Ph4GFNPAf
590 -----END PGP SIGNATURE-----
592 The only commands implemented at this time are 'rm' and 'mv'. No
593 options are implemented, and filenames may not contain slashes and are
594 interpreted relative to the queue directory. This ensures that only
595 files there can be modified. 'mv' always takes two arguments. 'rm' can
596 take any number of args. It also knows about the following shell
597 wildcard chars: *, ?, and [...]. {..,..} constructs are *not*
598 supported. The daemon expands these patterns itself and doesn't use sh
599 for that (for security reasons).
601 *.commands files are processed before the usual *.changes jobs, so if
602 a commands file fixes a job so that it can be processed, that
603 processing happens in the same queue run and no unnecessary delay is
606 The uploader of a *.commands will receive a reply mail with a comment
607 (OK or error message) to each of the commands given. The daemon not
608 only logs the contents of the Uploader: field, but also the owner of
609 the PGP/GnuPG key that was used to sign the file. In case you want to
610 find out who issued some commands, the Uploader: field is insecure,
611 since its contents can't be checked.
614 Security Considerations
615 -----------------------
617 You already know that debianqueued uses ssh & Co. to get access to
618 master, or in general any target host. You also probably know that you
619 need to unlock your ssh secret key with a passphrase before it can be
620 used. For the daemon this creates a problem: It needs the passphrase
621 to be able to use ssh/scp, but obviously you can't type in the phrase
622 every time the daemon needs it... It would also be very ugly and
623 insecure to write the passphase into some config file of the daemon!
625 The solution is using ssh-agent, which comes with the ssh package.
626 This agent's purpose is to store passphrases and give it to
627 ssh/scp/... if they need it. ssh-agent has to ways how it can be
628 accessed: through a Unix domain socket, or with an inherited file
629 descriptor (ssh-agent is the father of your login shell then). The
630 second method is much more secure than the first, because the socket
631 can be easily exploited by root. On the other hand, an inherited file
632 descriptor can be access *only* from a child process, so even root has
633 bad chances to get its hands on it. Unfortunately, the fd method has
634 been removed in ssh-1.2.17, so I STRONGLY recommend to use ssh-1.2.16.
635 (You can still have a newer version for normal use, but separate
636 binaries for debianqueued.) Also, using debianqueued with Unix domain
637 sockets is basically untested, though I've heard that it doesn't
640 debianqueued starts the ssh-agent automatically and runs ssh-add. This
641 will ask you for your passphrase. The phrase is stored in the agent
642 and available only to child processes of the agent. The agent will
643 also start up a second instance of the queue daemon that notices that
644 the agent is already running.
646 Currently, there's no method to store the passphrase in a file, due to
647 all the security disadvantages of this. If you don't mind this and
648 would like to have some opportunity to do it nevertheless, please ask
649 me. If there's enough demand, I'll do it.
655 Since release 0.9, debianqueued has two new upload methods as
656 alternatives to ssh: copy and ftp.
658 The copy method simply moves the files to another directory on the
659 same host. This seems a bit silly, but is for a special purpose: The
660 admins of master intend to run an upload queue there, too, in the
661 future to avoid non-anonymous FTP connections, which transmit the
662 password in cleartext. And, additionally to simply moving the files,
663 the queue daemon also checks the signature and integrity of uploads
664 and can reject non-US packages.
666 The ftp method uploads to a standard anon-FTP incoming directory. The
667 intention here is that you could create second-level queue daemons.
668 I.e., those daemons would upload into the queue of another daemon
669 (and, for example, this could be the queue of the daemon on master).
671 However, the ftp method still has some limitations:
673 1) Files in the target dir can't be deleted.
674 2) Uploaded files can't be verified as good as with the other methods.
675 3) $chmod_on_target often doesn't work.
676 4) The check for a writable incoming directory leaves temporary files
679 Ad 1): In anon-FTP incoming directories removing of files usually
680 isn't allowed (this would widely open doors to denial-of-service
681 attacks). But debianqueued has to remove files on the target as part
682 of handling upload errors. So if an transmission error happens during
683 a job, the bad file can't be deleted. On the next try, the file is
684 already present on the target and can't be overwritten, so all the
685 following tries will fail, too, except the upstream queue daemon has
686 deleted them already. And if the .changes was among the files already
687 (at least partially) uploaded, the daemon even will think that the
688 whole job is already present on the target and will delete the job in
691 Ad 2): Uploaded files are usually verified with md5sum if they're
692 really the same as the originals. But getting the md5sum for a file on
693 a FTP server usually isn't possible. It's currently handled as
694 follows: If the server supports a SITE MD5SUM command (non-standard!),
695 then this is used and you have the same checking quality. Otherwise,
696 debianqueued falls back to only comparing the file sizes. This is
697 better than nothing, but doesn't detected changed contents that don't
698 result in size changes.
700 Ad 3): Often SITE CHMOD (standard) isn't allowed in incoming
701 directories. If this is the case, $chmod_on_target must be off,
702 otherwise all uploads will fail. The mode of uploaded files if forced
703 anyway by the FTP server in most cases.
705 Ad 4): As you know, the queue daemon has a special check if the target
706 directory is writable at all (it isn't during a freeze) to protect
707 against repeated upload errors. (Jobs would be even deleted otherwise
708 if the target dir is unaccessible for too long.) This check is
709 performed by creating a test file and deleting it immediately again.
710 But since in FTP incoming dirs deletion isn't permitted, the temporary
711 file ("junk-for-writable-test-DATE") will remain there. As a partial
712 fix, the daemon deletes such files immediately, it doesn't even wait
713 for $stray_remove_timeout. So if the upload goes to the queue dir of
714 an upstream debianqueued, those temporary files won't be there for
717 These problems of the FTP method might be remove in future, if I have
718 better ideas how to bypass the limitations of anon-FTP incoming
719 directories. Hints welcome :-)
723 # mode: indented-text
projects / dak.git / blob