Mailing List Archive

Moving the QMail Queue.
Hi all,

I have to create a new mail server which will take over the job of the
current mail server. It was intended to have some new hardware for this,
but it now looks like it is not going to happen. Basically, I am going
to have to build on the existing hardware.

The bit I am most worried about (aside from the obvious downtime) is
moving the qmail queue from the old setup to the new one. Does anyone
have a document relating to the procedure for this? I am aware I can't
just copy the queue files from one setup to another and expect it to
work.

Any opinions on the best approach out of:

A: setup the old qmail on another box and change some settings on it so
it forwards all mail to the new qmail server (basically smarthosting) -
although I am worried in this circumstance that putting the queue on
another disk will probably break it.

B: moving the queue from the old setup to the new one, via file
transfer, and making it a valid queue, allowing the new machine to
deliver all the mail from the old one. Also worried about moving the
queue, to another disk, and breaking it.

Either of these options is not optimal. Optimal would be having new
hardware to build on so if it breaks we have the old server to fall back
to. Has anyone been asked to do such an upgrade without being able to
use new hardware? And, is this too risky a process to do without new
hardware? (I personally think it is too risky but would like some backup
:)

Thanks for any input,

Cheers,
Michael Hutchinson
Re: Moving the QMail Queue. [ In reply to ]
On Thu, Mar 06, 2008 at 05:18:35PM +1300, Michael Hutchinson wrote:
> A: setup the old qmail on another box and change some settings on it so
> it forwards all mail to the new qmail server (basically smarthosting) -
> although I am worried in this circumstance that putting the queue on
> another disk will probably break it.

Did this more than once :)
1) reduce the TTL in DNS to a few seconds for the name of the mailserver
if you are using different IP addresses. This will make transition
faster and also switching back in case of a problem.
2) rsync the user mail store (maildirs) over to the new machine. Depending
on the size of the mail store this can take quite some time. This can be
and should be done while the old server is still running. Shortly
before you do the final transition rsync again to keep downtime
short.
3) shutdown qmail on the old system
4) rsync again (that should be rather fast now)
5) change DNS to point to the new system
6) monitor the new system whether it works smoothly
7) on the new system configure the IP address of the old system to have
RELAYCLIENT set (tcprules), so you can use the new system as a
smarthost from the old system
8) on the old machine rename control/virtualdomains and user/assign.cdb
away (e.g. by adding .away)
9) on the old machine make control/smtproutes look like
:[ipaddress.of.new.mailserver]
this tells qmail-remote to send all remote mail via the new machine
using it as a smarthost. the [] are needed if you specifyu an IP
address which I would recommend to circumvent DNS problems.
10) restart qmail on the old machine and do a "svc -a /service/qmail-send"
(or "kill -ALRM" the pid of qmail-send)
this should send all the mails in the queue to the new mailserver,
which will put them in its queue.
11) check with qmail-qstat. As soon as it shows
messages in queue: 0
messages in queue but not yet preprocessed: 0
all mails are transferred to the new system.
12) shut down the old system

Done. (Hope I didn't miss something)

The only drawback of this method is that mails in the queue loose their
queue-time. This means if you have a queuelifetime of 8 days and the
mail spent 7 days on the old server it will spend another 8 days on the
new server (if it is undeliverable e.g.)

If you want to be "user friendly" and disburden your support/help desk
you can only shut down the qmail-smtpd and qmail-send portions and replace
the qmail-pop3d (if you are running one) with a dummy that always
authenticates with "correct" and always deliveres the same message saying
that the mailserver has moved and is using a new machine and people should
reconfigure their mail client settings to use the name "hostname" instead
of the ip address. (Can't find the pop3d I used right now, but maybe there is
even some info on it on qmail.org).

\Maex
Re: Moving the QMail Queue. [ In reply to ]
On 2008-03-05, at 2318, Michael Hutchinson wrote:
>
> I have to create a new mail server which will take over the job of the
> current mail server. It was intended to have some new hardware for
> this,
> but it now looks like it is not going to happen. Basically, I am going
> to have to build on the existing hardware.
>
> The bit I am most worried about (aside from the obvious downtime) is
> moving the qmail queue from the old setup to the new one. Does anyone
> have a document relating to the procedure for this? I am aware I can't
> just copy the queue files from one setup to another and expect it to
> work.

usually when i've done upgrades, i try to make sure the queue is empty
(i.e. "qmail-qstat" shows two zeros.) this way i don't have to worry
about preserving the queue.

however, if you have to upgrade in-place, and you absolutely cannot
empty the queue first, you will need to make sure that the new qmail
is compiled with the same "conf-split" value as the old one, and that
the numeric UID and GID values for the qmail users and groups do not
change. you also have to be very careful to never "copy" the queue,
only "mv" it... and never "mv" it from one filesystem to another.

my normal procedure looks like this:

- shut down everything on the machine which relates to qmail. if
possible, drop to single-user mode and unplug the network cables
entirely.

- rename "/var/qmail" to "/var/oldqmail" or something like that-
basically move it out of the way , but don't delete it, and don't move
it from one filesystem to another (hopefully you don't have "/var/
qmail" as a separate filesystem- it's okay if "/var" is a filesystem
though.)

- make a new, empty /var/qmail directory, and compile the new qmail
using whatever patches and options you need. again, make sure "conf-
split" doesn't change, and don't delete or re-create the qmail users.

- copy or "mv" the control/*, alias/*, and users/* files from the old
qmail to the new one.

- remove "/var/qmail/queue" (the empty queue in the new version.)

- rename "/var/oldqmail/queue" to "/var/qmail/queue". this will "move"
the entire queue, en masse, from the old instance of qmail to the new
one.

- start up qmail-send and cross your fingers.

i have done this before, probably seven or eight times over the past
ten years, and haven't lost a queue yet.

the reason for all this is that the "message number" is actually the
inode number of the "mess" file. when a new message is added to the
queue, the message data is written to a dummy file, then the file is
renamed to the inode number of that dummy file. using the inode number
is a way to guarantee that the filename will be unique within the
queue (presuming the queue is all stored within a single filesystem,
which it must be, according to the directions.)

if the queue is copied, the "mess" files' inode numbers won't match
the filenames, and there's a risk of a new message happening to get
the same inode number as the filename of an existing message- which
would be a BAD THING.

there are a few "queue fixer" scripts out there (including my own
"qfixq" script) which can take a "copied" queue and rename the files
so that the filenames match the inode numbers properly. (you can
change the name of a file, but you can't explicitly change the inode
number of a file.) these scripts do work, so copying a queue
improperly isn't the end of the world... but doing this will cause the
"message numbers" to change, which means you won't be able to follow a
message through the log files to trace where it went (when a difficult
client, boss, or other user demands that you do so.)

--------------------------------------------------------
| John M. Simpson -- KG4ZOW -- Programmer At Large |
| http://www.jms1.net/ <jms1@jms1.net> |
--------------------------------------------------------
| Hope for America -- http://www.ronpaul2008.com/ |
--------------------------------------------------------