Mailing List Archive

Exim 5.x
Yeah, Exim 5 I think is not so far away, but that is not why I am writing.

Mailman 3 is knocking on the doors and I was just wondering if the Exim
gurus have crafted a config that works with it.

So far I see there is only support for Postfix. My whole life is, has been
and will always be Exim.

Gurus - any "perhaps" for Exim 4+Mailman 3 configs?

Thank you


--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254733744121/+254722743223
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
I can't hear you -- I'm using the scrambler.
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
> On Tue, 17 May 2011 18:46:41 +0300
> Odhiambo Washington <odhiambo@gmail.com> wrote:

> My whole life is, has been
> and will always be Exim.

Same here :-)

--
RMA.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
>
> Yeah, Exim 5 I think is not so far away, but that is not why
> I am writing.
>
> Mailman 3 is knocking on the doors and I was just wondering
> if the Exim
> gurus have crafted a config that works with it.
>
> So far I see there is only support for Postfix. My whole life
> is, has been
> and will always be Exim.
>
> Gurus - any "perhaps" for Exim 4+Mailman 3 configs?
>

Have you seen
http://wiki.list.org/display/DEV/LMTP+process
?

There's a section on Exim and Mailman 3 there which should be enough to get you started perhaps. It basically seems that you just deliver messages destined for Mailman via LMTP.

John
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On Tue, May 17, 2011 at 18:59, John Burnham <John.Burnham@admin.cam.ac.uk>wrote:

> >
> > Yeah, Exim 5 I think is not so far away, but that is not why
> > I am writing.
> >
> > Mailman 3 is knocking on the doors and I was just wondering
> > if the Exim
> > gurus have crafted a config that works with it.
> >
> > So far I see there is only support for Postfix. My whole life
> > is, has been
> > and will always be Exim.
> >
> > Gurus - any "perhaps" for Exim 4+Mailman 3 configs?
> >
>
> Have you seen
> http://wiki.list.org/display/DEV/LMTP+process
> ?
>
>
I actually did not have any clue about that link at all.


> There's a section on Exim and Mailman 3 there which should be enough to get
> you started perhaps. It basically seems that you just deliver messages
> destined for Mailman via LMTP.
>

I have never used Exim with LMTP so this is going to be new and as such the
biggest hurdle is what to throw in in mailman.cf in the MTA sections:

<cut>
[mta]
# The class defining the interface to the incoming mail transport agent.
incoming: mailman.mta.postfix.LMTP

# The callable implementing delivery to the outgoing mail transport agent.
# This must accept three arguments, the mailing list, the message, and the
# message metadata dictionary.
outgoing: mailman.mta.deliver.deliver

# How to connect to the outgoing MTA. If smtp_user and smtp_pass is given,
# then Mailman will attempt to log into the MTA when making a new
connection.
smtp_host: localhost
smtp_port: 25
smtp_user:
smtp_pass:

# Where the LMTP server listens for connections. Use 127.0.0.1 instead of
# localhost for Postfix integration, because Postfix only consults DNS
# (e.g. not /etc/hosts).
lmtp_host: 127.0.0.1
lmtp_port: 8024

</cut>

Any ardent Exim LMTP users around? Pls help.

--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254733744121/+254722743223
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
I can't hear you -- I'm using the scrambler.
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On 17/05/2011 17:21, Odhiambo Washington wrote:

> Any ardent Exim LMTP users around? Pls help.

I've never used LMTP before, but it looks really simple. Take a look at:

http://www.exim.org/exim-html-current/doc/html/spec_html/ch28.html

I guess you need something like the example on that page, but using
"socket" instead of "command".

You'd need to write a router to call that transport as well of course.

--
Mike Cardwell https://grepular.com/ https://twitter.com/mickeyc
Professional http://cardwellit.com/ http://linkedin.com/in/mikecardwell
PGP.mit.edu 0018461F/35BC AF1D 3AA2 1F84 3DC3 B0CF 70A5 F512 0018 461F
Re: Exim 5.x [ In reply to ]
On Tue, 17 May 2011, Mike Cardwell wrote:

> From: Mike Cardwell <exim-users@lists.grepular.com>
> To: exim-users@exim.org
> Date: Tue, 17 May 2011 18:09:07
> Subject: Re: [exim] Exim 5.x
> X-Spam-Score: 0.0 (/)
>
> On 17/05/2011 17:21, Odhiambo Washington wrote:
>
> > Any ardent Exim LMTP users around? Pls help.
>
> I've never used LMTP before, but it looks really simple. Take a
> look at:

...

LMTP is simple. I've used it in the past over TCP/IP to deliver
email to a Cyrus IMAP server listening on the loopback address.
Worked a treat.
--
Dennis Davis, BUCS, University of Bath, Bath, BA2 7AY, UK
D.H.Davis@bath.ac.uk Phone: +44 1225 386101

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On 17 May 2011, at 18:09, Mike Cardwell wrote:

> On 17/05/2011 17:21, Odhiambo Washington wrote:
>
>> Any ardent Exim LMTP users around? Pls help.
>
> I've never used LMTP before, but it looks really simple. Take a look at:
>
> http://www.exim.org/exim-html-current/doc/html/spec_html/ch28.html

No, really, don't use the LMTP transport. It's much easier to use an SMTP Transport and add the line "protocol = lmtp".

You'd imagine that the LMTP transport would use the same code base, but it doesn't. The only advantage of the LMTP transport is that it allows you to deliver through a unix socket. The disadvantage is that it lacks most of the SMTP transport features.

We use LMTP to deliver to Cyrus-IMAP mailstores, but quickly decided against the LMTP transport.

The advantage of Mailman 3 is that it will permit SMTP time rejection of emails based on sender/list permissions. No more bounce messages or discards, unless you want them. Also, the use of LMTP means that you might safely send one message to two lists, and allow the lists to discriminate on the basis of message content (including, for example, the From header, presence of attachments, or body size).

It's possible that the only modification you'll need to make to your Exim config is to add the line "protocol = lmtp" to your transport. If you use that transport for other purposes, then duplicate fix and rename it, and then point your mailman router at the new transport.

--
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On 18 May 2011, at 10:31, Ian Eiloart wrote:

> No, really, don't use the LMTP transport. It's much easier to use an SMTP Transport and add the line "protocol = lmtp".
>
> You'd imagine that the LMTP transport would use the same code base, but it doesn't. The only advantage of the LMTP transport is that it allows you to deliver through a unix socket. The disadvantage is that it lacks most of the SMTP transport features.

In fact, I think it would be good to permit the SMTP transport to deliver through a socket, then to deprecate the LMTP transport completely.

--
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
Odhiambo Washington, 2011-05-17 17:46:
> Yeah, Exim 5 I think is not so far away,

Huh? What makes you think that? I don't see any sign of it.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On 18 May 2011, at 15:15, Jakob Hirsch wrote:

> Odhiambo Washington, 2011-05-17 17:46:
>> Yeah, Exim 5 I think is not so far away,
>
> Huh? What makes you think that? I don't see any sign of it.
>

I guess that with 4.76 we're more than three-quarters done! ;^)

But no, I don't see a sign of 5.0, or even discussion of what desirable radical changes would justify a major version number change. It might just be that we run out of minor version numbers!

For the core server, LEMONADE compliance might be worth a full version bump. Since that requires several distinct changes, it's something that could be the end point of several minor version bumps.




--
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On Thu, 2011-05-19 at 09:58 +0000, Ian Eiloart wrote:
> For the core server, LEMONADE compliance might be worth a full version
> bump. Since that requires several distinct changes, it's something
> that could be the end point of several minor version bumps.

I'm mostly familiar with LEMONADE in the context of IMAP. What needs
doing on the SMTP side?

--
dwmw2


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
Ian Eiloart wrote:
> LEMONADE compliance
?

I don't see that what has been published so far has any interaction with
an MTA-only such as Exim at all. Integrated MTA+IMAP, perhaps....

Otherwise, the LEMONADE features proposed appear to me to be all in the
IMAP daemon's realm:

- IMAP IDLE for low b/w & virtual 'push'?
Old stuff.

- Forward w/o downloading body/attachments?
See Courier-IMAP's 'out tray', VERY old stuff.

What have I missed?

- small-screens?
More of a webmail function, IMNSHO.
See Gophermail/Chickadee fork of Prayer.
Or just add an optionable theme...

- 'true' push? (no continuously open data connection)
Email to SMS gateway with selective headers-only notification.
Also old stuff.

Bill


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On 2011-05-19 at 11:09 +0100, David Woodhouse wrote:
> I'm mostly familiar with LEMONADE in the context of IMAP. What needs
> doing on the SMTP side?

BDAT, BURL (and thus urlauth validation, a policy engine, etc)

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On 2011-05-19 at 09:58 +0000, Ian Eiloart wrote:
> But no, I don't see a sign of 5.0, or even discussion of what
> desirable radical changes would justify a major version number change.
> It might just be that we run out of minor version numbers!

From the June 2010 maintainers meet-up summary notes (excuse long lines
please):

----------------------------8< cut here >8------------------------------
Release:
* No major new release, universal consensus
* No fixed schedule for releases
* Aim for at least one every six months, with "go with what we have" if we have been idle
* Don't break lexical comparison, not going 4.9901
* So will go from 4.99 to 5.00
* might jump ahead to 5.00 release after a new queuing system or another major new feature
* consensus that if we go for separate queue directories, backwards compat will be a Perl script for re-merging queues
* no on-disk format changes, just need re-merge
* consensus that full 8BITMIME support is needed and we should bite the bullet (and we're unhappy)
* regular spool file remains unmolested, auxilliary spool file with down-conversion created on demand and
unlinked as needed; if down-convert on-fly, omit SIZE
----------------------------8< cut here >8------------------------------

New queuing system refers to an approach to scale up the spool directory
to something more queue-like, with segregated admin-defined queues (eg,
"big_freemail_provider_x"). This is because while Exim is excellent at
inbound mail, it doesn't always scale as well as some would like for
outbound mail which can't be immediately delivered. Nothing has been
done on this. Patches welcome.

8BITMIME -- so much of the email system is 8-bit clean these days that
the whole down-conversion system of 8BITMIME is a travesty. But before
you can advertise 8BITMIME, you have to be able to either down-convert
or bounce, and bouncing leads to backscatter. So, we need to be able to
down-convert, in the expectation of never needing to do it. Exim's the
only major MTA to not advertise 8BITMIME by default. (And if you're
running an inbound-only 8-bit clean system, I encourage you to turn on
the "accept_8bitmime" option, which really just causes Exim to advertise
the EHLO keyword and to parse the extra MAIL adverbs).

Between security releases and "every 6 months", we have about 6 years to
get a major feature added for which we can justify a 5.00 bump before we
just hit it naturally, shrug, say "meh" and move on with no especially
substantive difference between 4.99 and 5.00. As long as it's not a
security fix, I'll be happy.

-Phil

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On 19 May 2011, at 11:24, W B Hacker wrote:

> Ian Eiloart wrote:
>> LEMONADE compliance
> ?
>
> I don't see that what has been published so far has any interaction with an MTA-only such as Exim at all. Integrated MTA+IMAP, perhaps....
>
> Otherwise, the LEMONADE features proposed appear to me to be all in the IMAP daemon's realm:
>
> - IMAP IDLE for low b/w & virtual 'push'?
> Old stuff.
>
> - Forward w/o downloading body/attachments?
> See Courier-IMAP's 'out tray', VERY old stuff.
>
> What have I missed?

The SMTP components listed at http://www.lemonadeformobiles.com/detail.html

These items are implemented already. But, their relationship to LEMONADE isn't in our docs:

SUBMIT
PIPELINING
SIZE
START-TLS
SMTPAUTH

These are sort of implemented:

8BITMIME - this is implemented, but off by default. 5.0 (or 4.8) could enable it by default, perhaps.

ENHANCEDSTATUSCODES - these aren't implemented by default, though they are configurable. Ideally, they'd be in the default configuration, and perhaps an ACL option would specify a status code. Version 5.0 might make an ACL invalid if it didn't specify an enhanced status code.

These aren't implemented at all, as far as I can see:
* BURL - this part allows integration with an IMAP server, a message is submitted with a an IMAP url to allow forward without download, etc.
* CHUNKING rfc3030 - allows large messages to be split into chunks.
* BINARYMIME rfc3030 - optimises bandwidth
* DSN - an SMTP extension defined in rfc3461 which allows the sender to specify conditions under which DSNs should be created.

Perhaps a starting point would be a wiki page - or a chapter in the documentation - explaining Exim's LEMONADE compliance.



> - small-screens?
> More of a webmail function, IMNSHO.
> See Gophermail/Chickadee fork of Prayer.
> Or just add an optionable theme...
>
> - 'true' push? (no continuously open data connection)
> Email to SMS gateway with selective headers-only notification.
> Also old stuff.
>
> Bill
>
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/

--
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On Thu, May 19, 2011 at 13:50, Phil Pennock <pdp@exim.org> wrote:

> On 2011-05-19 at 09:58 +0000, Ian Eiloart wrote:
> > But no, I don't see a sign of 5.0, or even discussion of what
> > desirable radical changes would justify a major version number change.
> > It might just be that we run out of minor version numbers!
>
> From the June 2010 maintainers meet-up summary notes (excuse long lines
> please):
>
> ----------------------------8< cut here >8------------------------------
> Release:
> * No major new release, universal consensus
> * No fixed schedule for releases
> * Aim for at least one every six months, with "go with what
> we have" if we have been idle
> * Don't break lexical comparison, not going 4.9901
> * So will go from 4.99 to 5.00
> * might jump ahead to 5.00 release after a new queuing system or
> another major new feature
> * consensus that if we go for separate queue directories, backwards
> compat will be a Perl script for re-merging queues
> * no on-disk format changes, just need re-merge
> * consensus that full 8BITMIME support is needed and we should bite
> the bullet (and we're unhappy)
> * regular spool file remains unmolested, auxilliary spool
> file with down-conversion created on demand and
> unlinked as needed; if down-convert on-fly, omit SIZE
> ----------------------------8< cut here >8------------------------------
>
>
Jakob, see?

Exim-5.00 (Codename Hazel) is not that far away. Could even take less than 6
years if some gifted hackers were paid to bring in the major changes
required to get there.

Can't we, the Exim community, raise funds to get some of us to do this new
queuing system? If such initiative can be realized, I am so in it!


--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254733744121/+254722743223
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
I can't hear you -- I'm using the scrambler.
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
Hi

While I don't really care about version numbers but only about
functionality, the following is just an worthless opinion, but still my 2
ct.

> * Don't break lexical comparison, not going 4.9901
> * So will go from 4.99 to 5.00

Not breaking lexical comparision and therefor regarding 4.99 to 5.00 similar
as 4.67 to 4.68 is a broken concept, or you must make sure, exim is REALLY
complete (feature-complete, bug-free, ...) before you reach 9.99. Because
then you will have to break on of the rules above.

The much cleaner solution would be to go from 4.99 to 4.100, which good
parsers can handle.
Reserve 5.0 (followed by 5.1) for really incompatible changes, conceptional
changes. Such as 3.x to 4.x.

Regards,
Steffen



--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On 19/05/11 11:50, Phil Pennock wrote:
>
> New queuing system refers to an approach to scale up the spool directory
> to something more queue-like, with segregated admin-defined queues (eg,
> "big_freemail_provider_x"). This is because while Exim is excellent at
> inbound mail, it doesn't always scale as well as some would like for
> outbound mail which can't be immediately delivered. Nothing has been
> done on this. Patches welcome.
>
Is this AKA bug 336? It sounds quite interesting, so I think I might
have a look at making some inroads into the problem.

If there are are any notes/thoughts about behaviour/config/use it would
be handy to hear them.



--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
Ian Eiloart wrote:
>
> On 19 May 2011, at 11:24, W B Hacker wrote:
>
>> Ian Eiloart wrote:
>>> LEMONADE compliance
>> ?
>>
>> I don't see that what has been published so far has any interaction
>> with an MTA-only such as Exim at all. Integrated MTA+IMAP,
>> perhaps....
>>
>> Otherwise, the LEMONADE features proposed appear to me to be all in
>> the IMAP daemon's realm:
>>
>> - IMAP IDLE for low b/w& virtual 'push'? Old stuff.
>>
>> - Forward w/o downloading body/attachments? See Courier-IMAP's 'out
>> tray', VERY old stuff.
>>
>> What have I missed?
>
> The SMTP components listed at
> http://www.lemonadeformobiles.com/detail.html

Thanks, thats' more useful...

>
> These items are implemented already. But, their relationship to
> LEMONADE isn't in our docs:
>
> SUBMIT PIPELINING SIZE START-TLS SMTPAUTH
>
> These are sort of implemented:
>
> 8BITMIME

RFC dated JUL 1994

> - this is implemented, but off by default. 5.0 (or 4.8)
> could enable it by default, perhaps.
>

ACK. Noting also Phil's recent 'roadmapish' comment.

Living in a high percentage of Chinese-encoding environment, I rely on
what we already have, do all composition with MUA that can either force
or auto-select UTF-8.

Not a hundred-percenter, especially with reply-to's, and we may still
have to select from among another 5 or so common encoding for Chinese
alone (but not all 20+).

Most interested in progress on that front, happy to test, and not
limited to Chinese.

> ENHANCEDSTATUSCODES

RFC dated OCT 1996

> - these aren't implemented by default, though
> they are configurable.

ACK. No show-stopper...

> ... Ideally, they'd be in the default
> configuration, and perhaps an ACL option would specify a status code.
> Version 5.0 might make an ACL invalid if it didn't specify an
> enhanced status code.
>
Uhhhh . .rather see it default to SOMETHING, as most now do..

> These aren't implemented at all, as far as I can see:

> * BURL

RFC dated MAY 2006.

>- this
> part allows integration with an IMAP server, a message is submitted
> with a an IMAP url to allow forward without download, etc.

??

IF the content is already located at a URI, all that is needed is the
URI. We all get such - mostly as advertising.

ELSE content is IN local MTA/IMAP mailstore OR an auxiliary
storage/location known to its Mother, and can be SENT to or linked to by
.. a URI, at which point the Luser still needs ... only that URI.

MLM archives & digests sound familiar?

Otherwise, I'm still missing the point of what and how wants changing.

> * CHUNKING
> rfc3030

RFC dated DEC 2000

> - allows large messages to be split into chunks.

> * BINARYMIME
> rfc3030 - optimises bandwidth

RFC dated DEC 2000

> * DSN - an SMTP extension defined in
> rfc3461

RFC dated JAN 2003

> which allows the sender to specify conditions under which
> DSNs should be created.
>

> Perhaps a starting point would be a wiki page - or a chapter in the
> documentation - explaining Exim's LEMONADE compliance.
>

Looking just at the ages of RFC's from six to seventeen years old - it
seems what was found useful enough to gain traction, did so ... and has
been actioned.

The rest?

Seems the world has been in no hurry to drink that kool-aide, er scratch
that

...'LEMONADE'...

;-)

Looks like yet-another a big-corp sponsored job-security exercise to
me... check out the players...


Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
Steffen Heil wrote:

*snip*

> Reserve 5.0 (followed by 5.1) for really incompatible changes, conceptional
> changes. Such as 3.x to 4.x.
>
> Regards,
> Steffen

+++

... and Amen.

Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On Thu, 2011-05-19 at 10:24 +0000, W B Hacker wrote:
>
> - 'true' push? (no continuously open data connection)
> Email to SMS gateway with selective headers-only notification.

More likely this would take the form of an SMS message with
data-coding-scheme 0xCA (see GSM-03.38 §4).

That would normal just turn on the 'email waiting' icon, but it's not
*such* an abuse to make it actually trigger a mail fetch...

--
dwmw2


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
On 2011-05-19 at 12:35 +0100, Dominic Benson wrote:
> On 19/05/11 11:50, Phil Pennock wrote:
> >
> > New queuing system refers to an approach to scale up the spool directory
> > to something more queue-like, with segregated admin-defined queues (eg,
> > "big_freemail_provider_x"). This is because while Exim is excellent at
> > inbound mail, it doesn't always scale as well as some would like for
> > outbound mail which can't be immediately delivered. Nothing has been
> > done on this. Patches welcome.
> >
> Is this AKA bug 336? It sounds quite interesting, so I think I might
> have a look at making some inroads into the problem.
>
> If there are are any notes/thoughts about behaviour/config/use it would
> be handy to hear them.

I wasn't aware of 336, but yes it's related.

At present, there's split_spool_directory, which divides things up with
one level of hashing, and then some people script their own queue-runner
launchers, running in parallel over sub-trees of the split spool instead
of having the Exim daemon launch runners over everything, which compete
with each other.

Nothing more specific was discussed, that I either recall or find in the
minutes; we all understood the general problem.

If we use a new sub-directory of spool_directory to hold named queues,
then previous Exim installs won't know of the content, but it should be
fairly easy to script a rollback tool which recombines queues into the
original queue.

Thinking briefly (while tired and prone to mistakes):
You'd need a way to declare "move this message to this other queue",
perhaps a way to restrict Routers to only apply to messages in certain
queues (seems optional as a performance optimisation, *could* be useful
for some more sophisticated Router rule bifurcations) and either a way
to control counts of queue-runners per named queue or an API to do so.
Probably best to have a global config option defining a list of allowed
queue names, so that a typo doesn't end up putting 100k messages into
"gmial" where they sit idle. ;)

Beyond that, I think there's a lot of freedom for the implementor to
make their own decisions. :) Have at it!


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: admin defined queue [was Exim 5.x] [ In reply to ]
On 19 May 2011, at 14:36, Phil Pennock wrote:
> If we use a new sub-directory of spool_directory to hold named queues,
> then previous Exim installs won't know of the content, but it should be
> fairly easy to script a rollback tool which recombines queues into the
> original queue.

or how about tagging messages with one (or more) identifiers, and having
a list of identifiers that match that tag contained in some lightweight
(hints - so not completely authoritative) database.

Can then work through items with tag x or skip items with tag y etc

And a fallback queue runner that does all messages (and updates tag
indexes as it runs).

[.This isn't thought through, its just an idea that came to me]

Nigel.
--
[ Nigel Metheringham ------------------------------ nigel@dotdot.it ]
[ Ellipsis Intangible Technologies ]



--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 5.x [ In reply to ]
Phil Pennock wrote:
> On 2011-05-19 at 12:35 +0100, Dominic Benson wrote:
>> On 19/05/11 11:50, Phil Pennock wrote:
>>>
>>> New queuing system refers to an approach to scale up the spool directory
>>> to something more queue-like, with segregated admin-defined queues (eg,
>>> "big_freemail_provider_x"). This is because while Exim is excellent at
>>> inbound mail, it doesn't always scale as well as some would like for
>>> outbound mail which can't be immediately delivered. Nothing has been
>>> done on this. Patches welcome.
>>>
>> Is this AKA bug 336? It sounds quite interesting, so I think I might
>> have a look at making some inroads into the problem.
>>
>> If there are are any notes/thoughts about behaviour/config/use it would
>> be handy to hear them.
>
> I wasn't aware of 336, but yes it's related.
>
> At present, there's split_spool_directory, which divides things up with
> one level of hashing, and then some people script their own queue-runner
> launchers, running in parallel over sub-trees of the split spool instead
> of having the Exim daemon launch runners over everything, which compete
> with each other.
>
> Nothing more specific was discussed, that I either recall or find in the
> minutes; we all understood the general problem.

Known bound, yes.

Problem?

Not sure.

I stand on the position that WHEN one is loading Exim so heavily that it
HITS that sort of bound, one has far too many eggs in one basket
downtime-risk-wise, and should split that load over multiple Exim (free)....

... on multiple separate boxen (generally cheap). Or at least usually
'cheapER' than answering complaints from such a huge user base - or
rushing to a remote data center.

>
> If we use a new sub-directory of spool_directory to hold named queues,
> then previous Exim installs won't know of the content, but it should be
> fairly easy to script a rollback tool which recombines queues into the
> original queue.
>
> Thinking briefly (while tired and prone to mistakes):
> You'd need a way to declare "move this message to this other queue",

.... or just a small critter managing ephemeral softlinks, perhaps?

> perhaps a way to restrict Routers to only apply to messages in certain
> queues (seems optional as a performance optimisation, *could* be useful
> for some more sophisticated Router rule bifurcations) and either a way
> to control counts of queue-runners per named queue or an API to do so.

Probably do-able already with a bit of metal bending.

For sure routers have no problem delivering wherever admin-config
directed, and retrieval is just the other side of the same coin should
it be seen useful.

> Probably best to have a global config option defining a list of allowed
> queue names, so that a typo doesn't end up putting 100k messages into
> "gmial" where they sit idle. ;)
>

Worse has happened.

Left a delimiter off the front of a complex SQL call once. Messages
disappeared for a while.

Took 'ls' to the mailstore.

Found a new user with the path to Maildir not the *result* of the SQL
returned from the DB...

.. but rather the full-text of the unbounded SQL call *itself* ..as the
directory name.

:-(

> Beyond that, I think there's a lot of freedom for the implementor to
> make their own decisions. :) Have at it!
>
>

Aye. 'Enough rope ....'

Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: admin defined queue [was Exim 5.x] [ In reply to ]
Nigel Metheringham wrote:
>
> On 19 May 2011, at 14:36, Phil Pennock wrote:
>> If we use a new sub-directory of spool_directory to hold named queues,
>> then previous Exim installs won't know of the content, but it should be
>> fairly easy to script a rollback tool which recombines queues into the
>> original queue.
>
> or how about tagging messages with one (or more) identifiers, and having
> a list of identifiers that match that tag contained in some lightweight
> (hints - so not completely authoritative) database.
>
> Can then work through items with tag x or skip items with tag y etc
>
> And a fallback queue runner that does all messages (and updates tag
> indexes as it runs).
>
> [.This isn't thought through, its just an idea that came to me]
>
> Nigel.
> --

Sounds akin to what is already used for retry & general thawing, no?

..or is that also in need of a tune up?

Bill



--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

1 2  View All