Mailing List Archive

Throttle sending rates?
Hi,

The Problem I'm Trying To Solve:

I'm having repeated trouble with Microsoft owned domains. They have a
proprietary SmartScreen system which they do not seem to be able to
control (according to their own support teams). It means that I
frequently find all mail to @hotmail @outlook etc  is returned with a
"550 5.5.0 Requested action not taken: mailbox unavailable." message.

I use exim to send bulk mail (e.g.newsletters from my CRM) and also
transactional emails - I run several petition tools which send a "Thanks
for signing, please ask your friends to sign" sort of emails - these can
get quite popular and generate a lot of mail in a short period.

All mail is DKIM signed; SPF is in place; PTR records too. I believe my
server is set up properly.


The Solution I'm Asking About

I was wondering if it would help things if I rate limited the sending of
mail to Microsoft. e.g. mail to these domains gets put in queue and that
queue gets processed at a certain rate.

How would I implement that? I'm using exim 4.89


Other advice?

If anyone has any other advice on the problem it would be gladly received.


Thanks,

Rich

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Throttle sending rates? [ In reply to ]
On 24/10/2018 07:37, Rich Lott - Artful Robot via Exim-users wrote:
> I was wondering if it would help things if I rate limited the sending of
> mail to Microsoft. e.g. mail to these domains gets put in queue and that
> queue gets processed at a certain rate.

https://github.com/Exim/exim/wiki/RatelimitOutbound

--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Throttle sending rates? [ In reply to ]
On 2018-10-24, Rich Lott - Artful Robot via Exim-users <exim-users@exim.org> wrote:
> Hi,
>
> The Problem I'm Trying To Solve:
>
> I'm having repeated trouble with Microsoft owned domains. They have a
> proprietary SmartScreen system which they do not seem to be able to
> control (according to their own support teams). It means that I
> frequently find all mail to @hotmail @outlook etc  is returned with a
> "550 5.5.0 Requested action not taken: mailbox unavailable." message.
>
> I use exim to send bulk mail (e.g.newsletters from my CRM) and also
> transactional emails - I run several petition tools which send a "Thanks
> for signing, please ask your friends to sign" sort of emails - these can
> get quite popular and generate a lot of mail in a short period.

So when they ask their friends, do they use their own methods? or are they
submitting names to your mailing list?

> All mail is DKIM signed; SPF is in place; PTR records too. I believe my
> server is set up properly.
>
> The Solution I'm Asking About
>
> I was wondering if it would help things if I rate limited the sending of
> mail to Microsoft. e.g. mail to these domains gets put in queue and that
> queue gets processed at a certain rate.
>
> How would I implement that? I'm using exim 4.89

one way would be a named queue and then a sleep in the pre-connect code
(or one of the routers) for that queue.

> If anyone has any other advice on the problem it would be gladly received.

Multiple ip-addresses is the only solution that I have found to
actually work vs outlook.com.

to avoid problems with greylisting make the IP choice permanent by
basing it on an invariant like $message_exim_id


--
When I tried casting out nines I made a hash of it.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Throttle sending rates? [ In reply to ]
On 27/10/2018 14:01, Jeremy Harris via Exim-users wrote:
> On 24/10/2018 07:37, Rich Lott - Artful Robot via Exim-users wrote:
>> I was wondering if it would help things if I rate limited the sending of
>> mail to Microsoft. e.g. mail to these domains gets put in queue and that
>> queue gets processed at a certain rate.
>
> https://github.com/Exim/exim/wiki/RatelimitOutbound

Would it work to chain 2 exims and set ratelimit on the input of the second?



James


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Throttle sending rates? [ In reply to ]
On 29/10/2018 10:06, James via Exim-users wrote:
>> https://github.com/Exim/exim/wiki/RatelimitOutbound
>
> Would it work to chain 2 exims and set ratelimit on the input of the
> second?

I don't see why not. But... fugly!
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Throttle sending rates? [ In reply to ]
Hi,

Thanks for replies.
https://github.com/Exim/exim/wiki/RatelimitOutbound

Sort of makes sense, as a sketched idea, but I'm not sure I'd be able to
implement that without a lot more exim knowledge than I have amassed to
date! I couldn't -for example- find any examples of how to move 0-/N/
messages from one queue to the main queue.



> Would it work to chain 2 exims and set ratelimit on the input of the
> second?


I've heardof that before on old mailing list threads; seems to be
basically like the different queues suggestion except bigger because you
have two big exim processes running, instead.


I think this is all going to be beyond my skills at the mo. But had to
ask here in case there was a documented example that had escaped my
googling.

Thanks though,

Rich


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Throttle sending rates? [ In reply to ]
On 29/10/2018 11:29, Rich Lott - Artful Robot via Exim-users wrote:
> I couldn't -for example- find any examples of how to move 0-/N/
> messages from one queue to the main queue.

Given there's no queue-runner for this alternate queue, there's
nothing looking there - so no locking issues. Just move the files.

Alternately, tell exim to run one (at a time) message ID from the
named alternate queue. (but that's harder scripting, as you have
to pull the ID out of the filename).
--
Cheers,
Jeremy


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Throttle sending rates? [ In reply to ]
Hi Jeremy,

Ah right, I could so that.

So I'd have

1. An ACL (rcpt?) identifies outbound email for Micro$oft mail and sets
queue = annoying
2. Emails in these queues won't be touched by exim; no further
processing happens.
3. Cron job or such does something like one of these two:
1. ls -1 -t /var/spool/exim4 |tail 50 | while read f ; do exim -M
$(extract_job_id "$f") l done
2. ls -1 -t /var/spool/exim4 |tail 50 | while read f ; do mv "$f"
/var/spool/exim4/input/. ;done

?

Thanks,

Rich



On 29/10/2018 11:50, Jeremy Harris via Exim-users wrote:
> On 29/10/2018 11:29, Rich Lott - Artful Robot via Exim-users wrote:
>> I couldn't -for example- find any examples of how to move 0-/N/
>> messages from one queue to the main queue.
> Given there's no queue-runner for this alternate queue, there's
> nothing looking there - so no locking issues. Just move the files.
>
> Alternately, tell exim to run one (at a time) message ID from the
> named alternate queue. (but that's harder scripting, as you have
> to pull the ID out of the filename).
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Throttle sending rates? [ In reply to ]
On 29/10/2018 12:15, Rich Lott - Artful Robot via Exim-users wrote:
>     1. ls -1 -t /var/spool/exim4 |tail 50 | while read f ; do exim -M
>        $(extract_job_id "$f") l done

You forgot the exact dirname, queue-specifier, and s/l/; but otherwise yup


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Throttle sending rates? [ In reply to ]
On Mon, Oct 29, 2018 at 12:54 PM Jeremy Harris via Exim-users <
exim-users@exim.org> wrote:

> On 29/10/2018 11:29, Rich Lott - Artful Robot via Exim-users wrote:
> > I couldn't -for example- find any examples of how to move 0-/N/
> > messages from one queue to the main queue.
>
> Given there's no queue-runner for this alternate queue, there's
> nothing looking there - so no locking issues. Just move the files.
>
> Alternately, tell exim to run one (at a time) message ID from the
> named alternate queue. (but that's harder scripting, as you have
> to pull the ID out of the filename).


It's not hard scripting.

The ID is the first 16 chars of the filename, or everything in the header
filename except "-H".

In a POSIX shell or bash shell, this is how you extract that from a
filename stored in $filename, to the variable $id:

id=${filename%-H};

So, assuming a typical Exim 4 single-letter subdirectories spool directory
tree (substitute your chosen specific tree if necessary):

pushd /var/spool/exim4/input;
for d in ?; do
pushd $d;
for filename in *-H; do
id=${filename%-H};
# Do something useful with $id
done;
popd;
done;
popd;

This way, you can completely replace Exim's queue runners with your own
logic, if you want, but, eh, be careful?
--
Jan
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/