Mailing List Archive

"exim -bpc", but only for messages not waiting for retry
Hello,
(Sorry if this has already been asked before.)

I'm drawin a blank here.
Is there some way I can get the number of messages in the queue that are
"unhandled"?
I mean, that has not yet been tried.

I'm writing a script to start a bunch of queue runners if the queue gets
large, for when we send out a bunch of mails at the same time. The problem
is that since a lot of mails keeps getting held back, för various reasons,
the "exim -bpc" command always counts them anyway - of course - and I do
not want to start queue runners for them.

Regards
Charlie Elgholm
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: "exim -bpc", but only for messages not waiting for retry [ In reply to ]
On 07/12/2018 13:02, Charlie Elgholm via Exim-users wrote:
> I mean, that has not yet been tried.
>
> The problem
> is that since a lot of mails keeps getting held back, för various reasons,

In what sense "not tried"?

What's wrong with the usual queue-runner auto-ramp?
--
Cheers,
Jeremy


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: "exim -bpc", but only for messages not waiting for retry [ In reply to ]
Am 07.12.18 um 14:02 schrieb Charlie Elgholm via Exim-users:

> I'm drawin a blank here.
> Is there some way I can get the number of messages in the queue that are
> "unhandled"?

You could search for a specific time:

https://www.exim.org/exim-html-current/doc/html/spec_html/ch-exim_utilities.html#SECTgreptheque

-y <seconds>
Match messages that are younger than the given time.

-o <seconds>
Match messages that are older than the given time.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: "exim -bpc", but only for messages not waiting for retry [ In reply to ]
On 2018-12-07, Charlie Elgholm via Exim-users <exim-users@exim.org> wrote:
> Hello,
> (Sorry if this has already been asked before.)
>
> I'm drawin a blank here.
> Is there some way I can get the number of messages in the queue that are
> "unhandled"?
> I mean, that has not yet been tried.
>
> I'm writing a script to start a bunch of queue runners if the queue gets
> large, for when we send out a bunch of mails at the same time. The problem
> is that since a lot of mails keeps getting held back, för various reasons,
> the "exim -bpc" command always counts them anyway - of course - and I do
> not want to start queue runners for them.

exipick -bpc '$first_delivery'

--
When I tried casting out nines I made a hash of it.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: "exim -bpc", but only for messages not waiting for retry [ In reply to ]
On 2018-12-07 14:01, Jeremy Harris wrote:
> In what sense "not tried"?
Not yet delivery-tried, that is => "has only been queued as of yet, not
tried for delivery".

> What's wrong with the usual queue-runner auto-ramp?
We have Exim both for normal emails and bulk emails for one of our clients
employees (internal information).

For normal emails, and hence normal queue handling, the normal "auto-ramp"
works fine (we have ours at "30s", if that's what you mean).
Port 25 uses the normal queue-mechanism, where we try to delivery up to 5
emails directly, if the load average permits, and queue the rest.

But for the bulk-sending, we generate a couple of thousand (individual)
emails in batches, queue them all, and then would like to process the queue
as quickly as possible.
We use another port, which only listens internally, for the "queue only"
(bulk) process.

I can manually monitor the queue, and issue a bunch of "exim -q &" to start
a bunch of queue runners.
But sometimes our clients Baltic smtp-servers (they have many offices) are
offline or tells us "too many emails", and then the queue-runner of course
flags that individual message as "try later".
The "problem" is that these messages are of course included in "exim -bpc",
so I don't really know how many of the emails in the queue are actually
waiting to be retried or have not been tried.
We fill the queue at a rate of 1000 emails per minute, since that is how
many individually generated emails our application server and database
manages to spit out.
We generate around 8000 to 50000 emails, depending on what kind of
information is to be shared. Which means we keep filling the queue for
about 8 to 50 minutes, and we of course want Exim to start processing the
queue as soon as the first emails arrive, and "build up" queue-runners
faster than our normal 30s window.

If "exim -bpc" gives me a count 3000, is that 3000 emails to be tried, or
is it 2 950 emails retry-waiting an hour or two, and only 50 emails to be
tried?
I would like to know, since then I can implement my queue-runners smarter.
There's no need for me to start 50 queue-runners to process 2 950 emails
that are retry-waiting for an hour, and no need for 50 queue-runners to
send 50 emails, where 2-3 should suffice.

The problem, as the script is written now, is that I only check "exim
-bpc", which gives me a total queue-count.
I would like to have queue-count/100 runners, that is for 5000 mails in the
queue I would like to have 50 queue-runners.
So far, so good, I autostart so that I always have queue-count/100 runners,
since some die of after a queue-run but more emails have been filled in the
queue.
The problem is when the queue "settles", and by that I mean that I get
2-4000 emails "retry-waiting" in the queue.
My system has not way of telling, so it keep starting 20-40 queue-runners
(every minute), trying to process the queue - which of course is futile
since the retry-timeout hasn't occured yet.

Regards
/Charlie
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: "exim -bpc", but only for messages not waiting for retry [ In reply to ]
On 11/12/2018 10:44, Charlie Elgholm via Exim-users wrote:
>> What's wrong with the usual queue-runner auto-ramp?
> We have Exim both for normal emails and bulk emails for one of our clients
> employees (internal information).
>
> For normal emails, and hence normal queue handling, the normal "auto-ramp"
> works fine (we have ours at "30s", if that's what you mean).
> Port 25 uses the normal queue-mechanism, where we try to delivery up to 5
> emails directly, if the load average permits, and queue the rest.
>
> But for the bulk-sending, we generate a couple of thousand (individual)
> emails in batches, queue them all, and then would like to process the queue
> as quickly as possible.
> We use another port, which only listens internally, for the "queue only"
> (bulk) process.

Sounds like a perfect case for running separate queues, assuming 4.88
or later.

--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/