Mailing List Archive

How to modify the MAIL FROM argument? (was: Re: Top problems with SPF acceptance)
Stuart D. Gathman wrote:
> [An MTA maintainer] *should* be interested in allowing user extensions
> (via postfix policy daemons or milter api) to modify MAIL FROM.

I'd still be interested in knowing what an API to modify MAIL FROM
might look like. I'm not quite familiar with neither postfix nor
sendmail (I use Courier).

I change the MAIL FROM via a -f command line switch, and I do it upon
delivery, i.e. after accepting responsibility for delivering the
message. For example, I have ".courier" files (similar to ".forward")
and for SPF compliance I had to substitute lines like

forward@example.com

with

|/usr/sbin/sendmail -f postmaster@mine forward@example.com

Using milter apparently implies doing it earlier. Is that right after
the MAIL FROM(!?), after RCPT, or after DATA?

Sometimes, I forward based on specific message contents, using
maildrop, that's the reason why I find it more natural to do it after
the message has been accepted. What are the pros of doing it "online"?

> Proposals should distance themselves from specific applications. If an example
> is needed to show why it would be useful to modify MAIL FROM, pick
> one like BATV, since SRS seems to have accumulated bad vibes in his mind.

Can that be understood in an MTA-independent manner? You already
mentioned a number of different policies (i.e. BATV, SRS, ecc.) that
can be applied according with different strategies (i.e. after RCPT,
after DATA, after acceptance), thus an MTA-wise fragmentation on top
of that is quite an hindrance for any development.

In other words, if we had a nice program that did The Right Thing for
each forwarded recipient, how would we fit it into our differing MTAs?
I assume that, in order to fix forwarding, we aim at designing such a
program. If not, please tell me what are the alternatives.

TIA

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://www.listbox.com/member/?member_id=1311532&id_secret=95897010-3d7186
Powered by Listbox: http://www.listbox.com
Re: How to modify the MAIL FROM argument? (was: Re: Top problems with SPF acceptance) [ In reply to ]
Alessandro,

Great post. I am the last to want to throw a wrench into forward
progress for SPF, but before we find that "nice program that did The
Right Thing for each forwarded recipient", I think the definition
(specs) for exactly how that nice program can work needs to be agreed upon.

As a practical matter, SPF enjoys a clear leadership position as
regards user adoption and acceptance of all of the domain identity
theft solutions. It seems to me that it should be the people on this
list (and more specifically the folks whose names appear in the RFC
for SPF), who have worked to drive the early SPF V1 standard and get
it so widely adopted, who should take a leadership position in
extending SPF to address and publish a specification to address
forwarding in such a way which works best within the SPF framework.

This add on specification should be created in the same spirit of not
breaking anything that exists to date as SPF itself. SPF v1 pretty
much does what it advertises, but we all understand that certain
acceptable forwarding situations won't work for certain
applications. While the problem generally does not impact most SPF
implementations (my own included), I think that it would be nice to
put these kind of concerns to bed once and for all.

I think that David MacQuigg had tried to begin the process by
creating a common lexicon of terms to nail down what are and are not
acceptable forwarding situations and how they could be addressed. It
seems to me that Dave Croker is also trying to update the definitions
of terms with his recent RFC (posted the other day on this list by
Frank Ellermann -
http://www.ietf.org/internet-drafts/draft-crocker-email-arch-10.txt
). It would be nice to see if we can use Dave Croker's RFC to create
that foundation for discussing the forwarding issues that David
MacQuigg was trying to accomplish.

One of the other members of this list had wished to avoid changing
terminology (which I understand and tend to agree with), but given
Dave Croker's efforts, it seems that there are also others who are
not SPF centric who see a need for some updating of terms too. So
maybe there is some room to look at refining terminology to reflect
the evolution of the way email usage has changed over the past 10 or so years.

As such, why not let's start to make a run at an SPF recommended
forwarding recommendation practices document that will allow SPF to
work and will address legitimate forwarding concerns of others?

I think in talking about the various MTA / MCA products, we will not
get too far for emotional reasons (I like this one, while you like
that one arguments), however if we can come up with a good answer
based on a specification that can be adopted by the various
suppliers, we might get some forward momentum on an answer to this
forwarding issue that seems the last FUD (fear uncertainty and doubt)
which seems to stop some from implementing SPF as a solution to
domain name identity theft.

So I ask, does this make sense and is this something we are willing
to do as a group?

Best,

AlanM
The Commerce Company
TZ.Com - Travel Zippy

At 11:20 AM 2/27/2008, you wrote:
>Stuart D. Gathman wrote:
>>[An MTA maintainer] *should* be interested in allowing user
>>extensions (via postfix policy daemons or milter api) to modify MAIL FROM.
>
>I'd still be interested in knowing what an API to modify MAIL FROM
>might look like. I'm not quite familiar with neither postfix nor
>sendmail (I use Courier).
>
>I change the MAIL FROM via a -f command line switch, and I do it
>upon delivery, i.e. after accepting responsibility for delivering
>the message. For example, I have ".courier" files (similar to
>".forward") and for SPF compliance I had to substitute lines like
>
> forward@example.com
>
>with
>
> |/usr/sbin/sendmail -f postmaster@mine forward@example.com
>
>Using milter apparently implies doing it earlier. Is that right
>after the MAIL FROM(!?), after RCPT, or after DATA?
>
>Sometimes, I forward based on specific message contents, using
>maildrop, that's the reason why I find it more natural to do it
>after the message has been accepted. What are the pros of doing it "online"?
>
>>Proposals should distance themselves from specific
>>applications. If an example
>>is needed to show why it would be useful to modify MAIL FROM, pick
>>one like BATV, since SRS seems to have accumulated bad vibes in his mind.
>
>Can that be understood in an MTA-independent manner? You already
>mentioned a number of different policies (i.e. BATV, SRS, ecc.) that
>can be applied according with different strategies (i.e. after RCPT,
>after DATA, after acceptance), thus an MTA-wise fragmentation on top
>of that is quite an hindrance for any development.
>
>In other words, if we had a nice program that did The Right Thing
>for each forwarded recipient, how would we fit it into our differing
>MTAs? I assume that, in order to fix forwarding, we aim at designing
>such a program. If not, please tell me what are the alternatives.
>
>TIA
>
>-------------------------------------------
>Sender Policy Framework: http://www.openspf.org
>Archives: http://www.listbox.com/member/archive/735/=now
>RSS Feed: http://www.listbox.com/member/archive/rss/735/
>Modify Your Subscription:
>http://www.listbox.com/member/?&
>Powered by Listbox: http://www.listbox.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://www.listbox.com/member/?member_id=1311532&id_secret=95897010-3d7186
Powered by Listbox: http://www.listbox.com
Re: How to modify the MAIL FROM argument? (was: Re: Top problems with SPF acceptance) [ In reply to ]
On Wed, 27 Feb 2008, Alessandro Vesely wrote:

> Stuart D. Gathman wrote:
> > [An MTA maintainer] *should* be interested in allowing user extensions
> > (via postfix policy daemons or milter api) to modify MAIL FROM.
>
> I'd still be interested in knowing what an API to modify MAIL FROM
> might look like. I'm not quite familiar with neither postfix nor
> sendmail (I use Courier).

https://www.milter.org/developers/api/smfi_chgfrom

https://www.milter.org/developers/api/index

> Using milter apparently implies doing it earlier. Is that right after
> the MAIL FROM(!?), after RCPT, or after DATA?

Right after MAIL FROM, while the SMTP session is in progress, before
the MTA has responded to MAIL FROM. You can also change the result code
and message:

https://www.milter.org/developers/api/smfi_setreply

> Sometimes, I forward based on specific message contents, using
> maildrop, that's the reason why I find it more natural to do it after
> the message has been accepted. What are the pros of doing it "online"?

The biggest pro of doing it "online" is to reject forged or blacklisted
MAIL FROM or HELO immediately, with no need to go to DATA. When using SRS, you
can also reject DSNs (empty MAIL FROM) with no SRS signature in the recipient
without going to data. (These are bounces from otherwise legit MTAs that didn't
check your SPF record before bouncing mail that forges your MAIL FROM.)

>
> > Proposals should distance themselves from specific applications. If an example
> > is needed to show why it would be useful to modify MAIL FROM, pick
> > one like BATV, since SRS seems to have accumulated bad vibes in his mind.
>
> Can that be understood in an MTA-independent manner? You already

Yes, the milter API is MTA-independent, and postfix supports a version
of it. And when they support the latest version, they will support
changing MAIL FROM. Just don't remind "him" that this will also enable
SRS :-)

> mentioned a number of different policies (i.e. BATV, SRS, ecc.) that
> can be applied according with different strategies (i.e. after RCPT,
> after DATA, after acceptance), thus an MTA-wise fragmentation on top
> of that is quite an hindrance for any development.

There is no MTA fragmentation. A milter runs as a separate process
written in any language and communicates with the MTA via a socket.

The other way to achieve the same MTA independence is with an SMTP
proxy. But these introduce timing problems. The milter API handshakes
with the MTA to prevent false timeouts.

> In other words, if we had a nice program that did The Right Thing for
> each forwarded recipient, how would we fit it into our differing MTAs?

If written to the milter API, and the MTAs support milter, just add it
to the milter list.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://www.listbox.com/member/?member_id=1311532&id_secret=95897010-3d7186
Powered by Listbox: http://www.listbox.com
Re: How to modify the MAIL FROM argument? (was: Re: Top problems with SPF acceptance) [ In reply to ]
Stuart D. Gathman wrote:
> On Wed, 27 Feb 2008, Alessandro Vesely wrote:
>>
>> I'd still be interested in knowing what an API to modify MAIL FROM
>> might look like. I'm not quite familiar with neither postfix nor
>> sendmail (I use Courier).
>
> https://www.milter.org/developers/api/smfi_chgfrom

I copy it below for those who are too lazy to click on the above link:

int smfi_chgfrom(SMFICTX *ctx, const char *mail, char *args);

where ctx is an opaque object, mail and args are the replacements for
the MAIL FROM argument and any additional parameters respectively.

>> Using milter apparently implies doing it earlier. Is that right after
>> the MAIL FROM(!?), after RCPT, or after DATA?
>
> Right after MAIL FROM, while the SMTP session is in progress, before
> the MTA has responded to MAIL FROM.

Apparently, libmilter docs differ: the message modification functions
may only be called in xxfi_eom, the end-of-message after xxfi_body.
(This may make sense, as there's no point in changing the recipient if
the message is going to be rejected.)

>> Sometimes, I forward based on specific message contents, using
>> maildrop, that's the reason why I find it more natural to do it after
>> the message has been accepted. What are the pros of doing it "online"?
>
> The biggest pro of doing it "online" is to reject forged or blacklisted
> MAIL FROM or HELO immediately, with no need to go to DATA. When using SRS, you
> can also reject DSNs (empty MAIL FROM) with no SRS signature in the recipient
> without going to data. (These are bounces from otherwise legit MTAs that didn't
> check your SPF record before bouncing mail that forges your MAIL FROM.)

Thanks for the insight. However, I note that most of the advantages
are not directly related with forwarding.

While it is good to perform delivery steps before closing the
transaction with the remote MTA, it might not always be possible.
OTOH, if forwarding addresses are configured as aliases, they may go
directly to the queue and cannot be changed any more. According to the
RFCs, in order to modify the MAIL FROM one should configure a "list".

>> Can that be understood in an MTA-independent manner? You already
>
> Yes, the milter API is MTA-independent, and postfix supports a version
> of it. And when they support the latest version, they will support
> changing MAIL FROM.

On the milter.org site they mention a number of MTAs:
* Open source Sendmail 8.12 and later
* Open source Postfix 2.4 and later
* Sentrionâ„¢ MP appliance from Sendmail Inc.
* Switchâ„¢ from Sendmail Inc.
* Mailstream Managerâ„¢ from Sendmail Inc.

> There is no MTA fragmentation. A milter runs as a separate process
> written in any language and communicates with the MTA via a socket.

Courier-MTA does not support libmilter. It would require major changes
to insert all milter callbacks in a sensible manner. Few posts in the
courier-users mailing list mention adding milter compatbility, e.g.
http://sourceforge.net/mailarchive/message.php?msg_name=455EDC9F.2090603%40tana.it
http://sourceforge.net/mailarchive/message.php?msg_name=45B4FC3B.1010409%40eburg.com

In addition, we should consider widely used commercial MTAs, such as
* Microsoft Exchange,
* Lotus Notes, and
* Novell GroupWise.

I never used any of those. Can anyone briefly mention the filtering
capabilities of any of them?

I report Courier's filtering callbacks here:
* rcpt API: for each RCPT that resolves to a local user,
* local filter: users that registered for it get a call after DATA,
* global filter: after DATA, before accepting it,
* anything can be done on delivery, including forwarding.

For languages, the first two items require maildrop or C; the 3rd is
also done via socket, and generic frameworks exist for Perl and
Python. Delivering may invoke any program, typically maildrop.

> The other way to achieve the same MTA independence is with an SMTP
> proxy. But these introduce timing problems. The milter API handshakes
> with the MTA to prevent false timeouts.

If we use an SMTP proxy, it should intercept _outgoing_ SMTP calls, as
that is what is more relevant for forwarding. My understanding of
milter is that it only operates on _incoming_ connections. However, a
proxy does not look like the simplest solution at a first glance.

>> In other words, if we had a nice program that did The Right Thing for
>> each forwarded recipient, how would we fit it into our differing MTAs?
>
> If written to the milter API, and the MTAs support milter, just add it
> to the milter list.

We'll have to evaluate pros and cons of using milter. We can also go
for a library that can be invoked from a any kind of filter, including
milter. However, filtering is more often associated to incoming calls
and I'm not sure it is always possible to "filter" outgoing mail
transactions. Possibly, interfacing the "list" model of forwarding is
the most portable approach.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://www.listbox.com/member/?member_id=1311532&id_secret=95897010-3d7186
Powered by Listbox: http://www.listbox.com
Re: How to modify the MAIL FROM argument? (was: Re: Top problems with SPF acceptance) [ In reply to ]
On Thu, 28 Feb 2008, Alessandro Vesely wrote:

> In addition, we should consider widely used commercial MTAs, such as
> * Microsoft Exchange,
> * Lotus Notes, and
> * Novell GroupWise.
>
> I never used any of those. Can anyone briefly mention the filtering
> capabilities of any of them?

Microsoft Exchange has Event Sinks that work similarly to the milter API
(a separate process that responds to events and modifies exchange behaviour),
but only in Windows, of course.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://www.listbox.com/member/?member_id=1311532&id_secret=95897010-3d7186
Powered by Listbox: http://www.listbox.com
RE: How to modify the MAIL FROM argument? (was: Re: Top problems with SPF acceptance) [ In reply to ]
Stuart D. Gathman wrote:

> On Thu, 28 Feb 2008, Alessandro Vesely wrote:
> > I never used any of those. Can anyone briefly mention the
> > filtering capabilities of any of them?

> Microsoft Exchange has Event Sinks that work similarly to the
> milter API (a separate process that responds to events and
> modifies exchange behaviour), but only in Windows, of course.

Note that events sinks are discontinued in Exchange Server 2007 and have been replaced by the Transport Agent API, which is a .NET API that (in theory) gives you all necessary functionality. Poking around the SDK, you can easily get access to the From header of the message itself, and Exchange fires an event after the MAIL command (SmtpReceiveAgent.OnMailCommand) that you can handle. However, it's not clear whether you actually have access to change the values in the SMTP session. I've got a query in to find out whether this access is read-only or not.

--
Devin L. Ganger, Exchange MVP Email: deving@3sharp.com
3Sharp Phone: 425.882.1032
14700 NE 95th Suite 210 Cell: 425.239.2575
Redmond, WA 98052 Fax: 425.558.5710
(e)Mail Insecurity: http://blogs.3sharp.com/blog/deving/

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://www.listbox.com/member/?member_id=1311532&id_secret=95897010-3d7186
Powered by Listbox: http://www.listbox.com
RE: How to modify the MAIL FROM argument? (was: Re: Top problems with SPF acceptance) [ In reply to ]
Devin Ganger wrote:

> Stuart D. Gathman wrote:

> > On Thu, 28 Feb 2008, Alessandro Vesely wrote:

> > > I never used any of those. Can anyone briefly mention the
> > > filtering capabilities of any of them?

From what I understand, the filtering capabilities of both Notes
and Groupwise are pretty substandard for this kind of thing. From
my work with them (and I don't claim to be an expert), they are
not designed to be a real mail gateway -- deployments should be
(and commonly are) designed to live behind Sendmail, Postfix, or
some other MTA more suitable for the edge gateway role.

> > Microsoft Exchange has Event Sinks that work similarly to the
> > milter API (a separate process that responds to events and
> > modifies exchange behaviour), but only in Windows, of course.

> Note that events sinks are discontinued in Exchange Server 2007 and
> have been replaced by the Transport Agent API, which is a .NET API
> that (in theory) gives you all necessary functionality. Poking
> around the SDK, you can easily get access to the From header of the
> message itself, and Exchange fires an event after the MAIL command
> (SmtpReceiveAgent.OnMailCommand) that you can handle. However, it's
> not clear whether you actually have access to change the values in
> the SMTP session. I've got a query in to find out whether this
> access is read-only or not.

I've been assured by the appropriate people that the new Transport
Agent API does in fact give the ability to modify the envelope sender.

Which brings up an interesting question: since IronPython allows
Python code to be compiled as .NET assemblies, would there be any
interest at all in helping port the Python SPF library to IronPython?
I'm willing to take the initial stab at seeing what compiles and
what breaks, but I'm no developer, so I'll need help in diagnosing
and fixing any problems.

I know I'd love to have a .NET native SPF implementation to use with
Windows systems.

--
Devin L. Ganger, Exchange MVP Email: deving@3sharp.com
3Sharp Phone: 425.882.1032
14700 NE 95th Suite 210 Cell: 425.239.2575
Redmond, WA 98052 Fax: 425.558.5710
(e)Mail Insecurity: http://blogs.3sharp.com/blog/deving/

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://www.listbox.com/member/?member_id=1311532&id_secret=95897010-3d7186
Powered by Listbox: http://www.listbox.com