Mailing List Archive

robots.txt with httpd and root level SetHandler perl-script
Hi,

I'd like to have a robots.txt on a site that has the following
apache httpd config:

<Location />
SetHandler perl-script
PerlHandler RT::Mason
</Location>

But if I install a robots.txt to the DocumentRoot and test it via wget
I just download the front page of the site, as its handled by
perl-script. It it possible to have a robots.txt in this situation?

thanks for any tips, Andy.
Re: robots.txt with httpd and root level SetHandler perl-script [ In reply to ]
a.smith@ukgrid.net wrote:
> Hi,
>
> I'd like to have a robots.txt on a site that has the following apache
> httpd config:
>
> <Location />
> SetHandler perl-script
> PerlHandler RT::Mason
> </Location>
>
> But if I install a robots.txt to the DocumentRoot and test it via wget I
> just download the front page of the site, as its handled by perl-script.
> It it possible to have a robots.txt in this situation?
>
> thanks for any tips, Andy.
>
Ideas :
1) Try a <FilesMatch ^robots\.txt$> section inside the above section, to reset the handler
to the default Apache (that may not be so easy)
2) Create a Mason handler to handle the URL "robots.txt" and return the file "as is"
Re: robots.txt with httpd and root level SetHandler perl-script [ In reply to ]
> a.smith@ukgrid.net wrote:
> > Hi,
> >
> > I'd like to have a robots.txt on a site that has the following apache
> > httpd config:
> >
> > <Location />
> > SetHandler perl-script
> > PerlHandler RT::Mason
> > </Location>
> >
> > But if I install a robots.txt to the DocumentRoot and test it via wget I
> > just download the front page of the site, as its handled by perl-script.
> > It it possible to have a robots.txt in this situation?
> >
> > thanks for any tips, Andy.
> >
> Ideas :
> 1) Try a <FilesMatch ^robots\.txt$> section inside the above section, to reset the handler
> to the default Apache (that may not be so easy)

It may be easier to have the files match only the needed suffixes
(e.g., .pl, .perl, .html, etc.), otherwise you may need to exclude
other types in the future such as .css, .js, .png, .jpg, .jpeg, .gif,
.ico, etc.

> 2) Create a Mason handler to handle the URL "robots.txt" and return the file "as is"

Be careful here -- you may need to return it with the correct
content type of "text/plain" since "as-is" may be for raw file output
(including HTTP headers).

Randolf Richardson - randolf@inter-corporate.com
Inter-Corporate Computer & Network Services, Inc.
Vancouver, British Columbia, Canada
http://www.inter-corporate.com
Re: robots.txt with httpd and root level SetHandler perl-script [ In reply to ]
On Monday, March 14, 2011 14:53:35 André Warnier wrote:
> > I'd like to have a robots.txt on a site that has the following apache
> >
> > httpd config:
> >
> > <Location />
> > SetHandler perl-script
> > PerlHandler RT::Mason
> > </Location>
> >
> >
> > But if I install a robots.txt to the DocumentRoot and test it via wget I
> > just download the front page of the site, as its handled by perl-script.
> > It it possible to have a robots.txt in this situation?
> >
> >
> >
> > thanks for any tips, Andy.
> >
> >
>
> Ideas :
> 1) Try a <FilesMatch ^robots\.txt$> section inside the above section, to
> reset the handler to the default Apache (that may not be so easy)

SetHandler None (imho)

> 2) Create a Mason handler to handle the URL "robots.txt" and return the
> file "as is"

or instead of the Location block:

PerlMapToStorageHandler "sub { \
use Apache2::Const -compile=>DECLINED; \
use Apache2::RequestRec (); \
use Apache2::RequestUtil (); \
unless( $_[0]->uri eq '/robots.txt' ) { \
$_[0]->add_config(['SetHandler perl-script', \
'PerlHandler RT::Mason']); \
} \
return Apache2::Const::DECLINED; \
}"

Torsten Förtsch

--
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net
Re: robots.txt with httpd and root level SetHandler perl-script [ In reply to ]
Quoting Torsten Förtsch <torsten.foertsch@gmx.net>:

> or instead of the Location block:
>
> PerlMapToStorageHandler "sub { \
> use Apache2::Const -compile=>DECLINED; \
> use Apache2::RequestRec (); \
> use Apache2::RequestUtil (); \
> unless( $_[0]->uri eq '/robots.txt' ) { \
> $_[0]->add_config(['SetHandler perl-script', \
> 'PerlHandler RT::Mason']); \
> } \
> return Apache2::Const::DECLINED; \
> }"

Thanks Torsten (and all). So in this situation do I put this code in
httpd.conf in between the <Location> </Location> lines? Just to check,
as you said "instead of the Location block"?

thanks Andy.
Re: robots.txt with httpd and root level SetHandler perl-script [ In reply to ]
On Monday, March 14, 2011 18:45:39 a.smith@ukgrid.net wrote:
> Quoting Torsten Förtsch <torsten.foertsch@gmx.net>:
> > or instead of the Location block:
> >
> > PerlMapToStorageHandler "sub { \
> >
> > use Apache2::Const -compile=>DECLINED; \
> > use Apache2::RequestRec (); \
> > use Apache2::RequestUtil (); \
> > unless( $_[0]->uri eq '/robots.txt' ) { \
> > $_[0]->add_config(['SetHandler perl-script', \
> > 'PerlHandler RT::Mason']); \
> > } \
> > return Apache2::Const::DECLINED; \
> >
> > }"
>
> Thanks Torsten (and all). So in this situation do I put this code in
> httpd.conf in between the <Location> </Location> lines? Just to check,
> as you said "instead of the Location block"?

A PerlMapToStorageHandler inside a Location block doesn't make sense.

See
http://perl.apache.org/docs/2.0/user/config/config.html#mod_perl_Directives_Argument_Types_and_Allowed_Location

BTW, it should read

use Apache2::Const -compile=>'DECLINED';

Torsten Förtsch

--
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net
Re: robots.txt with httpd and root level SetHandler perl-script [ In reply to ]
Quoting Torsten Förtsch <torsten.foertsch@gmx.net>:

>
> A PerlMapToStorageHandler inside a Location block doesn't make sense.
>
> See
> http://perl.apache.org/docs/2.0/user/config/config.html#mod_perl_Directives_Argument_Types_and_Allowed_Location
>
> BTW, it should read
>
> use Apache2::Const -compile=>'DECLINED';
>

Hi,

so, if I have other <Location> blocks, will setting as you suggest
override all of them? That would break other parts of the httpd
virtual host on my config...

thanks Andy.
Re: robots.txt with httpd and root level SetHandler perl-script [ In reply to ]
On Monday, March 14, 2011 19:14:35 a.smith@ukgrid.net wrote:
> if I have other <Location> blocks, will setting as you suggest
> override all of them? That would break other parts of the httpd
> virtual host on my config...

no, Location blocks are evaluated *after* a PerlMapToStorageHandler. You can
also try

<Location /robots.txt>
SetHandler None
</Location>

Torsten Förtsch

--
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net