Mailing List Archive

You can safely reuse a single analyzer across threads. The Analyzer
class maintains ThreadLocal storage for TokenStreams internally so you
can just create the analyzer once and use it throughout your
application.

simon

On Thu, Jul 12, 2012 at 10:13 PM, Dave Seltzer <dseltzer@tveyes.com> wrote:
> I have one more question to pose to the group today:
>
> I have several thousand searches being performed against MemoryIndexes on
> a regular basis.
>
> I'd like the ability for each search to choose it's own Analyzer, such
> that some queries could use a regex pattern, other queries could use the
> Standard Analyzer.
>
> Does anyone know how expensive it is to create an Analyzer? Am I better
> off creating analyzers as needed, or should I store them once and re-use
> them?
>
> Thanks very much!
>
> -Dave
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Hi Simon,

I'm trying to reuse a custom analyzer and it's not working unless I manually
call reset() on the TokenStream. Basically the analyzer will work on the
first string, but complete fail on any string after that. The weird part is
that this is only necessary when using the SynonymFilter.

I wrote a short piece of code that shows what's going on. Let me know if you
want me to post the code. But it's more likely that I don't understand what
I'm doing - I am new to Lucene.

Thank you,

-ricardo



--
View this message in context: http://lucene.472066.n3.nabble.com/Is-creating-an-analyzer-expensive-tp3994731p4001878.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

You have to use the TokenStream retrieved by Analyzer in the specified
order, otherwise it will not work correctly and will behave as described by
you:

reset()
while (incrementToken())
end()
close()

You have to call reset() also when using for first time! That's specified in
the specs. If you do this, it will work as expected.
See:
https://builds.apache.org/job/Lucene-Artifacts-4.x/javadoc/core/org/apache/l
ucene/analysis/TokenStream.html
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: rrs [mailto:rrs@rand.org]
> Sent: Friday, August 17, 2012 6:38 PM
> To: java-user@lucene.apache.org
> Subject: Re: Is creating an analyzer expensive?
>
> Hi Simon,
>
> I'm trying to reuse a custom analyzer and it's not working unless I
manually call
> reset() on the TokenStream. Basically the analyzer will work on the first
string,
> but complete fail on any string after that. The weird part is that this is
only
> necessary when using the SynonymFilter.
>
> I wrote a short piece of code that shows what's going on. Let me know if
you
> want me to post the code. But it's more likely that I don't understand
what I'm
> doing - I am new to Lucene.
>
> Thank you,
>
> -ricardo
>
>
>
> --
> View this message in context:
http://lucene.472066.n3.nabble.com/Is-creating-
> an-analyzer-expensive-tp3994731p4001878.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org