Would some sort of caching strategy work? How big is your overall
Also, lately there have been a few threads on TV (term
performance. I don't recall anyone having actively profiled or
examined it for improvements, so perhaps that would be helpful.
Another thought: could you have a stored field that contains the top
X terms for a given document with their freqs and then just do a
merge based on your hit results? Part of the problem w/ TVs is that
not only do you have to load them, but then you have to iterate
through them to sort them by frequency
. I could see that it might be
beneficial to have alternate strategies for loading them, say into a
map of terms -> frequencies or terms to TVInfo (freqs, offsets,
positions) or parallel arrays sorted by frequency
or something like
It _might_ be possible to do this in a HitCollector or FieldSelector
style way. This way, perhaps, you could build the TV structure you
want as it is read from disk. Do you have any interest in digging
down into the Lucene code to help on such an idea?
On Apr 10, 2007, at 9:38 PM, Sengly Heng wrote: > Once again, thank you for your help.
>> >> We don't really know what your problem is. Explaining that rathern
>> >> than the solution you have thought of might render a couple of
>> >> alternate solutions. Perhaps something could be precalculated and
>> >> stored in the documents. Perhaps feature selection (reduction)
>> of the
>> >> terms might do the trick for you. And so on.
>> > I have a corpus of documents indexed with different fields.
>> > Approximately
>> > each document indexed has an average of 30 fields. Each field has
>> > about 100
>> > terms.
>> > Normally, the hit will return less than 100 documents. For each of
>> > the 30
>> > fields of the documents, I have to calculate the top 35 keywords
>> > from all
>> > the documents as well as the top 30 popular keywords (the keywords
>> > that are
>> > distributed in many documents - something like docFreq or IDF).
>> Right, but /why/ do you need these values? Do you present them as
>> they are, or do you use them for some secondary calculation? Then
>> what is the result of this secondary calculation?
> Yes, I just want those values as they are. No second calculation is
> to be
>> Please let me know if you are still have some more questions.
>> I'll reask of of the questions I placed in my previous reply:
>> >> How slow is it, and how fast did you expect it to be?
> We expect to get those values sorted as fast as possible. Currently
> for 100
> documents, the process is about 1~1.5 minutes. I believe this is
> because of
> the loop.
>>> Can you limit the evaulation to the top n documents?
> Yes, we limit to only the top 100 documents.
> Thank you.
Center for Natural Language Processing http://www.cnlp.org
Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/
To unsubscribe, e-mail: email@example.com
For additional commands, e-mail: firstname.lastname@example.org