Mailing List Archive

java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively
Hi all,

When indexing with multiple threads, and under heavy load, I get the
following exception:

java.io.IOException: Access is denied
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:850)
at org.apache.lucene.store.FSDirectory$1.obtain(FSDirectory.java:363)
at org.apache.lucene.store.Lock.obtain(Lock.java:63)

I am using the Compass framework from Open Symphony, however this simply
sits on top of Lucene and the folks at Open Symphony have assured me that
the access to Lucene indexes within their framework is appropriately
synchronized. In either case, I would have thought Lucene should be
managing its locks regardless of how the indexes are accessed (within the
same JVM at least).

There is a bug report on the Sun website for this issue:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6325169

This appears to be relevant, however it maybe Lucene is not synchronizing
appropriately when obtaining a lock?

The org.apache.lucene.store.FSDirectory implementation in Lucene
2.0.0appears to obtain a file lock thus:

public boolean obtain() throws IOException {
if (disableLocks)
return true;

if (!lockDir.exists()) {
if (!lockDir.mkdirs()) {
throw new IOException("Cannot create lock directory: " +
lockDir);
}
}

return lockFile.createNewFile();
}

Should the line "lockFile.createNewFile();" be within some form of
synchronizion block?

Thanks.
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
Jason Polites wrote:
> Hi all,
>
> When indexing with multiple threads, and under heavy load, I get the
> following exception:
>
> java.io.IOException: Access is denied
> at java.io.WinNTFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:850)
> at org.apache.lucene.store.FSDirectory$1.obtain(FSDirectory.java:363)
> at org.apache.lucene.store.Lock.obtain(Lock.java:63)
>
> I am using the Compass framework from Open Symphony, however this simply
> sits on top of Lucene and the folks at Open Symphony have assured me that
> the access to Lucene indexes within their framework is appropriately
> synchronized. In either case, I would have thought Lucene should be
> managing its locks regardless of how the indexes are accessed (within the
> same JVM at least).

Are you also running searchers against this index? Are they re-init'ing
frequently or being opened and then held open?

This looks similar to http://issues.apache.org/jira/browse/LUCENE-665
(that was just opened) as well as other issues in the past, but in these
cases there are usually also IndexReaders/Searchers against the index.
Could you try the patch to FSDirectory.java (in that issue) and see if
it helps?

> There is a bug report on the Sun website for this issue:
>
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6325169

I think that bug is probably a different issue.

> Should the line "lockFile.createNewFile();" be within some form of
> synchronizion block?

Good question ... however, in your use case (the commits that
IndexWriter is doing) the code above Lock.obtain is already
synchronized(directory) so it's only one thread that can make it down
into the Lock.obtain method. So I don't believe adding synchronization
in this method will help you.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
On 8/26/06, Michael McCandless <lucene@mikemccandless.com> wrote:
>
> Are you also running searchers against this index? Are they re-init'ing
> frequently or being opened and then held open?


No searches running in my initial test, although I can't be certain what is
happening under the Compass hood.

This looks similar to http://issues.apache.org/jira/browse/LUCENE-665
> (that was just opened) as well as other issues in the past, but in these
> cases there are usually also IndexReaders/Searchers against the index.
> Could you try the patch to FSDirectory.java (in that issue) and see if
> it helps?


Does look related, although I'm not sure I like the suggested fix.
Detecting a specific exception is always a tricky deal (if only Sun
described error codes in the spec!), but whilst the solution suggested will
almost certainly fix my problem, it seems a little "loose".

I think that bug is probably a different issue.


Roger that.

Good question ... however, in your use case (the commits that
> IndexWriter is doing) the code above Lock.obtain is already
> synchronized(directory) so it's only one thread that can make it down
> into the Lock.obtain method. So I don't believe adding synchronization
> in this method will help you.


Maybe I'm off-base here, but I'm not sure how you were able to determine
that. I had to double check, but didn't have the original stack trace (only
the snippet I posted). When I looked at the source, it seemed that the
Lock.obtain() method (in my case) is being called by "public Lock
makeLock(String name)" (FSDirectory: line 344); which in turn is called by:

mergeSegments(int, int) - org.apache.lucene.index.IndexWriter (2
matches)
unlock(Directory) - org.apache.lucene.index.IndexReader (2 matches)
IndexWriter(Directory, Analyzer, boolean, boolean) -
org.apache.lucene.index.IndexWriter (2 matches)
open(Directory, boolean) - org.apache.lucene.index.IndexReader
isCurrent() - org.apache.lucene.index.IndexReader
aquireWriteLock() - org.apache.lucene.index.IndexReader
getCurrentVersion(Directory) - org.apache.lucene.index.IndexReader
addIndexes(IndexReader[]) - org.apache.lucene.index.IndexWriter (2
matches)
isLocked(Directory) - org.apache.lucene.index.IndexReader (2 matches)
commit() - org.apache.lucene.index.IndexReader

Not all of which synchronize on the Directory object. In my case, the call
is made by:

org.apache.lucene.index.TransIndex.<init>(TransIndex.java:137)

(which oddly enough is not actually part of Lucene, but part of Compass).

This does not appear to provide any index-relative synchronization.

Synchronization at this low level would ensure that outer application layers
would be guaranteed of IO isolation.

Obviously you shouldn't have to bother. Windows should take care of it, but
in this case it doesn't appear to.

I'll have a deeper look into the Compass source to see if there are any
obvious problem areas in this regard.

Thanks for your help.
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
Jason Polites wrote:

>> Are you also running searchers against this index? Are they re-init'ing
>> frequently or being opened and then held open?
>
>
> No searches running in my initial test, although I can't be certain what is
> happening under the Compass hood.

OK.

> This looks similar to http://issues.apache.org/jira/browse/LUCENE-665
>> (that was just opened) as well as other issues in the past, but in these
>> cases there are usually also IndexReaders/Searchers against the index.
>> Could you try the patch to FSDirectory.java (in that issue) and see if
>> it helps?
>
>
> Does look related, although I'm not sure I like the suggested fix.
> Detecting a specific exception is always a tricky deal (if only Sun
> described error codes in the spec!), but whilst the solution suggested will
> almost certainly fix my problem, it seems a little "loose".
>
> I think that bug is probably a different issue.

Agreed, it's not a very clean fix. I'd really love to get to the root
cause in your case (and from that bug). I'm unable to reproduce that
bug on my Windows XP machine ...

> Good question ... however, in your use case (the commits that
>> IndexWriter is doing) the code above Lock.obtain is already
>> synchronized(directory) so it's only one thread that can make it down
>> into the Lock.obtain method. So I don't believe adding synchronization
>> in this method will help you.
>
>
> Maybe I'm off-base here, but I'm not sure how you were able to determine
> that. I had to double check, but didn't have the original stack trace
> (only
> the snippet I posted). When I looked at the source, it seemed that the
> Lock.obtain() method (in my case) is being called by "public Lock
> makeLock(String name)" (FSDirectory: line 344); which in turn is called
> by:

I'm sorry, you are correct: I had assumed your case was the commit lock
ant it was coming through the standard IndexWriter commit paths (under
ther Lock.With() calls). In fact, even in standard Lucene sources, the
write lock obtain is not synchronized.

> Not all of which synchronize on the Directory object. In my case, the call
> is made by:
>
> org.apache.lucene.index.TransIndex.<init>(TransIndex.java:137)
>
> (which oddly enough is not actually part of Lucene, but part of Compass).
>
> This does not appear to provide any index-relative synchronization.

Ahh, OK. It's interesting that compass acquires the lock directly
itself. That line is actually acquiring the write lock.

> Synchronization at this low level would ensure that outer application
> layers
> would be guaranteed of IO isolation.

Agreed, this really hinges on whether java.io.File.createNewFile is
thread safe. I would assume/hope/expect that it would be (??) but alas
the javadocs don't say one way or another.

> Obviously you shouldn't have to bother. Windows should take care of it,
> but
> in this case it doesn't appear to.

Oh: if you add "synchronized" to the Lock.obtain method, does this in
fact completely fix your issue?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
On 8/26/06, Jason Polites <jason.polites@gmail.com> wrote:
> Synchronization at this low level would ensure that outer application layers
> would be guaranteed of IO isolation.

That still wouldn't solve two JVMs (or even two webapps) trying to
grab the same lock and getting an exception, correct?

It seems like it's possible to immediately get an IOException while
opening an IndexReader or IndexWriter w/o the wait/retry logic kicking
in. So perhaps IOException should be caught by obtain() and false
returned (i.e. treated as a failure to aquire)?

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
I would have thought that simultaneous cross-JVM access to an index was
outside of scope of the core Lucene API (although it would be great), but
maybe the file system basis allows for this (?).

I like the idea of catching IOExceptions and returning false. Conceptually,
failing to obtain a lock due to any reason can be thought of as the same
thing, regardless of the reason (so long as its logged).

Seems like the simplest solution too.

On 8/28/06, Yonik Seeley <yonik@apache.org> wrote:
>
> On 8/26/06, Jason Polites <jason.polites@gmail.com> wrote:
> > Synchronization at this low level would ensure that outer application
> layers
> > would be guaranteed of IO isolation.
>
> That still wouldn't solve two JVMs (or even two webapps) trying to
> grab the same lock and getting an exception, correct?
>
> It seems like it's possible to immediately get an IOException while
> opening an IndexReader or IndexWriter w/o the wait/retry logic kicking
> in. So perhaps IOException should be caught by obtain() and false
> returned (i.e. treated as a failure to aquire)?
>
> -Yonik
> http://incubator.apache.org/solr Solr, the open-source Lucene search
> server
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
"Jason Polites" <jason.polites@gmail.com> wrote on 27/08/2006 09:36:07:

> I would have thought that simultaneous cross-JVM access to an index was
> outside of scope of the core Lucene API (although it would be great), but
> maybe the file system basis allows for this (?).

Lucene does protect you from mis-accessing its indexes from separate jvms
or even, to a certain extent, separate machines.

>
> I like the idea of catching IOExceptions and returning false.
Conceptually,
> failing to obtain a lock due to any reason can be thought of as the same
> thing, regardless of the reason (so long as its logged).
>
> Seems like the simplest solution too.

I agree with this - as this would be a more correct behavior when the lock
could not be obtained.

However it would not handle the problem that I had, and I think you had too
- that due to some temporary situation in the OS/FS there is a temporary
inability to perform an action that should be possible - i.e. creating a
new file, or renaming another. Application writers would be "surprised" to
be having to deal with this failure to obtain a lock with no "expected"
reason.

Since this painful situation is evidently possible, and since this is so
frustrating for Lucene users to get their index corrupted in this case, I
still think it is worth the effort of spending some 30ms or so on the rare
occasions that this happen, and retry, trying to bypass this temporary
obstacle.

Did you try that patch? Does it work for you?

I plan to submit an update to that patch later today accommodating your
comments (and others); It will most probably retry for IOExceptions (not
analyzing the exception text); also, it would return false if the *retry*
for obtain() failed with exception.

If you can try out this patch it would be very valuable information to know
whether it prevents the problem or not.

Thanks,
Doron


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
On 8/27/06, Doron Cohen <DORONC@il.ibm.com> wrote:
> I plan to submit an update to that patch later today accommodating your
> comments (and others); It will most probably retry for IOExceptions (not
> analyzing the exception text); also, it would return false if the *retry*
> for obtain() failed with exception.

Lock.obtain(timeout) already has retry/timeout logic around
Lock.obtain(), so wouldn't the change to return false on an
IOException be sufficient? User level applications should then use
obtain(timeout) rather than obtain() unless they want to do
timeout/retry logic themselves.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
Yonik Seeley wrote:
> On 8/27/06, Doron Cohen <DORONC@il.ibm.com> wrote:
>> I plan to submit an update to that patch later today accommodating your
>> comments (and others); It will most probably retry for IOExceptions (not
>> analyzing the exception text); also, it would return false if the *retry*
>> for obtain() failed with exception.
>
> Lock.obtain(timeout) already has retry/timeout logic around
> Lock.obtain(), so wouldn't the change to return false on an
> IOException be sufficient? User level applications should then use
> obtain(timeout) rather than obtain() unless they want to do
> timeout/retry logic themselves.

I would agree: the retry/timeout logic already exists above so let's
just return false on hitting any IOException when calling createNewFile.

(Doron, this still wouldn't fix the other strange "across the board" IO
problems you see on Windows. I'd really like to reproduce this but so
far (running your stress test) I can't -- are you sure it's not a virus
checker or something?).

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
Doron Cohen wrote:
> "Jason Polites" <jason.polites@gmail.com> wrote on 27/08/2006 09:36:07:
>
>> I would have thought that simultaneous cross-JVM access to an index was
>> outside of scope of the core Lucene API (although it would be great), but
>> maybe the file system basis allows for this (?).
>
> Lucene does protect you from mis-accessing its indexes from separate jvms
> or even, to a certain extent, separate machines.

I would strengthen this statement: it is within scope of Lucene to allow
cross-JVM access to the same underlying index. Lucene has file-based
locks for precisely this reason. Lucene in Action (the book) has a
great discussion about concurrency.

It is also within scope for cross-JVM access when the JVMs are running
on different machines, however, there are known problems (related to
locking) when the index resides on an NFS mounted filesystems (and
possibly other remote-mounted filesystems). We are trying to address
these known issues (see java-dev for the gory details) but it's not
clear when these fixes will become available.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
Wow. That's awesome.

I had just assumed (foolishly) that cross-JVM access would be problematic.
Maybe I should read the manual ;)

Cross machine access could be solved with some RMI magic.. but
performance/scalability may be an issue.

On 8/28/06, Michael McCandless <lucene@mikemccandless.com> wrote:
>
> Doron Cohen wrote:
> > "Jason Polites" <jason.polites@gmail.com> wrote on 27/08/2006 09:36:07:
> >
> >> I would have thought that simultaneous cross-JVM access to an index was
> >> outside of scope of the core Lucene API (although it would be great),
> but
> >> maybe the file system basis allows for this (?).
> >
> > Lucene does protect you from mis-accessing its indexes from separate
> jvms
> > or even, to a certain extent, separate machines.
>
> I would strengthen this statement: it is within scope of Lucene to allow
> cross-JVM access to the same underlying index. Lucene has file-based
> locks for precisely this reason. Lucene in Action (the book) has a
> great discussion about concurrency.
>
> It is also within scope for cross-JVM access when the JVMs are running
> on different machines, however, there are known problems (related to
> locking) when the index resides on an NFS mounted filesystems (and
> possibly other remote-mounted filesystems). We are trying to address
> these known issues (see java-dev for the gory details) but it's not
> clear when these fixes will become available.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
: I had just assumed (foolishly) that cross-JVM access would be problematic.

nope .. the whole point of the lockfiles is to deal with multiple JVMs ...
otherwise Lucene locking could be accomplished entirely with
synchronization.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
Yeah.. I had a think about this, and I now remember why I originally came to
the conclusion about cross-JVM access.

When I was adding documents to the index, and searching at the same time
(from a different JVM) I would get the occassional (but regular)
FileNotFoundException.

I don't recall the exact stack trace, but I put it down to cross JVM access
at the time...

On 8/28/06, Chris Hostetter <hossman_lucene@fucit.org> wrote:
>
>
> : I had just assumed (foolishly) that cross-JVM access would be
> problematic.
>
> nope .. the whole point of the lockfiles is to deal with multiple JVMs ...
> otherwise Lucene locking could be accomplished entirely with
> synchronization.
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
Jason Polites wrote:
> Yeah.. I had a think about this, and I now remember why I originally
> came to
> the conclusion about cross-JVM access.
>
> When I was adding documents to the index, and searching at the same time
> (from a different JVM) I would get the occassional (but regular)
> FileNotFoundException.
>
> I don't recall the exact stack trace, but I put it down to cross JVM access
> at the time...

Was this with an index accessed via NFS or Samba or other remote
filesystem? I'd be interested in getting the details if you still have
/ remember them.

We very much want to get use cases like to work correctly but there are
known issues with locking against at least NFS.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
It was definately NTFS, unfortunately it was a while ago, and most of the
code has changed.

Basically I had a multi-threaded app where multiple threads were writing to
the index (but exclusively... that is, I had my own locking mechanism
preventing concurrent writes).

In a separate JVM, I had a fairly standard IndexSearcher based component via
a web interface. Rapid hits on "refresh" in the browser executing a search
while indexing was being performed resulted in occasional but recurring
FileNotFound errors.

I recall that it was a "segments" file reported as not found.. if that
helps.

On 8/28/06, Michael McCandless <lucene@mikemccandless.com> wrote:
>
> Jason Polites wrote:
> > Yeah.. I had a think about this, and I now remember why I originally
> > came to
> > the conclusion about cross-JVM access.
> >
> > When I was adding documents to the index, and searching at the same time
> > (from a different JVM) I would get the occassional (but regular)
> > FileNotFoundException.
> >
> > I don't recall the exact stack trace, but I put it down to cross JVM
> access
> > at the time...
>
> Was this with an index accessed via NFS or Samba or other remote
> filesystem? I'd be interested in getting the details if you still have
> / remember them.
>
> We very much want to get use cases like to work correctly but there are
> known issues with locking against at least NFS.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively [ In reply to ]
Jason Polites wrote:
> It was definately NTFS, unfortunately it was a while ago, and most of the
> code has changed.
>
> Basically I had a multi-threaded app where multiple threads were writing to
> the index (but exclusively... that is, I had my own locking mechanism
> preventing concurrent writes).
>
> In a separate JVM, I had a fairly standard IndexSearcher based component
> via
> a web interface. Rapid hits on "refresh" in the browser executing a search
> while indexing was being performed resulted in occasional but recurring
> FileNotFound errors.
>
> I recall that it was a "segments" file reported as not found.. if that
> helps.

OK thanks for the info. It's odd that you saw this with a local NTFS
drive. I was under the impression that locking worked correctly in that
case. Lucene's commit lock is supposed to prevent a searcher from
init'ing while a writer is writing a new segments file.

One of the things in process (see java-dev for gory details) is to
switch to the OS native locking that's now available through java.nio.
Currently Lucene uses java.io.File.createNewFile for locking and the
javadoc for that method has a spooky warning about not using it for locking.

Mike


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org