Mailing List Archive

File delete rate
Apologies if this seems a bit vague - I'm in need of a quick sanity check.
Doing a qtree delete - it's taking 'a while'.
This seems to be because it's deleting every file at a rate of 50-100 files
per second, and there's about a million files in the qtree.

Is that a 'reasonable' rate of deletion? (This is using an API based
'file-delete-file' operation, if that's relevant).
Re: File delete rate [ In reply to ]
Yes, that seems possible. Operations working on large numbers of small files always take a long time.
Sent from my BlackBerry 10 smartphone on the Bell network. From: Edward RolisonSent: Friday, September 25, 2015 8:00 AMTo: toasters@teaparty.netSubject: File delete rate
Apologies if this seems a bit vague - I'm in need of a quick sanity check.
Doing a qtree delete - it's taking 'a while'.
This seems to be because it's deleting every file at a rate of 50-100 files per second, and there's about a million files in the qtree.
Is that a 'reasonable' rate of deletion? (This is using an API based 'file-delete-file' operation, if that's relevant).
Re: File delete rate [ In reply to ]
Bonus question would be - does that also apply if you do it via a "qtree
delete -f" or is the API delete run at about the same speed?

On 25 September 2015 at 13:31, <basilberntsen@gmail.com> wrote:

> Yes, that seems possible. Operations working on large numbers of small
> files always take a long time.
>
> Sent from my BlackBerry 10 smartphone on the Bell network.
> *From: *Edward Rolison
> *Sent: *Friday, September 25, 2015 8:00 AM
> *To: *toasters@teaparty.net
> *Subject: *File delete rate
>
> Apologies if this seems a bit vague - I'm in need of a quick sanity check.
> Doing a qtree delete - it's taking 'a while'.
> This seems to be because it's deleting every file at a rate of 50-100
> files per second, and there's about a million files in the qtree.
>
> Is that a 'reasonable' rate of deletion? (This is using an API based
> 'file-delete-file' operation, if that's relevant).
>
>
Re: File delete rate [ In reply to ]
Pretty sure the only shortcut‎ is volume operations like qtree snapmirror.
Sent from my BlackBerry 10 smartphone on the Bell network. From: Edward RolisonSent: Friday, September 25, 2015 8:48 AMTo: Basil BCc: toasters@teaparty.netSubject: Re: File delete rate
Bonus question would be - does that also apply if you do it via a "qtree delete -f" or is the API delete run at about the same speed?
On 25 September 2015 at 13:31, <basilberntsen@gmail.com> wrote:
Yes, that seems possible. Operations working on large numbers of small files always take a long time.
Sent from my BlackBerry 10 smartphone on the Bell network. From: Edward RolisonSent: Friday, September 25, 2015 8:00 AMTo: toasters@teaparty.netSubject: File delete rate
Apologies if this seems a bit vague - I'm in need of a quick sanity check.
Doing a qtree delete - it's taking 'a while'.
This seems to be because it's deleting every file at a rate of 50-100 files per second, and there's about a million files in the qtree.
Is that a 'reasonable' rate of deletion? (This is using an API based 'file-delete-file' operation, if that's relevant).
Re: File delete rate [ In reply to ]
I wonder if (like in Unix), as the number of files left drops, the rate of
deletion will increase or go faster. I found in one case that the time was
being taken in reading then re-writing the filesystem directory index of
files. As the number dropped from 100,000+ files the rate of deletes
increased.

On Fri, Sep 25, 2015 at 7:19 AM, <basilberntsen@gmail.com> wrote:

> Pretty sure the only shortcut‎ is volume operations like qtree snapmirror.
>
> Sent from my BlackBerry 10 smartphone on the Bell network.
> *From: *Edward Rolison
> *Sent: *Friday, September 25, 2015 8:48 AM
> *To: *Basil B
> *Cc: *toasters@teaparty.net
> *Subject: *Re: File delete rate
>
> Bonus question would be - does that also apply if you do it via a "qtree
> delete -f" or is the API delete run at about the same speed?
>
> On 25 September 2015 at 13:31, <basilberntsen@gmail.com> wrote:
>
>> Yes, that seems possible. Operations working on large numbers of small
>> files always take a long time.
>>
>> Sent from my BlackBerry 10 smartphone on the Bell network.
>> *From: *Edward Rolison
>> *Sent: *Friday, September 25, 2015 8:00 AM
>> *To: *toasters@teaparty.net
>> *Subject: *File delete rate
>>
>> Apologies if this seems a bit vague - I'm in need of a quick sanity
>> check.
>> Doing a qtree delete - it's taking 'a while'.
>> This seems to be because it's deleting every file at a rate of 50-100
>> files per second, and there's about a million files in the qtree.
>>
>> Is that a 'reasonable' rate of deletion? (This is using an API based
>> 'file-delete-file' operation, if that's relevant).
>>
>>
>
>
>
> _______________________________________________
> Toasters mailing list
> Toasters@teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters
>
>
Re: File delete rate [ In reply to ]
It's probably a single thread process to do the deletions.
If possible, from a client, start multiple deletions deep in the tree.
I have had to do this trick with deletions and with rsync in the past
Forever the multithreaded-ness 

Sent from Mobile Outlook




On Sat, Sep 26, 2015 at 9:16 AM -0700, "Patrick Giagnocavo" <xemacs5@gmail.com> wrote:










I wonder if (like in Unix), as the number of files left drops, the rate of deletion will increase or go faster.  I found in one case that the time was being taken in reading then re-writing the filesystem directory index of files.  As the number dropped from 100,000+ files the rate of deletes increased.

On Fri, Sep 25, 2015 at 7:19 AM, <basilberntsen@gmail.com> wrote:
Pretty sure the only shortcut‎ is volume operations like qtree snapmirror. 
Sent from my BlackBerry 10 smartphone on the Bell network. From: Edward RolisonSent: Friday, September 25, 2015 8:48 AMTo: Basil BCc: toasters@teaparty.netSubject: Re: File delete rate
Bonus question would be - does that also apply if you do it via a "qtree delete -f" or is the API delete run at about the same speed? 
On 25 September 2015 at 13:31, <basilberntsen@gmail.com> wrote:
Yes, that seems possible. Operations working on large numbers of small files always take a long time.
Sent from my BlackBerry 10 smartphone on the Bell network. From: Edward RolisonSent: Friday, September 25, 2015 8:00 AMTo: toasters@teaparty.netSubject: File delete rate
Apologies if this seems a bit vague - I'm in need of a quick sanity check. 
Doing a qtree delete - it's taking 'a while'. 
This seems to be because it's deleting every file at a rate of 50-100 files per second, and there's about a million files in the qtree.
Is that a 'reasonable' rate of deletion? (This is using an API based 'file-delete-file' operation, if that's relevant). 








_______________________________________________

Toasters mailing list

Toasters@teaparty.net

http://www.teaparty.net/mailman/listinfo/toasters