Mailing List Archive

[Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ?
Hi folks ,

I'm thinking about the better approach to manage "an user" or "an account"
space usage quota in swift.
Is there any related blueprint or sub-project even an idea around ?
Any suggestion of benefits to be an external service or to be a middle-ware
in swift-proxy ?

I'm concerning about such feature will reduce the performance of entire
Swift environment.

Appreciate :>



--
+Hugo Kuo+
tonytkdk@gmail.com
+ <tonytkdk@gmail.com>886 935004793
Re: [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ? [ In reply to ]
Hi Kuo,

One option would be to keep the usage information (num files, num bytes,
etc) per container / account in an sqlite DB, just like it is done for
account and container info.

To avoid having to loop through all data at regular intervals (to update
the info), additional logic could be added to the api methods to update the
sqlite DB's when new files are added, files are deleted, etc. Such approach
will require more lines of code, but will be far less stressful on
performance.

(the brute-force approach to loop through it at regular intervals will be
hell on performance on large deployments..)


For data transfer billing based on download / upload amounts, a similar
approach could be used.

If no one else is looking into this, I would certainly be willing to help
to help get this started.


Kind regards,
Frederik Van Hecke

*T:* +32487733713
*E:* frederik@cluttr.be
*W:* www.cluttr.be



*This e-mail and any attachments thereto may contain information which
is confidential and/or protected by intellectual property rights and
are intended for the sole use of the recipient(s)named above. Any use
of the information contained herein (including, but not limited to,
total or partial reproduction, communication or distribution in any
form) by persons other than the designated recipient(s) is prohibited.
If you have received this e-mail in error, please notify the sender
either by telephone or by e-mail and delete the material from any
computer. Thank you for your cooperation.*




On Thu, Apr 12, 2012 at 17:45, Kuo Hugo <tonytkdk@gmail.com> wrote:

> Hi folks ,
>
> I'm thinking about the better approach to manage "an user" or "an account"
> space usage quota in swift.
> Is there any related blueprint or sub-project even an idea around ?
> Any suggestion of benefits to be an external service or to be a
> middle-ware in swift-proxy ?
>
> I'm concerning about such feature will reduce the performance of entire
> Swift environment.
>
> Appreciate :>
>
>
>
> --
> +Hugo Kuo+
> tonytkdk@gmail.com
> + <tonytkdk@gmail.com>886 935004793
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>
Re: [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ? [ In reply to ]
Swift keeps total bytes, container, and object count (eventually) up-to-date in the account metadata. There are also log processing tools (like slogging - http://github.com/notmyname/slogging) that can provide usage information (including bandwidth) based on swift logs.

While I think that it's appropriate for swift to generate the usage information (via internal processes or log processing), the appropriate place for quotas is in whatever system handles the concept of a user (normally the auth system). This way quotas are enforced by revoking or limiting access of the auth token.

--John


On Apr 12, 2012, at 11:53 AM, Frederik Van Hecke wrote:

> Hi Kuo,
>
> One option would be to keep the usage information (num files, num bytes, etc) per container / account in an sqlite DB, just like it is done for account and container info.
>
> To avoid having to loop through all data at regular intervals (to update the info), additional logic could be added to the api methods to update the sqlite DB's when new files are added, files are deleted, etc. Such approach will require more lines of code, but will be far less stressful on performance.
>
> (the brute-force approach to loop through it at regular intervals will be hell on performance on large deployments..)
>
>
> For data transfer billing based on download / upload amounts, a similar approach could be used.
>
> If no one else is looking into this, I would certainly be willing to help to help get this started.
>
>
> Kind regards,
> Frederik Van Hecke
>
> T: +32487733713
> E: frederik@cluttr.be
> W: www.cluttr.be
>
>
>
>
>
> This e-mail and any attachments thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the recipient(s)named above. Any use of the information contained herein (including, but not limited to, total or partial reproduction, communication or distribution in any form) by persons other than the designated recipient(s) is prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you for your cooperation.
>
>
>
> On Thu, Apr 12, 2012 at 17:45, Kuo Hugo <tonytkdk@gmail.com> wrote:
> Hi folks ,
>
> I'm thinking about the better approach to manage "an user" or "an account" space usage quota in swift.
> Is there any related blueprint or sub-project even an idea around ?
> Any suggestion of benefits to be an external service or to be a middle-ware in swift-proxy ?
>
> I'm concerning about such feature will reduce the performance of entire Swift environment.
>
> Appreciate :>
>
>
>
> --
> +Hugo Kuo+
> tonytkdk@gmail.com
> +886 935004793
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
Re: [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ? [ In reply to ]
I should also mention the summit session talking about this very topic led by Everett Toews. It's (currently) scheduled for 9am on wednesday.

http://summit.openstack.org/sessions/view/81

--John



On Apr 12, 2012, at 8:51 PM, John Dickinson wrote:

> Swift keeps total bytes, container, and object count (eventually) up-to-date in the account metadata. There are also log processing tools (like slogging - http://github.com/notmyname/slogging) that can provide usage information (including bandwidth) based on swift logs.
>
> While I think that it's appropriate for swift to generate the usage information (via internal processes or log processing), the appropriate place for quotas is in whatever system handles the concept of a user (normally the auth system). This way quotas are enforced by revoking or limiting access of the auth token.
>
> --John
>
>
> On Apr 12, 2012, at 11:53 AM, Frederik Van Hecke wrote:
>
>> Hi Kuo,
>>
>> One option would be to keep the usage information (num files, num bytes, etc) per container / account in an sqlite DB, just like it is done for account and container info.
>>
>> To avoid having to loop through all data at regular intervals (to update the info), additional logic could be added to the api methods to update the sqlite DB's when new files are added, files are deleted, etc. Such approach will require more lines of code, but will be far less stressful on performance.
>>
>> (the brute-force approach to loop through it at regular intervals will be hell on performance on large deployments..)
>>
>>
>> For data transfer billing based on download / upload amounts, a similar approach could be used.
>>
>> If no one else is looking into this, I would certainly be willing to help to help get this started.
>>
>>
>> Kind regards,
>> Frederik Van Hecke
>>
>> T: +32487733713
>> E: frederik@cluttr.be
>> W: www.cluttr.be
>>
>>
>>
>>
>>
>> This e-mail and any attachments thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the recipient(s)named above. Any use of the information contained herein (including, but not limited to, total or partial reproduction, communication or distribution in any form) by persons other than the designated recipient(s) is prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you for your cooperation.
>>
>>
>>
>> On Thu, Apr 12, 2012 at 17:45, Kuo Hugo <tonytkdk@gmail.com> wrote:
>> Hi folks ,
>>
>> I'm thinking about the better approach to manage "an user" or "an account" space usage quota in swift.
>> Is there any related blueprint or sub-project even an idea around ?
>> Any suggestion of benefits to be an external service or to be a middle-ware in swift-proxy ?
>>
>> I'm concerning about such feature will reduce the performance of entire Swift environment.
>>
>> Appreciate :>
>>
>>
>>
>> --
>> +Hugo Kuo+
>> tonytkdk@gmail.com
>> +886 935004793
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>
[Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ? [ In reply to ]
In my view, the biggest problem of swift is not the new features but the
improvement of performance.

At first, we knew that container-updater, *-auditor, *-replicator will loop
all the files in the disk during the running interval. When the number of
files is very large, the process of replicator, updater and auditor will
spend much time. So, the time of eventual consistency is very long.

Our practice of Sina Web Service Team https://launchpad.net/~sws:

total accounts: 121,961;

total containers: 160,703;

total objects: 14,291,519;

total storage usage: 1.3T

account replication time: 10 hours;

container replication time: 10 hours;

object replication time: 48 hours;

account audit time: 2 hours;

container audit time: 9 hours;

container update time: 19 hours;

This is terrible If we develop quota upon account db. There is long time
for eventual consistency.

Secondly, there is a vicious circle during replication. The replicator
will query the account-server, container-server and object-server to
compare the metadata, and determine whether to sync. When the number of
files is very large, the frequent query make the account-server,
container-server and object-server become a bottle neck. This will
influence the process of proxy-server to work with back-end servers. There
a lot of Timeout(10s) ERROR in proxy-server's log and the load-average is
very high.
So, some PUT, POST operations failed, and the replicator to sync, and fail
more, and sync more...
.
In my opinion, we need to improve the process of replication and container
update by using event drive framework or something else..
My leader may discuss this topic at Design Summit,
http://openstackconferencespring2012.sched.org/speaker/huicheng

2012/4/13 John Dickinson <me@not.mn <javascript:_e({}, 'cvml',
'me@not.mn');>>

> Swift keeps total bytes, container, and object count (eventually)
> up-to-date in the account metadata. There are also log processing tools
> (like slogging - http://github.com/notmyname/slogging) that can provide
> usage information (including bandwidth) based on swift logs.
>
> While I think that it's appropriate for swift to generate the usage
> information (via internal processes or log processing), the appropriate
> place for quotas is in whatever system handles the concept of a user
> (normally the auth system). This way quotas are enforced by revoking or
> limiting access of the auth token.
>
> --John
>
>
> On Apr 12, 2012, at 11:53 AM, Frederik Van Hecke wrote:
>
> > Hi Kuo,
> >
> > One option would be to keep the usage information (num files, num bytes,
> etc) per container / account in an sqlite DB, just like it is done for
> account and container info.
> >
> > To avoid having to loop through all data at regular intervals (to update
> the info), additional logic could be added to the api methods to update the
> sqlite DB's when new files are added, files are deleted, etc. Such approach
> will require more lines of code, but will be far less stressful on
> performance.
> >
> > (the brute-force approach to loop through it at regular intervals will
> be hell on performance on large deployments..)
> >
> >
> > For data transfer billing based on download / upload amounts, a similar
> approach could be used.
> >
> > If no one else is looking into this, I would certainly be willing to
> help to help get this started.
> >
> >
> > Kind regards,
> > Frederik Van Hecke
> >
> > T: +32487733713
> > E: frederik@cluttr.be <javascript:_e({}, 'cvml',
> 'frederik@cluttr.be');>
> > W: www.cluttr.be
> >
> >
> >
> >
> >
> > This e-mail and any attachments thereto may contain information which is
> confidential and/or protected by intellectual property rights and are
> intended for the sole use of the recipient(s)named above. Any use of the
> information contained herein (including, but not limited to, total or
> partial reproduction, communication or distribution in any form) by persons
> other than the designated recipient(s) is prohibited. If you have received
> this e-mail in error, please notify the sender either by telephone or by
> e-mail and delete the material from any computer. Thank you for your
> cooperation.
> >
> >
> >
> > On Thu, Apr 12, 2012 at 17:45, Kuo Hugo <tonytkdk@gmail.com<javascript:_e({}, 'cvml', 'tonytkdk@gmail.com');>>
> wrote:
> > Hi folks ,
> >
> > I'm thinking about the better approach to manage "an user" or "an
> account" space usage quota in swift.
> > Is there any related blueprint or sub-project even an idea around ?
> > Any suggestion of benefits to be an external service or to be a
> middle-ware in swift-proxy ?
> >
> > I'm concerning about such feature will reduce the performance of entire
> Swift environment.
> >
> > Appreciate :>
> >
> >
> >
> > --
> > +Hugo Kuo+
> > tonytkdk@gmail.com <javascript:_e({}, 'cvml', 'tonytkdk@gmail.com');>
> > +886 935004793
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net <javascript:_e({}, 'cvml',
> 'openstack@lists.launchpad.net');>
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net <javascript:_e({}, 'cvml',
> 'openstack@lists.launchpad.net');>
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net <javascript:_e({}, 'cvml',
> 'openstack@lists.launchpad.net');>
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>


--
Email: alex890714@gmail.com
GitHub: https://github.com/AlexYangYu
Blog: http://alexyang.sinaapp.com
Re: [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ? [ In reply to ]
Alex,

Thank you for important point and interesting information on large-scale
Swift performance!

Can you please explain a little what these times stand for? Is this a
single process runtime, or the time needed to converge cluster in case of
device failure, or something else?

--
Best regards,
Oleg

On Fri, Apr 13, 2012 at 7:23 AM, Alex Yang <alex890714@gmail.com> wrote:

> In my view, the biggest problem of swift is not the new features but the
> improvement of performance.
>
> At first, we knew that container-updater, *-auditor, *-replicator will
> loop all the files in the disk during the running interval. When the number
> of files is very large, the process of replicator, updater and auditor will
> spend much time. So, the time of eventual consistency is very long.
>
> Our practice of Sina Web Service Team https://launchpad.net/~sws:
>
> total accounts: 121,961;
>
> total containers: 160,703;
>
> total objects: 14,291,519;
>
> total storage usage: 1.3T
>
> account replication time: 10 hours;
>
> container replication time: 10 hours;
>
> object replication time: 48 hours;
>
> account audit time: 2 hours;
>
> container audit time: 9 hours;
>
> container update time: 19 hours;
>
> This is terrible If we develop quota upon account db. There is long time
> for eventual consistency.
>
> Secondly, there is a vicious circle during replication. The replicator
> will query the account-server, container-server and object-server to
> compare the metadata, and determine whether to sync. When the number of
> files is very large, the frequent query make the account-server,
> container-server and object-server become a bottle neck. This will
> influence the process of proxy-server to work with back-end servers. There
> a lot of Timeout(10s) ERROR in proxy-server's log and the load-average is
> very high.
> So, some PUT, POST operations failed, and the replicator to sync, and fail
> more, and sync more...
> .
> In my opinion, we need to improve the process of replication and container
> update by using event drive framework or something else..
> My leader may discuss this topic at Design Summit,
> http://openstackconferencespring2012.sched.org/speaker/huicheng
>
>
> 2012/4/13 John Dickinson <me@not.mn>
>
>> Swift keeps total bytes, container, and object count (eventually)
>> up-to-date in the account metadata. There are also log processing tools
>> (like slogging - http://github.com/notmyname/slogging) that can provide
>> usage information (including bandwidth) based on swift logs.
>>
>> While I think that it's appropriate for swift to generate the usage
>> information (via internal processes or log processing), the appropriate
>> place for quotas is in whatever system handles the concept of a user
>> (normally the auth system). This way quotas are enforced by revoking or
>> limiting access of the auth token.
>>
>> --John
>>
>>
>> On Apr 12, 2012, at 11:53 AM, Frederik Van Hecke wrote:
>>
>> > Hi Kuo,
>> >
>> > One option would be to keep the usage information (num files, num
>> bytes, etc) per container / account in an sqlite DB, just like it is done
>> for account and container info.
>> >
>> > To avoid having to loop through all data at regular intervals (to
>> update the info), additional logic could be added to the api methods to
>> update the sqlite DB's when new files are added, files are deleted, etc.
>> Such approach will require more lines of code, but will be far less
>> stressful on performance.
>> >
>> > (the brute-force approach to loop through it at regular intervals will
>> be hell on performance on large deployments..)
>> >
>> >
>> > For data transfer billing based on download / upload amounts, a similar
>> approach could be used.
>> >
>> > If no one else is looking into this, I would certainly be willing to
>> help to help get this started.
>> >
>> >
>> > Kind regards,
>> > Frederik Van Hecke
>> >
>> > T: +32487733713
>> > E: frederik@cluttr.be
>> > W: www.cluttr.be
>> >
>> >
>> >
>> >
>> >
>> > This e-mail and any attachments thereto may contain information which
>> is confidential and/or protected by intellectual property rights and are
>> intended for the sole use of the recipient(s)named above. Any use of the
>> information contained herein (including, but not limited to, total or
>> partial reproduction, communication or distribution in any form) by persons
>> other than the designated recipient(s) is prohibited. If you have received
>> this e-mail in error, please notify the sender either by telephone or by
>> e-mail and delete the material from any computer. Thank you for your
>> cooperation.
>> >
>> >
>> >
>> > On Thu, Apr 12, 2012 at 17:45, Kuo Hugo <tonytkdk@gmail.com> wrote:
>> > Hi folks ,
>> >
>> > I'm thinking about the better approach to manage "an user" or "an
>> account" space usage quota in swift.
>> > Is there any related blueprint or sub-project even an idea around ?
>> > Any suggestion of benefits to be an external service or to be a
>> middle-ware in swift-proxy ?
>> >
>> > I'm concerning about such feature will reduce the performance of entire
>> Swift environment.
>> >
>> > Appreciate :>
>> >
>> >
>> >
>> > --
>> > +Hugo Kuo+
>> > tonytkdk@gmail.com
>> > +886 935004793
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to : openstack@lists.launchpad.net
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help : https://help.launchpad.net/ListHelp
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to : openstack@lists.launchpad.net
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help : https://help.launchpad.net/ListHelp
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Email: alex890714@gmail.com
> GitHub: https://github.com/AlexYangYu
> Blog: http://alexyang.sinaapp.com
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>
Re: [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ? [ In reply to ]
Please explain, how did you measure replication time?

On Fri, Apr 13, 2012 at 7:23 AM, Alex Yang <alex890714@gmail.com> wrote:

> In my view, the biggest problem of swift is not the new features but the
> improvement of performance.
>
> At first, we knew that container-updater, *-auditor, *-replicator will
> loop all the files in the disk during the running interval. When the number
> of files is very large, the process of replicator, updater and auditor will
> spend much time. So, the time of eventual consistency is very long.
>
> Our practice of Sina Web Service Team https://launchpad.net/~sws:
>
> total accounts: 121,961;
>
> total containers: 160,703;
>
> total objects: 14,291,519;
>
> total storage usage: 1.3T
>
> account replication time: 10 hours;
>
> container replication time: 10 hours;
>
> object replication time: 48 hours;
>
> account audit time: 2 hours;
>
> container audit time: 9 hours;
>
> container update time: 19 hours;
>
> This is terrible If we develop quota upon account db. There is long time
> for eventual consistency.
>
> Secondly, there is a vicious circle during replication. The replicator
> will query the account-server, container-server and object-server to
> compare the metadata, and determine whether to sync. When the number of
> files is very large, the frequent query make the account-server,
> container-server and object-server become a bottle neck. This will
> influence the process of proxy-server to work with back-end servers. There
> a lot of Timeout(10s) ERROR in proxy-server's log and the load-average is
> very high.
> So, some PUT, POST operations failed, and the replicator to sync, and fail
> more, and sync more...
> .
> In my opinion, we need to improve the process of replication and container
> update by using event drive framework or something else..
> My leader may discuss this topic at Design Summit,
> http://openstackconferencespring2012.sched.org/speaker/huicheng
>
>
> 2012/4/13 John Dickinson <me@not.mn>
>
>> Swift keeps total bytes, container, and object count (eventually)
>> up-to-date in the account metadata. There are also log processing tools
>> (like slogging - http://github.com/notmyname/slogging) that can provide
>> usage information (including bandwidth) based on swift logs.
>>
>> While I think that it's appropriate for swift to generate the usage
>> information (via internal processes or log processing), the appropriate
>> place for quotas is in whatever system handles the concept of a user
>> (normally the auth system). This way quotas are enforced by revoking or
>> limiting access of the auth token.
>>
>> --John
>>
>>
>> On Apr 12, 2012, at 11:53 AM, Frederik Van Hecke wrote:
>>
>> > Hi Kuo,
>> >
>> > One option would be to keep the usage information (num files, num
>> bytes, etc) per container / account in an sqlite DB, just like it is done
>> for account and container info.
>> >
>> > To avoid having to loop through all data at regular intervals (to
>> update the info), additional logic could be added to the api methods to
>> update the sqlite DB's when new files are added, files are deleted, etc.
>> Such approach will require more lines of code, but will be far less
>> stressful on performance.
>> >
>> > (the brute-force approach to loop through it at regular intervals will
>> be hell on performance on large deployments..)
>> >
>> >
>> > For data transfer billing based on download / upload amounts, a similar
>> approach could be used.
>> >
>> > If no one else is looking into this, I would certainly be willing to
>> help to help get this started.
>> >
>> >
>> > Kind regards,
>> > Frederik Van Hecke
>> >
>> > T: +32487733713
>> > E: frederik@cluttr.be
>> > W: www.cluttr.be
>> >
>> >
>> >
>> >
>> >
>> > This e-mail and any attachments thereto may contain information which
>> is confidential and/or protected by intellectual property rights and are
>> intended for the sole use of the recipient(s)named above. Any use of the
>> information contained herein (including, but not limited to, total or
>> partial reproduction, communication or distribution in any form) by persons
>> other than the designated recipient(s) is prohibited. If you have received
>> this e-mail in error, please notify the sender either by telephone or by
>> e-mail and delete the material from any computer. Thank you for your
>> cooperation.
>> >
>> >
>> >
>> > On Thu, Apr 12, 2012 at 17:45, Kuo Hugo <tonytkdk@gmail.com> wrote:
>> > Hi folks ,
>> >
>> > I'm thinking about the better approach to manage "an user" or "an
>> account" space usage quota in swift.
>> > Is there any related blueprint or sub-project even an idea around ?
>> > Any suggestion of benefits to be an external service or to be a
>> middle-ware in swift-proxy ?
>> >
>> > I'm concerning about such feature will reduce the performance of entire
>> Swift environment.
>> >
>> > Appreciate :>
>> >
>> >
>> >
>> > --
>> > +Hugo Kuo+
>> > tonytkdk@gmail.com
>> > +886 935004793
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to : openstack@lists.launchpad.net
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help : https://help.launchpad.net/ListHelp
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to : openstack@lists.launchpad.net
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help : https://help.launchpad.net/ListHelp
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Email: alex890714@gmail.com
> GitHub: https://github.com/AlexYangYu
> Blog: http://alexyang.sinaapp.com
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>


--
Kind regards
Dmitry Ukov
IT Engineer
Mirantis, Inc.
Re: [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ? [ In reply to ]
On Thu, 2012-04-12 at 20:51 -0500, John Dickinson wrote:
> Swift keeps total bytes, container, and object count (eventually) up-to-date in the account metadata. There are also log processing tools (like slogging - http://github.com/notmyname/slogging) that can provide usage information (including bandwidth) based on swift logs.
>
> While I think that it's appropriate for swift to generate the usage information (via internal processes or log processing), the appropriate place for quotas is in whatever system handles the concept of a user (normally the auth system). This way quotas are enforced by revoking or limiting access of the auth token.

Hello,

If it helps, that's the way we're accounting and managing quotas in our
swift deployment (Memstore).

We use a background process in each proxy server that runs through the
logs to account source IP, bytes in, bytes out and requests per account,
and another background process to account bytes, container and object
count using swift API.

We have a small middleware in the proxy that limits HTTP operations.

For example, if the disk is over quota we forbid PUT requests for that
account (GET, DELETE are still valid).

We have found quotas are little bit difficult to enforce though. Being
accurate has an important performance hit, so our quotas are
"eventually" enforced (blame memcache for that), and by now is close
enough for us.

Regards,

Juan

--
Juan J. Martinez
Development, MEMSET

mail: juan@memset.com
web: http://www.memset.com/

Memset Ltd., registration number 4504980. 25 Frederick Sanger Road, Guildford, Surrey, GU2 7YD, UK.


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp