Mailing List Archive

Exception as the primary error handling mechanism?
I observe that python library primarily use exception for error
handling rather than use error code.

In the article API Design Matters by Michi Henning

Communications of the ACM
Vol. 52 No. 5, Pages 46-56
10.1145/1506409.1506424
http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext

It says "Another popular design flaw—namely, throwing exceptions for
expected outcomes—also causes inefficiencies because catching and
handling exceptions is almost always slower than testing a return
value."

My observation is contradicted to the above statement by Henning. If
my observation is wrong, please just ignore my question below.

Otherwise, could some python expert explain to me why exception is
widely used for error handling in python? Is it because the efficiency
is not the primary goal of python?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Thu, Dec 31, 2009 at 8:47 PM, Peng Yu <pengyu.ut@gmail.com> wrote:
> I observe that python library primarily use exception for error
> handling rather than use error code.
>
> In the article API Design Matters by Michi Henning
>
> Communications of the ACM
> Vol. 52 No. 5, Pages 46-56
> 10.1145/1506409.1506424
> http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
>
> It says "Another popular design flaw—namely, throwing exceptions for
> expected outcomes—also causes inefficiencies because catching and
> handling exceptions is almost always slower than testing a return
> value."
>
> My observation is contradicted to the above statement by Henning. If
> my observation is wrong, please just ignore my question below.
>
> Otherwise, could some python expert explain to me why exception is
> widely used for error handling in python? Is it because the efficiency
> is not the primary goal of python?

Correct; programmer efficiency is a more important goal for Python instead.
Python is ~60-100x slower than C;[1] if someone is worried by the
inefficiency caused by exceptions, then they're using completely the
wrong language.

Cheers,
Chris
--
http://blog.rebertia.com

[1] http://shootout.alioth.debian.org/u64/which-programming-languages-are-fastest.php?gcc=on&python=on&calc=chart
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Thu, Dec 31, 2009 at 11:47 PM, Peng Yu <pengyu.ut@gmail.com> wrote:
> I observe that python library primarily use exception for error
> handling rather than use error code.
>
> In the article API Design Matters by Michi Henning
>
> Communications of the ACM
> Vol. 52 No. 5, Pages 46-56
> 10.1145/1506409.1506424
> http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
>
> It says "Another popular design flaw—namely, throwing exceptions for
> expected outcomes—also causes inefficiencies because catching and
> handling exceptions is almost always slower than testing a return
> value."
>
> My observation is contradicted to the above statement by Henning. If
> my observation is wrong, please just ignore my question below.
>
> Otherwise, could some python expert explain to me why exception is
> widely used for error handling in python? Is it because the efficiency
> is not the primary goal of python?
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Read the quote again "Another popular design flaw—namely, throwing
exceptions *for expected outcomes*"
In Python, throwing exceptions for expected outcomes is considered
very bad form (well, except for StopIteration but that should almost
never be handled directly by the programmer).

To answer why people recommend using "Easier to Ask Forgiveness than
Permission" as opposed to "Look Before You Leap" : Because of the way
it's implemented, Python works quite differently from most languages.
An attribute look-up is rather expensive because it's a hash table
look-up at run time. Wrapping a small piece of code in a try block
however, isn't (relatively) very expensive at all in Python. It's only
catching the exception that's expensive, but if you're catching the
exception, something has gone wrong anyway and performance isn't your
biggest issue.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Thu, 31 Dec 2009 20:47:49 -0800, Peng Yu wrote:

> I observe that python library primarily use exception for error handling
> rather than use error code.
>
> In the article API Design Matters by Michi Henning
>
> Communications of the ACM
> Vol. 52 No. 5, Pages 46-56
> 10.1145/1506409.1506424
> http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
>
> It says "Another popular design flaw—namely, throwing exceptions for
> expected outcomes—also causes inefficiencies because catching and
> handling exceptions is almost always slower than testing a return
> value."

This is very, very wrong.

Firstly, notice that the author doesn't compare the same thing. He
compares "catching AND HANDLING" the exception (emphasis added) with
*only* testing a return value. Of course it is faster to test a value and
do nothing, than it is to catch an exception and then handle the
exception. That's an unfair comparison, and that alone shows that the
author is biased against exceptions.

But it's also wrong. If you call a function one million times, and catch
an exception ONCE (because exceptions are rare) that is likely to be
much, much faster than testing a return code one million times.

Before you can talk about which strategy is faster, you need to
understand your problem. When exceptions are rare (in CPython, about one
in ten or rarer) then try...except is faster than testing each time. The
exact cut-off depends on how expensive the test is, and how much work
gets done before the exception is raised. Using exceptions is only slow
if they are common.

But the most important reason for preferring exceptions is that the
alternatives are error-prone! Testing error codes is the anti-pattern,
not catching exceptions.

See, for example:

http://c2.com/cgi/wiki?UseExceptionsInsteadOfErrorValues
http://c2.com/cgi/wiki?ExceptionsAreOurFriends
http://c2.com/cgi/wiki?AvoidExceptionsWheneverPossible

Despite the title of that last page, it has many excellent arguments for
why exceptions are better than the alternatives.

(Be warned: the c2 wiki is filled with Java and C++ programmers who
mistake the work-arounds for quirks of their language as general design
principles. For example, because exceptions in Java are evcen more
expensive and slow than in Python, you will find lots of Java coders
saying "don't use exceptions" instead of "don't use exceptions IN JAVA".)

There are many problems with using error codes:

* They complicate your code. Instead of returning the result you care
about, you have to return a status code and the return result you care
about. Even worse is to have a single global variable to hold the status
of the last function call!

* Nobody can agree whether the status code means the function call
failed, or the function call succeeded.

* If the function call failed, what do you return as the result code?

* You can't be sure that the caller will remember to check the status
code. In fact, you can be sure that the caller WILL forget sometimes!
(This is human nature.) This leads to the frequent problem that by the
time a caller checks the status code, the original error has been lost
and the program is working with garbage.

* Even if you remember to check the status code, it complicates the code,
makes it less readable, confuses the intent of the code, and often leads
to the Arrow Anti-pattern: http://c2.com/cgi/wiki?ArrowAntiPattern

That last argument is critical. Exceptions exist to make writing correct
code easier to write, understand and maintain.

Python uses special result codes in at least two places:

str.find(s) returns -1 if s is not in the string
re.match() returns None is the regular expression fails

Both of these are error-prone. Consider a naive way of getting the
fractional part of a float string:

>>> s = "234.567"
>>> print s[s.find('.')+1:]
567

But see:

>>> s = "234"
>>> print s[s.find('.')+1:]
234

You need something like:

p = s.find('.')
if p == -1:
print ''
else:
print s[p+1:]



Similarly, we cannot safely do this in Python:


>>> re.match(r'\d+', '123abcd').group()
'123'
>>> re.match(r'\d+', 'abcd').group()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'group'

You need to do this:

mo = re.match(r'\d+', '123abcd')
if mo is not None: # or just `if mo` will work
mo.group()


Exceptions are about making it easier to have correct code. They're also
about making it easier to have readable code. Which is easier to read,
easier to understand and easier to debug?

x = function(1, 2, 3)
if x != -1:
y = function(x, 1, 2)
if y != -1:
z = function(y, x, 1)
if z != -1:
print "result is", z
else:
print "an error occurred"
else:
print "an error occurred"
else:
print "an error occurred"


versus:


try:
x = function(1, 2, 3)
y = function(x, 1, 2)
print "result is", function(y, x, 1)
except ValueError:
print "an error occurred"



In Python, setting up the try...except block is very fast, about as fast
as a plain "pass" statement, but actually catching the exception is quite
slow. So let's compare string.find (which returns an error result) and
string.index (which raises an exception):


>>> from timeit import Timer
>>> setup = "source = 'abcd'*100 + 'e'"
>>> min(Timer("p = source.index('e')", setup).repeat())
1.1308379173278809
>>> min(Timer("p = source.find('e')", setup).repeat())
1.2237567901611328

There's hardly any difference at all, and in fact index is slightly
faster. But what about if there's an exceptional case?


>>> min(Timer("""
... try:
... p = source.index('z')
... except ValueError:
... pass
... """, setup).repeat())
3.5699808597564697
>>> min(Timer("""
... p = source.find('z')
... if p == -1:
... pass
... """, setup).repeat())
1.7874350070953369


So in Python, catching the exception is slower, in this case about twice
as slow. But remember that the "if p == -1" test is not free. It might be
cheap, but it does take time. If you call find() enough times, and every
single time you then test the result returned, that extra cost may be
more expensive than catching a rare exception.

The general rule in Python is:

* if the exceptional event is rare (say, on average, less than about one
time in ten) then use a try...except and catch the exception;

* but if it is very common (more than one time in ten) then it is faster
to do a test.


> My observation is contradicted to the above statement by Henning. If my
> observation is wrong, please just ignore my question below.
>
> Otherwise, could some python expert explain to me why exception is
> widely used for error handling in python? Is it because the efficiency
> is not the primary goal of python?

Yes.

Python's aim is to be fast *enough*, without necessarily being as fast as
possible.

Python aims to be readable, and to be easy to write correct, bug-free
code.


--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
In article <mailman.300.1262323578.28905.python-list@python.org>,
Benjamin Kaplan <benjamin.kaplan@case.edu> wrote:
>
>In Python, throwing exceptions for expected outcomes is considered
>very bad form [...]

Who says that? I certainly don't.
--
Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/

Weinberg's Second Law: If builders built buildings the way programmers wrote
programs, then the first woodpecker that came along would destroy civilization.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
Peng Yu wrote:
> I observe that python library primarily use exception for error
> handling rather than use error code.
[...]
> It says "Another popular design flaw—namely, throwing exceptions for
> expected outcomes—also causes inefficiencies because catching and
> handling exceptions is almost always slower than testing a return
> value."
>
> My observation is contradicted to the above statement by Henning. If
> my observation is wrong, please just ignore my question below.

Your observation is not wrong, but, as Benjamin already explained,
you are misinterpreting Michi Henning's statement. He doesn't condemn
exception handling per se, but only for the handling of *expected*
outcomes. He would consider using exceptions fine for *exceptional*
output, and that is exactly the way they are used in the Python API.

Notice that in cases where the failure may be expected, Python
also offers variants that avoid the exception:
- if you look into a dictionary, expecting that a key may not
be there, a regular access, d[k], may give a KeyError. However,
alternatively, you can use d.get(k, default) which raises no
exception, and you can test "k in d" in advance.
- if you open a file, not knowing whether it will be there,
you get an IOError. However, you can use os.path.exists in
advance to determine whether the file is present (and create
it if it's not).

So, in these cases, it is a choice of the user to determine whether
the error case is exceptional or not.

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Jan 1, 12:43 am, a...@pythoncraft.com (Aahz) wrote:
> In article <mailman.300.1262323578.28905.python-l...@python.org>,
> Benjamin Kaplan  <benjamin.kap...@case.edu> wrote:
> >In Python, throwing exceptions for expected outcomes is considered
> >very bad form [...]
>
> Who says that?  I certainly don't.

Agreed.

int("asdf") is supposed to return what, exactly? Any language that
tries to return an int is horribly broken.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Fri, 01 Jan 2010 11:34:19 +0100 "Martin v. Loewis"
<martin@v.loewis.de> wrote:

> Your observation is not wrong, but, as Benjamin already explained,
> you are misinterpreting Michi Henning's statement. He doesn't condemn
> exception handling per se, but only for the handling of *expected*
> outcomes. He would consider using exceptions fine for *exceptional*
> output, and that is exactly the way they are used in the Python API.

May I point out at this point that "exceptional" does not mean
"unexpected"? You catch exceptions, not unexpectations. An exception
is rare, but not surprising. Case in point: StopIteration.

To put it differently: When you write "catch DeadParrot", you certainly
expect to get a DeadParrot once in a while -- why else would you get it
in your head to try and catch it? An unexpected exception is the one
that crashes your program.

/W

--
INVALID? DE!

--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Fri, 01 Jan 2010 02:43:21 -0800, Jonathan Gardner wrote:

> On Jan 1, 12:43 am, a...@pythoncraft.com (Aahz) wrote:
>> In article <mailman.300.1262323578.28905.python-l...@python.org>,
>> Benjamin Kaplan  <benjamin.kap...@case.edu> wrote:
>> >In Python, throwing exceptions for expected outcomes is considered
>> >very bad form [...]
>>
>> Who says that?  I certainly don't.
>
> Agreed.
>
> int("asdf") is supposed to return what, exactly? Any language that tries
> to return an int is horribly broken.


[sarcasm]
No no, the right way to deal with that is have int("asdf") return some
arbitrary bit pattern, and expect the user to check a global variable to
see whether the function returned a valid result or not. That's much
better than catching an exception!
[/sarcasm]


--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Fri, 01 Jan 2010 00:26:09 -0500, Benjamin Kaplan wrote:

> On Thu, Dec 31, 2009 at 11:47 PM, Peng Yu <pengyu.ut@gmail.com> wrote:
>> I observe that python library primarily use exception for error
>> handling rather than use error code.
>>
>> In the article API Design Matters by Michi Henning
>>
>> Communications of the ACM
>> Vol. 52 No. 5, Pages 46-56
>> 10.1145/1506409.1506424
>> http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
>>
>> It says "Another popular design flaw—namely, throwing exceptions for
>> expected outcomes—also causes inefficiencies because catching and
>> handling exceptions is almost always slower than testing a return
>> value."
>>
>> My observation is contradicted to the above statement by Henning. If my
>> observation is wrong, please just ignore my question below.
>>
>> Otherwise, could some python expert explain to me why exception is
>> widely used for error handling in python? Is it because the efficiency
>> is not the primary goal of python?
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>
>>
> Read the quote again "Another popular design flaw—namely, throwing
> exceptions *for expected outcomes*"
> In Python, throwing exceptions for expected outcomes is considered very
> bad form (well, except for StopIteration but that should almost never be
> handled directly by the programmer).


Exceptions are *exceptional*, not "errors" or "unexpected". They are
exceptional because they aren't the "normal" case, but that doesn't mean
they are surprising or unexpected. Are you surprised that your "for x in
range(1000)" loop comes to an end? Of course you are not -- it is
completely expected, even though less than 1% of the iterations are the
last loop. The end of the sequence is EXCEPTIONAL but not UNEXPECTED.

If you program without expecting that keys can sometimes be missing from
dictionaries (KeyError), or that you might sometimes have to deal with a
list index that is out of range (IndexError), or that the denominator in
a division might be zero (ZeroDivisionError), then you must be writing
really buggy code. None of these things are unexpected, but they are all
exceptional.

The urllib2 module defines a HTTPError class, which does double-duty as
both an exception and a valid HTTP response. If you're doing any HTTP
programming, you better expect to deal with HTTP 301, 302 etc. codes, or
at least trust that the library you use will transparently handle them
for you.


> To answer why people recommend using "Easier to Ask Forgiveness than
> Permission" as opposed to "Look Before You Leap" : Because of the way
> it's implemented, Python works quite differently from most languages. An
> attribute look-up is rather expensive because it's a hash table look-up
> at run time. Wrapping a small piece of code in a try block however,
> isn't (relatively) very expensive at all in Python.

It's not just relatively inexpensive, it's absolutely inexpensive: it
costs about as much as a pass statement in CPython, which is pretty much
as cheap as it gets. (If anyone can demonstrate a cheaper operation
available from pure Python, I'd love to see it.)


> It's only catching the exception that's expensive,

True.


> but if you're catching the exception,
> something has gone wrong anyway and performance isn't your biggest
> issue.


The second try...except clause in the urllib2 module reads:

try:
kind = int(kind)
except ValueError:
pass

In this case, the error is simply ignored. Nothing has gone wrong.


Here's an example from my own code: I have an API where you pass a
mapping (either a dict or a list of (key, value) tuples) to a function.
So I do this:

try:
it = obj.iteritems()
except AttributeError:
it = obj
for key, value in it:
do_stuff()



There's nothing wrong with catching exceptions.


--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On 1/1/2010 3:47 PM, Peng Yu wrote:
> I observe that python library primarily use exception for error
> handling rather than use error code.
>
> In the article API Design Matters by Michi Henning
>
> Communications of the ACM
> Vol. 52 No. 5, Pages 46-56
> 10.1145/1506409.1506424
> http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
>
> It says "Another popular design flaw—namely, throwing exceptions for
> expected outcomes—also causes inefficiencies because catching and
> handling exceptions is almost always slower than testing a return
> value."
>
> My observation is contradicted to the above statement by Henning. If
> my observation is wrong, please just ignore my question below.
>
> Otherwise, could some python expert explain to me why exception is
> widely used for error handling in python?

Simple, when an exception is thrown and I don't catch it, the exception
terminates the program immediately and I got a traceback showing the
point of failure. When I return error value and I don't check for it, I
passed passed errors silently and gets a traceback forty-two lines later
when trying to use the resources I failed to acquire forty-two lines prior.

> Is it because the efficiency
> is not the primary goal of python?

Efficiency is not primary goal of python, but since python encourages
EAFP (Easier to Ask Forgiveness than Permission); the design decisions
chosen makes setting up a try-block much cheaper than a language
designed over LBYL (Look Before You Leap) and return codes.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Fri, Jan 1, 2010 at 9:49 AM, Steven D'Aprano
<steve@remove-this-cybersource.com.au> wrote:
>
> Exceptions are *exceptional*, not "errors" or "unexpected". They are
> exceptional because they aren't the "normal" case, but that doesn't mean
> they are surprising or unexpected. Are you surprised that your "for x in
> range(1000)" loop comes to an end? Of course you are not -- it is
> completely expected, even though less than 1% of the iterations are the
> last loop. The end of the sequence is EXCEPTIONAL but not UNEXPECTED.
>

Sorry if my word choice was confusing- I was trying to point out that
in Python, you don't test errors for your typical conditions, but for
ones that you know still exist but don't plan on occurring often.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
Steven D'Aprano wrote:

> On Fri, 01 Jan 2010 02:43:21 -0800, Jonathan Gardner wrote:
>
>> On Jan 1, 12:43 am, a...@pythoncraft.com (Aahz) wrote:
>>> In article <mailman.300.1262323578.28905.python-l...@python.org>,
>>> Benjamin Kaplan <benjamin.kap...@case.edu> wrote:
>>> >In Python, throwing exceptions for expected outcomes is considered
>>> >very bad form [...]
>>>
>>> Who says that? I certainly don't.
>>
>> Agreed.
>>
>> int("asdf") is supposed to return what, exactly? Any language that tries
>> to return an int is horribly broken.
>
>
> [sarcasm]
> No no, the right way to deal with that is have int("asdf") return some
> arbitrary bit pattern, and expect the user to check a global variable to
> see whether the function returned a valid result or not. That's much
> better than catching an exception!
> [/sarcasm]

Or the other way around, as in C (I suspect the original ACM article assumed
C.) Look at the legion of C library subroutines that return only 0 for good
or -1 for bad, and do all their real work in side-effects (through pointers
as function arguments.) Python is a big improvement: use the function
return values for the payload, and push the out-of-band "omyghod" response
into an Exception.

Mel.


--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Fri, 01 Jan 2010 11:02:28 -0500, Benjamin Kaplan wrote:

> I was trying to point out that in
> Python, you don't test errors for your typical conditions, but for ones
> that you know still exist but don't plan on occurring often.

I'm sorry, but that makes no sense to me at all. I don't understand what
you are trying to say.


You do understand that exceptions aren't just for errors? They are raised
under specific circumstances. Whether that circumstance is an error or
not is entirely up to the caller.


try:
n = mylist.index('fault')
except ValueError:
print "All is good, no fault detected"
else:
print "Your data contains a fault in position", n



People get hung up on the idea that exceptions == errors, but they
aren't. They *may* be errors, and that is one common use, but that
depends entirely on the caller.


--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
> You do understand that exceptions aren't just for errors? They are raised
> under specific circumstances. Whether that circumstance is an error or
> not is entirely up to the caller.

I think that's a fairly narrow definition of the word error, and
probably also the source of confusion in this thread.

ISTM that there is a long tradition of giving different meaning to
the word "error" in computing. For example, the Unix man pages
list various conditions as "errors" purely by their outcome, and
completely ignoring on whether the caller would consider the result
erroneous - ISTM that a system call reports an "error" iff it is
"unsuccessful".

By that (common) usage of "error", it is a condition determined by
the callee, not the caller (i.e. callee could not successfully
complete the operation). In that sense, it is indeed equivalent
to Python's usage of exceptions, which are also determined by the
callee, and typically also in cases where successful completion is
not possible. Whether these cases are "exceptional" in the word
sense (i.e. deviating from the norm) would have to be decided by
the application, again (which would set the norm).

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Thu, Dec 31, 2009 at 11:24 PM, Chris Rebert <clp2@rebertia.com> wrote:
> On Thu, Dec 31, 2009 at 8:47 PM, Peng Yu <pengyu.ut@gmail.com> wrote:
>> I observe that python library primarily use exception for error
>> handling rather than use error code.
>>
>> In the article API Design Matters by Michi Henning
>>
>> Communications of the ACM
>> Vol. 52 No. 5, Pages 46-56
>> 10.1145/1506409.1506424
>> http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
>>
>> It says "Another popular design flaw—namely, throwing exceptions for
>> expected outcomes—also causes inefficiencies because catching and
>> handling exceptions is almost always slower than testing a return
>> value."
>>
>> My observation is contradicted to the above statement by Henning. If
>> my observation is wrong, please just ignore my question below.
>>
>> Otherwise, could some python expert explain to me why exception is
>> widely used for error handling in python? Is it because the efficiency
>> is not the primary goal of python?
>
> Correct; programmer efficiency is a more important goal for Python instead.
> Python is ~60-100x slower than C;[1] if someone is worried by the
> inefficiency caused by exceptions, then they're using completely the
> wrong language.

Could somebody let me know how the python calls and exceptions are
dispatched? Is there a reference for it?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
In article <4B3DCFAB.3030909@v.loewis.de>,
Martin v. Loewis <martin@v.loewis.de> wrote:
>
>Notice that in cases where the failure may be expected, Python
>also offers variants that avoid the exception:
>- if you look into a dictionary, expecting that a key may not
> be there, a regular access, d[k], may give a KeyError. However,
> alternatively, you can use d.get(k, default) which raises no
> exception, and you can test "k in d" in advance.
>- if you open a file, not knowing whether it will be there,
> you get an IOError. However, you can use os.path.exists in
> advance to determine whether the file is present (and create
> it if it's not).

But you *still* need to catch IOError: someone might delete the file
after the test. Figuring out how to deal with race conditions is one of
the main reasons Alex Martelli advocates EAFP over LBYL.

Of course, in the real world, you often end up wanting to do it both
ways.
--
Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/

Weinberg's Second Law: If builders built buildings the way programmers wrote
programs, then the first woodpecker that came along would destroy civilization.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Fri, Jan 1, 2010 at 5:36 PM, Peng Yu <pengyu.ut@gmail.com> wrote:

> >> Otherwise, could some python expert explain to me why exception is
> >> widely used for error handling in python? Is it because the efficiency
> >> is not the primary goal of python?
> >
> > Correct; programmer efficiency is a more important goal for Python
> instead.
> > Python is ~60-100x slower than C;[1] if someone is worried by the
> > inefficiency caused by exceptions, then they're using completely the
> > wrong language.
>
> Could somebody let me know how the python calls and exceptions are
> dispatched? Is there a reference for it?
>

I don't quite understand what you're asking here, but it sounds almost like
you're looking at the question from an incorrect POV. "Exceptions" are a
general sort of concept in computer science and various computer programming
languages, but they are not at all equal from one language to another. The
document you referenced was speaking to a particular implementation of the
concept, and speaking to particular characteristics of that language's
implementation. Even though its not talking just about say, C, C#, Java, or
anything -- its speaking from a certain POV of a certain classes of
languages.

In Python, setting up an exception -- the 'try' clause -- costs virtually
nothing. Its about equivalent to having a 'pass' statement in there. If you
do a test for every iteration of some activity, you're incurring a
non-negligable cost each time. If you're performing an action and "usually"
(to varying definitions of 'usually'), it's going to succeed-- then that
test will result in far more cost in time then using a try/except clause in
Python.

Because in the implementation of exceptions in Python, you only pay a more
expensive cost /if/ that exception is thrown and handled. If its very likely
that in a situation an exception would be thrown, then yes-- then you should
probably test first... if that exception-catch is so expensive as to be
important to your. In most cases, its not. In the vast majority of cases,
this is premature optimization and often adds additional weight to the test
as you have to protect against race conditions. (As an aside, in many cases
using exceptions actually helps you in a wider problem of preventing race
conditions. Its not a cure-all by any means, but it helps)

If someone specifies a file, the chances are-- the file is there. Its
cheaper for you to just try to open it then to test if its there first,
because if you do the test? Then the likely circumstance (the file exists)
is running code to test that case /every time/. Whereas if you just try to
open it, in the likely circumstance -- it works. In the exceptional
circumstance, it might cost more then if you had tested first... but that's
an exceptional circumstance and therefore more rare.

--S
Re: Exception as the primary error handling mechanism? [ In reply to ]
Peng Yu wrote:
> Could somebody let me know how the python calls and exceptions are
> dispatched? Is there a reference for it?

I'm not a Python expert, but I have read some parts of the implementation.
Hopefully someone steps up if I misrepresent things here...

In order to understand Python exception handling, take a look at various C
function implementations. You will see that they commonly return a pointer
to a Python object (PyObject*), even if it is a pointer to the 'None'
singleton. So, a function in Python _always_ returns something, even if it
is 'None'.

If, at the C level, a function doesn't return anything (i.e. a C NULL
pointer) that means that the function raised an exception. Checking this
pointer is pretty easy, typically you check that, clean up and return NULL
yourself. Further functions for manipulating the exception stack and
declarations of exception types and singletons are found in pyerrors.h (in
Python 2.5, at least).

I mentioned an "exception stack" above, though I'm not 100% sure if that is
the proper term. I think that exceptions can be stacked upon each other
(e.g. an HTTPD throwing a high-level RequestError when it encounters a low-
level IOError) and that that is also how the backtrace is implemented, but
I'm not sure about that.


Hopefully someone can confirm or correct me here and that it helped you.

Cheers!

Uli


--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
Peng Yu schrieb:
> On Thu, Dec 31, 2009 at 11:24 PM, Chris Rebert <clp2@rebertia.com> wrote:
>> On Thu, Dec 31, 2009 at 8:47 PM, Peng Yu <pengyu.ut@gmail.com> wrote:
>>> I observe that python library primarily use exception for error
>>> handling rather than use error code.
>>>
>>> In the article API Design Matters by Michi Henning
>>>
>>> Communications of the ACM
>>> Vol. 52 No. 5, Pages 46-56
>>> 10.1145/1506409.1506424
>>> http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
>>>
>>> It says "Another popular design flaw—namely, throwing exceptions for
>>> expected outcomes—also causes inefficiencies because catching and
>>> handling exceptions is almost always slower than testing a return
>>> value."
>>>
>>> My observation is contradicted to the above statement by Henning. If
>>> my observation is wrong, please just ignore my question below.
>>>
>>> Otherwise, could some python expert explain to me why exception is
>>> widely used for error handling in python? Is it because the efficiency
>>> is not the primary goal of python?
>> Correct; programmer efficiency is a more important goal for Python instead.
>> Python is ~60-100x slower than C;[1] if someone is worried by the
>> inefficiency caused by exceptions, then they're using completely the
>> wrong language.
>
> Could somebody let me know how the python calls and exceptions are
> dispatched? Is there a reference for it?

The source?

http://python.org/ftp/python/2.6.4/Python-2.6.4.tgz

These are really deep internals that - if they really concern you - need
intensive studies, not casual reading of introductionary documents. IMHO
you shouldn't worry, but then, there's a lot things you seem to care I
wouldn't... :)

Diez
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
> I mentioned an "exception stack" above, though I'm not 100% sure if that is
> the proper term. I think that exceptions can be stacked upon each other
> (e.g. an HTTPD throwing a high-level RequestError when it encounters a low-
> level IOError) and that that is also how the backtrace is implemented, but
> I'm not sure about that.

Not exactly. In this scenario, the IOError exception gets caught, its
entire traceback discarded, and an entirely new exception RequestError
gets raised (that has no connection to the original IOError anymore,
unless the httpd code explicitly links the two).

Instead, the traceback objects are created for a single exception.
They are essentially the same as the call stack, just in reverse
order (so that you get the "most recent call last" traceback output).
Each traceback links to a frame object, and a next traceback object.

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Sat, Jan 2, 2010 at 6:05 AM, Diez B. Roggisch <deets@nospam.web.de> wrote:
> Peng Yu schrieb:
>>
>> On Thu, Dec 31, 2009 at 11:24 PM, Chris Rebert <clp2@rebertia.com> wrote:
>>>
>>> On Thu, Dec 31, 2009 at 8:47 PM, Peng Yu <pengyu.ut@gmail.com> wrote:
>>>>
>>>> I observe that python library primarily use exception for error
>>>> handling rather than use error code.
>>>>
>>>> In the article API Design Matters by Michi Henning
>>>>
>>>> Communications of the ACM
>>>> Vol. 52 No. 5, Pages 46-56
>>>> 10.1145/1506409.1506424
>>>> http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
>>>>
>>>> It says "Another popular design flaw—namely, throwing exceptions for
>>>> expected outcomes—also causes inefficiencies because catching and
>>>> handling exceptions is almost always slower than testing a return
>>>> value."
>>>>
>>>> My observation is contradicted to the above statement by Henning. If
>>>> my observation is wrong, please just ignore my question below.
>>>>
>>>> Otherwise, could some python expert explain to me why exception is
>>>> widely used for error handling in python? Is it because the efficiency
>>>> is not the primary goal of python?
>>>
>>> Correct; programmer efficiency is a more important goal for Python
>>> instead.
>>> Python is ~60-100x slower than C;[1] if someone is worried by the
>>> inefficiency caused by exceptions, then they're using completely the
>>> wrong language.
>>
>> Could somebody let me know how the python calls and exceptions are
>> dispatched? Is there a reference for it?
>
> The source?
>
> http://python.org/ftp/python/2.6.4/Python-2.6.4.tgz
>
> These are really deep internals that - if they really concern you - need
> intensive studies, not casual reading of introductionary documents. IMHO you
> shouldn't worry, but then, there's a lot things you seem to care I
> wouldn't... :)

For my own interest, I want understand the run time behavior of python
and what details causes it much slower. Although people choose python
for its programming efficiency, but sometimes the runtime still
matters. This is an important aspect of the language. I'm wondering
this is not even documented. Why everybody has to go to the source
code to understand it?

Are you sure that there is no document that describes how python is
working internally (including exceptions)?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
> For my own interest, I want understand the run time behavior of python
> and what details causes it much slower. Although people choose python
> for its programming efficiency, but sometimes the runtime still
> matters. This is an important aspect of the language. I'm wondering
> this is not even documented. Why everybody has to go to the source
> code to understand it?

There are two answers to this question:

a) Because the source is the most precise and most complete way of
documenting it. Any higher-level documentation would necessarily be
incomplete.
b) Because nobody has contributed documentation.

The two causes correlate: because writing documentation of VM
internals takes a lot of effort and is of questionable use, nobody
has written any.

> Are you sure that there is no document that describes how python is
> working internally (including exceptions)?

Such documents certainly exist, but not as part of the Python
distribution. See

http://wiki.python.org/moin/CPythonVmInternals

for one such document.

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On 1/2/2010 10:04 AM, Peng Yu wrote:

> For my own interest, I want understand the run time behavior of python

That depends on the implementation.

> and what details causes it much slower.

A language feature that slows all implementation is the dynamic
name/slot binding and resolution. Any implementation can be made faster
by restricting the dynamism (which makes the imlementaion one of a
subset of Python).

> Although people choose python
> for its programming efficiency, but sometimes the runtime still
> matters.

There is no 'the' runtime. Whether or not there even *is* a runtime, as
usually understoold, is a matter of the implementation.


> This is an important aspect of the language.

It is an aspect of each implementation, of which there are now more than
one.

Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception as the primary error handling mechanism? [ In reply to ]
On Jan 1, 2:47 pm, Peng Yu <pengyu...@gmail.com> wrote:
>
> In the article API Design Matters by Michi Henning
>
> Communications of the ACM
> Vol. 52 No. 5, Pages 46-56
> 10.1145/1506409.1506424http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
>
> It says "Another popular design flaw—namely, throwing exceptions for
> expected outcomes—also causes inefficiencies because catching and
> handling exceptions is almost always slower than testing a return
> value."
>
> My observation is contradicted to the above statement by Henning. If
> my observation is wrong, please just ignore my question below.

Seeing that quite a few people have put their own interpretation on
what I wrote, I figured I'll post a clarification.

The quoted sentence appears in a section of the article that deals
with efficiency. I point out in that section that bad APIs often have
a price not just in terms of usability and defect rate, but that they
are often inefficient as well. (For example, wrapper APIs often
require additional memory allocations and/or data copies.) Incorrect
use of exceptions also incurs an efficiency penalty.

In many language implementations, exception handling is expensive;
significantly more expensive than testing a return value. Consider the
following:

int x;
try {
x = func();
} catch (SomeException) {
doSomething();
return;
}
doSomethingElse();

Here is the alternative without exceptions. (func() returns
SpecialValue instead of throwing.)

int x;
x = func();
if (x == SpecialValue) {
doSomething();
return;
}
doSomethingElse();

In many language implementations, the second version is considerably
faster, especially when the exception may be thrown from deep in the
bowels of func(), possibly many frames down the call tree.

If func() throws an exception for something that routinely occurs in
the normal use of the API, the extra cost can be noticeable. Note that
I am not advocating not to use exceptions. I *am* advocating to not
throw exceptions for conditions that are not exceptional.

The classic example of this are lookup functions that, for example,
retrieve the value of an environment variable, do a table lookup, or
similar. Many such APIs throw an exception when the lookup fails
because the key isn't the table. However, very often, looking for
something that isn't there is a common case, such as when looking for
a value and, if the value isn't present already, adding it. Here is an
example of this:

KeyType k = ...;
ValueType v;

try {
v = collection.lookup(k);
} catch (NotFoundException) {
collection.add(k, defaultValue);
v = defaultValue;
}
doSomethingWithValue(v);

The same code if collection doesn't throw when I look up something
that isn't there:

KeyType k = ...;
ValueType v;

v = collection.lookup(k);
if (v == null) {
collection.add(k, defaultValue);
v = defaultValue;
}
doSomethingWithValue(v);

The problem is that, if I do something like this in a loop, and the
loop is performance-critical, the exception version can cause a
significant penalty.

As the API designer, when I make the choice between returning a
special value to indicate some condition, or throwing an exception, I
should consider the following questions:

* Is the special condition such that, under most conceivable
circumstances, the caller will treat the condition as an unexpected
error?

* Is it appropriate to force the caller to deal with the condition in
a catch-handler?

* If the caller fails to explicitly deal with the condition, is it
appropriate to terminate the program?

Only if the answer to these questions is "yes" is it appropriate to
throw an exception. Note the third question, which is often forgotten.
By throwing an exception, I not only force the caller to handle the
exception with a catch-handler (as opposed to leaving the choice to
the caller), I also force the caller to *always* handle the exception:
if the caller wants to ignore the condition, he/she still has to write
a catch-handler and failure to do so terminates the program.

Apart from the potential performance penalty, throwing exceptions for
expected outcomes is bad also because it forces a try-catch block on
the caller. One example of this is the .NET socket API: if I do non-
blocking I/O on a socket, I get an exception if no data is ready for
reading (which is the common and expected case), and I get a zero
return value if the connection was lost (which is the uncommon and
unexpected case).

In other words, the .NET API gets this completely the wrong way round.
Code that needs to do non-blocking reads from a socket turns into a
proper mess as a result because the outcome of a read() call is tri-
state:

* Data was available and returned: no exception

* No data available: exception

* Connection lost: no exception

Because such code normally lives in a loop that decrements a byte
count until the expected number of bytes have been read, the control
flow because really awkward because the successful case must be dealt
with in both the try block and the catch handler, and the error
condition must be dealt with in the try block as well.

If the API did what it should, namely, throw an exception when the
connection is lost, and not throw when I do a read (whether data was
ready or not), the code would be far simpler and far more
maintainable.

At no point did I ever advocate not to use exception handling.
Exceptions are the correct mechanism to handle errors. However, what
is considered an error is very much in the eye of the beholder. As the
API creator, if I indicate errors with exceptions, I make a policy
decision about what is an error and what is not. It behooves me to be
conservative in that policy: I should throw exceptions only for
conditions that are unlikely to arise during routine and normal use of
the API.

Cheers,

Michi.
--
http://mail.python.org/mailman/listinfo/python-list

1 2 3 4  View All