Assert hook should evaluate recursively

[gabrielelanaro]

I'm testing the last revisions, using the assert_hook stuff. Maybe it's a bit premature to post issues about this feature, but I've written a little test to show the problematic behaviour.

from attest import Tests

testone = Tests()

def the_one():
    """A function that returns 1
    """
    return 1

@testone.test
def test_one():
    # First case, doesn't show AssertionError: not (1 == 2)
    assert the_one() == 2
    # Second test, correct behaviour
    a = the_one()
    assert a == 2

if __name__ == '__main__':
    testone.main() 

[dag]

This is intentional. It seems more useful to me in for example:

assert isinstance(a, float)

to get

not isinstance(1, float)

rather than

not False

As you note yourself, it's easily worked around by making it a variable. Alternative solutions would be to limit non-evaluation to builtins, or evaluate and show in "steps", something like

AssertionError: not (the_one() == 2)
                not (1 == 2)

Also I suppose it could evaluate calls that aren't taking any arguments ...

What do you think is the best solution (or combination thereof) here?

[gabrielelanaro]

I would say that the best solution is to evaluate and show it in steps, but (sometimes) it can add too much noise and unnecessary information, like in py.test http://pytest.org/assert.html#assert-with-the-assert-statement.

Let me explain better the "scenario", so you can understand better the problem:

I was testing the correct behaviour of a function, given an input, I want that output:

def test_min_scale():
    Assert(MinScale.step_up(0)) == 2
    Assert(MinScale.step_down(0)) == 10

In this scenario the Assert wrapping only the left member was perfect, thanks to that for example I discovered that MinScale.step_down was returning shifted results (and let me quickly correct them).

Using assert it was giving me:

>>> assert MinScale.step_down(0) == 2 
 AssertionError: not (MinScale.step_up(0) == 2)

That by the way is equivalent to this expression:

Assert(MinScale.step_down(0) == 2)

So in this case the problem was to find a way to distinguish between this two cases:

Assert(MinScale.step_down(0) == 2)
Assert(MinScale.step_down(0)) == 2

[dag]

Do you think it sufficient to evaluate in only two steps, and not evaluate nested calls recursively? That'd be less noisy. Complicated nesting in a one-liner doesn't seem very Pythonic anyway.

Example:

assert isinstance(int('5'), float)

Show on failure:

not isinstance(int('5'), float)
not False

Skipped:

not isinstance(5, float)

Hm, writing this example I realise it might be useful to recurse anyway, and examples such as "not False" should perhaps just be excluded. The most useful output for the above example would be:

not isinstance(int('5'), float)
not isinstance(5, float)

Thoughts?

[gabrielelanaro]

Analyzing the tests in attest sourcecode and some of my old projects. Most tests fall in two main categories, comparisons and assertive functions (there are also assertRaises but usually they are simple).

Example comparisons:

assert len(result.succeeded) == 1
assert result.failed[0].exc_info[0] is AssertionError
assert repr(ExpressionEvaluator(expr, globals(), locals())) == result

Example assertive:
assert issubclass(DecoratedTest, TestBase)
assert hasattr(instance, 'two')

I think that comparisons usually need both the last call (for example len([1,2,3])) and the return value. As you said there aren't so much calls in a one-liner so adding all the nested calls won't result in much noise and covers more cases.

assertive functions ('not' and everything that isn't a comparison operator) don't need the return call but need the steps.

A solution can be to distinguish between this two classes (actually just make the comparison special case):

comparisons:

==, !=, <> etc.. http://docs.python.org/reference/expressions.html#comparisons
They need both the nested calls and the return values of the operands.

all the others:

• boolean http://docs.python.org/reference/expressions.html#boolean-operations
• nothing, just the call of a function

They need just the calls.

[dag]

Not all predicates necessarily return True or False, though. Objects in Python have boolean "adaption" such that zero is "False-like" etc. Are these return values useful to see evaluated? I expect not, but it should be considered. What do you think?

[dag]

Also,

Special cases aren't special enough [to break the rules.]

It scares me to "special-case" things too much. If it has real practical value and is consistent without exceptional cases I suppose it's fine but care should be taken here to not make too many assumptions, and to keep it simple.

[gabrielelanaro]

I don't think that the return value of the predicates is useful (most of the time), for example:

assert return_something()

In this assertion I may be interested in which the return value of something is, but more commonly one would write the (imho more correct) following expression:

assert return_something() is not None

The special case scares me also, it's very easy in fact to fall in inconsistencies of some sort.

What I was thinking is like, if the last expression is an 'or_expr' (don't know if it is an 'or_expr', expand all the calls except the last:

assert isinstance(return_something(),Something)
          isinstance(<Something>, Something)

if the last expression is a 'comparison', expand all the calls (I'm inventing):

assert repr(ExpressionEvaluator(expr, globals(), locals())) == result
assert repr(ExpressionEvaluator('a is 2', [...], [...]) == 'a == 3'
assert repr('a is 2') == 'a == 3'
assert 'a is 2' == 'a == e'

since:

comparison    ::=  or_expr ( comp_operator or_expr )*

It should work also with multiple comparisons:

a() == b() == c()
a() < b() < c()

But there should be problem also with Don't know how it would behave with:

not ( a() == b() )
(a() == b())

Because the first is a "not" + not_test and don't know what the second is.

Maybe the answer to what is more general and mantainablle is contained in the way that python parse code, aka to find a common breakpoint for all the cases or the most general cases (comparison, not_test, or_expr ?).