Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the accumulate filter #9133

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

VannTen
Copy link
Contributor

@VannTen VannTen commented Nov 16, 2024

SUMMARY

Add the accumulate filter, a passthrough to python itertools.accumulate
See the below issue for motivation.
Fixes #9129

ISSUE TYPE
  • New Module/Plugin Pull Request
COMPONENT NAME

filter/accumulate

ADDITIONAL INFORMATION

N/A

@ansibullbot ansibullbot added integration tests/integration new_contributor Help guide this first time contributor plugins plugin (any type) tests tests labels Nov 16, 2024
@ansibullbot

This comment was marked as outdated.

@ansibullbot ansibullbot added ci_verified Push fixes to PR branch to re-run CI needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR labels Nov 16, 2024
- Add myself as a maintainer for it.
- Some integration tests.
@ansibullbot ansibullbot removed ci_verified Push fixes to PR branch to re-run CI needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR labels Nov 16, 2024
@felixfontein felixfontein added check-before-release PR will be looked at again shortly before release and merged if possible. backport-10 Automatically create a backport for the stable-10 branch labels Nov 16, 2024
@VannTen
Copy link
Contributor Author

VannTen commented Nov 16, 2024 via email

Copy link
Collaborator

@felixfontein felixfontein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution!

plugins/filter/accumulate.py Outdated Show resolved Hide resolved
plugins/filter/accumulate.py Outdated Show resolved Hide resolved
plugins/filter/accumulate.py Outdated Show resolved Hide resolved
plugins/filter/accumulate.py Outdated Show resolved Hide resolved
plugins/filter/accumulate.py Outdated Show resolved Hide resolved
plugins/filter/accumulate.py Outdated Show resolved Hide resolved
plugins/filter/accumulate.py Outdated Show resolved Hide resolved
tests/integration/targets/filter_accumulate/tasks/main.yml Outdated Show resolved Hide resolved
The aliases file was copied over from
tests/integrations/targets/filter_dict/aliases as the documentation[1]
suggests to use the same group as existing similar tests.

[1]: https://docs.ansible.com/ansible/latest/dev_guide/testing/sanity/integration-aliases.html

Suggested-by: Felix Fontein <[email protected]>
@russoz
Copy link
Collaborator

russoz commented Nov 17, 2024

Other than the small adjustment in the markup, LGTM


def filters(self):
return {
'accumulate': list_accumulate,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not

Suggested change
'accumulate': list_accumulate,
'accumulate': accumulate,

?

list_accumulate is not really doing anything.

Copy link
Collaborator

@felixfontein felixfontein Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is doing something: it's allowing exactly one positional argument, thus restricting the arguments to what is documented.

(It also allows to add better error checking at some point.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fair enough

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I had considered whether to forward more arguments to python accumulate, but:

  • I don't think there is an ergonomic way to pass a binary function
  • Passing initial would be roughly equivalent to ([initial_elem] + input_list) | accumulate) so I don't think it's worth adding an optional argument.

(It also allows to add better error checking at some point.)

What do you have in mind ? I could just as well add it now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somethings like checking for args and kward and raising an exception in there is some ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you have in mind ? I could just as well add it now.

Checking whether the sequence is actually a sequence.

With from collections.abc import Sequence you can use if not isinstance(sequence, Sequence): to test it.

(This also accepts strings as sequences. That's generally not so great, but here strings are also OK, I guess.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have time to work on this before say in a week?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking back on this, I'm wondering if I should add that. There is already some quite readable errors message coming directly from accumulate, for instance I I tried to feed an integer to the implementation as-is:

TASK [filter_accumulate : Test accumulate filter] ******************************
fatal: [testhost]: FAILED! => {"msg": "The conditional check '1 | community.general.accumulate == 1' failed. The error was: Unexpected templating type error occurred on ({% if 1 | community.general.accumulate == 1 %} True {% else %} False {% endif %}): 'int' object is not iterable. 'int' object is not iterable"}

However, the filter accepts some input which, while they might be useful, could be questionnable from a correctness POV.

TASK [filter_accumulate : debug] ***********************************************
ok: [testhost] => {
    "{'test': 1, 'ee': 1} | community.general.accumulate": [
        "test",
        "testee"
    ]
}

-> this one errors when checking for a Sequence type.

Wdyt ? I would tend to got the more permissive way, and document the unobvious cases, but I don't have strong opinions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it appears that checking for Sequence does not catch subtler invalid case, such as non compatible list members :

TASK [filter_accumulate : Test accumulate filter] ******************************
fatal: [testhost]: FAILED! => {"msg": "The conditional check '['ac', 1] | community.general.accumulate == ['ac', 'ac1']' failed. The error was: Unexpected templating type error occurred on ({% if ['ac', 1] | community.general.accumulate == ['ac', 'ac1'] %} True {% else %} False {% endif %}): can only concatenate str (not \"int\") to str. can only concatenate str (not \"int\") to str"}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-> this one errors when checking for a Sequence type.

If you really want to do that, you can always use the | list filter to convert the dict to a list (of its keys) first.

Also, it appears that checking for Sequence does not catch subtler invalid case, such as non compatible list members :

Obviously it cannot catch all invalid input, but it can catch some of the more obvious ones.

Wdyt ? I would tend to got the more permissive way, and document the unobvious cases, but I don't have strong opinions.

I would definitely check for Sequence.

@ansibullbot ansibullbot removed the new_contributor Help guide this first time contributor label Nov 18, 2024
@VannTen
Copy link
Contributor Author

VannTen commented Nov 23, 2024 via email

Accepting arbitrary iterables might lead to surprising behavior so we
are stricter on what we accept in the filter.
Relaxing those requirements is easier than retrofitting them, in terms
of backwards compatibility.

Suggested-by: Felix Fontein <[email protected]>
Signed-off-by: Max Gautier <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-10 Automatically create a backport for the stable-10 branch check-before-release PR will be looked at again shortly before release and merged if possible. integration tests/integration plugins plugin (any type) tests tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add an 'accumulate' filter
4 participants