-
Notifications
You must be signed in to change notification settings - Fork 108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Declarative Testing of Policies #645
Comments
I think this is an interesting idea and is similar to #455, which was a draft attempt at adding a way to simulate policies from branches or under different approval conditions. I believe an implementation of this should be relatively straight-forward, since this is how Policy Bot's own tests work. Create a new implementation of The major open questions for me are about the user experience. Is this a subcommand of the server binary that you run locally or as part of a CI build? Is it an API endpoint, where you submit everything to a running server and it returns a result? Is it a new UI, like the simulate page? What did you have in mind? |
@bluekeyes that's a great question. From a user experience I think having a binary that you can run locally, like conftest or helm unittest would be helpful. Ideally that way you can both run it locally and in a CI system to ensure your policy is behaviorally correct as well as syntactically correct. Specifically I'm imagining a yaml file that looks something like: policy-file: <path to policy.yml>
scenarios:
- name: <name of scenario>
result: approved|disapproved|pending
authors:
- <author>
reviewers:
- <reviewer>
... Then perhaps a local binary whose job it is to translate that yaml/struct into a That allows for a clean user experience of something like: Benefits:
I could see an argument for not allowing it to run against a remoteURL for security reasons. It would be another endpoint that would need to be opened up, or you would need to be able to authenticate as github in order to run the tests. It could also expose a variety of information about policy behaviors allowing someone to brute force an outcome (though security through obscurity isn't something that should be relied on). |
I like the idea of a local binary that can do everything without a server connection. This avoids possible security issues with the server, but also means your tests do not require a network connection and won't fail if the server or GitHub is unavailable. If you want to match the local behavior against a real Policy Bot server, the health check endpoint returns the version, so you can use that to make sure the local version matches. One complication with a purely local system is that you'll have to specify the details of the PR and background information about teams and organizations in GitHub. For instance, you'll need to define which teams exist and which users are members of those teams. This makes the scenario definitions longer but I think is probably better overall because it means your tests will not break as team and organization membership changes. Maybe the YAML definition can include a global section that contains information, like team definitions, that is true for all of the scenarios? |
I agree. While having to declare teams and orgs requires slightly more yaml, not a massive amount. And I think is vital to ensuring the tool could test the correct state. Perhaps I don't have a user who is part of groups A and B but I want to ensure that their hypothetical approval under rule foo wouldn't be enough. Then the test would need that information anyway. I don't think it needs to be global to a test run. Instead making it global to a single file (test or test suite definition) then we could allow for testing different scenarios. |
As a starting point towards something like this feature, #725 added a simulation API that allows testing a policy in different situations given a real repository and running instance of the server. This doesn't fulfill all of the same uses as a local tool, but does allow some new types of tests. It also provides a starting point for future expansion, since the types of things that can be simulated are kind of limited right now. |
I'd like to be able to declaratively test that the policies.yml matches the desired rules and end state.
Similar to behavior driven tests I'd like to be able to define the .policy.yml, input conditions (comments, approvals, authors, etc), and an expected state: approved|pending|disapproved and then be able to run that suite against policy-bot.
This would allow for a lot more confidence in the policy intentions and a lot more confidence for auditors.
I'm happy to start writing a test runner for something like this if this is something others would find useful.
The text was updated successfully, but these errors were encountered: