Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add possibility to execute tests in parallel #1623

Open
Renegatto opened this issue Jun 5, 2024 · 0 comments
Open

Add possibility to execute tests in parallel #1623

Renegatto opened this issue Jun 5, 2024 · 0 comments

Comments

@Renegatto
Copy link
Collaborator

Summary

Running plutip tests take ages and lengthens the feedback loop.
It is proven to be possible to execute tests in parallel, however currently it require forking CTL and altering some parts of the framework.

Testing framework

purescript-spec is a good candidate:
It is very similar to mote, and it is able to run test cases in parallel, with flexible chunking.
However it's SpecT is a bit less general than MoteT and require altering, but that's not a big issue, and a small contribution to the library can easily fix the issue.

Parallelizing

Each testing instance should have it's own PlutipConfig, with non duplicating ports. This way we ensure they don't interfere.
Kupo workdirs will not interfere as Kupo's workdir contain a randomly generated number.
Ogmios and Plutip workirs interference have not proven to exist.

Importantly, graceful shutdown must be properly implemented so no matter what all temp directories must be cleaned up, and all the services must be shut down after tests.

Design

It is probably not the best idea to left the implementation in the CTL itself, but instead put it in the separate library.

To leave it CTL:

  • testnet cluster creation and running tools
  • contract for redistributing funds between tests
    To separate:
  • MoteT/SpecT manipulation, creation, configuring and running a test tree.

Potential problems

Decreased cardano-node/ogmios/kupo performance on multiple parallel instances

Accidental errors will occur more often because of the CPU, Mem and disk usage.
Some operations may take longer to execute, and it should be concerned.
So for epoch-dependent (the ones that should be finished in a certain amount of epochs) operations increasing amount of parallel instances could require increasing the slot or epoch length.

Setting up tests to run in parallel may be complicated

This is not a big issue because

  1. Parallel testing is optional
  2. It is still better than enforcing every project that need this feature to alter part of the CTL codebase on every project and implement it themselves
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant