Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fixture to cleanup dataloader after each test runs #156

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

oliverholworthy
Copy link
Member

Adds a fixture to cleanup dataloader after each test runs. This ensures that if a test using the Merlin Dataloader only partially consumes a dataloader instance (and isn't using it as a context manager (with statement), or calling stop explicitly). Then we automatically clean it up by calling stop on it (it's ok if we call stop more than once).

Related:

Background

In the scenario where we have a partially consumed dataloader in a test. When combined with something like pytest-cov which scans subprocesses, this can result in the dataloader background thread hanging around and onto resources that conflict with subsequent tests. In NVTabular this resulted in subsequent tests of operators to take significantly longer than usual (>5 hours instead of ~20 mins)

@oliverholworthy oliverholworthy added this to the Merlin 23.07 milestone Jun 28, 2023
@oliverholworthy oliverholworthy self-assigned this Jun 28, 2023
@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/dataloader/review/pr-156

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant