💥 model database migrations #40
Labels
enhancement
New feature or request
performance
Improve the performance or better resource management
Milestone
Is your feature request related to a problem? Please describe.
Dropping and re-creating data every time a change is needed is not performant and prone to errors: if intermediate and processed data file are not removed and the collections not restored in the initial state we can have inconsistencies
Describe the solution you'd like
Database changes need to be modeled with migration when possible. A change could be done without running the entire import process. Test database migrations to understand if they could be applied in this context
Describe alternatives you've considered
src.features.smarterdb.get_or_create_sample
could be modified to check if sample was changed. The same concept could be applied on dataset imports.Additional context
See:
The text was updated successfully, but these errors were encountered: