Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

💥 model database migrations #40

Open
bunop opened this issue Nov 11, 2021 · 0 comments
Open

💥 model database migrations #40

bunop opened this issue Nov 11, 2021 · 0 comments
Labels
enhancement New feature or request performance Improve the performance or better resource management

Comments

@bunop
Copy link
Member

bunop commented Nov 11, 2021

Is your feature request related to a problem? Please describe.
Dropping and re-creating data every time a change is needed is not performant and prone to errors: if intermediate and processed data file are not removed and the collections not restored in the initial state we can have inconsistencies

Describe the solution you'd like
Database changes need to be modeled with migration when possible. A change could be done without running the entire import process. Test database migrations to understand if they could be applied in this context

Describe alternatives you've considered
src.features.smarterdb.get_or_create_sample could be modified to check if sample was changed. The same concept could be applied on dataset imports.

Additional context
See:

@bunop bunop added enhancement New feature or request performance Improve the performance or better resource management labels Nov 11, 2021
@bunop bunop added this to the SMARTER database v1.0.0 milestone Nov 11, 2021
@bunop bunop added this to SMARTER Aug 3, 2023
@bunop bunop moved this to To do in SMARTER Aug 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance Improve the performance or better resource management
Projects
Status: To do
Development

No branches or pull requests

1 participant