Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parameter --maxConcurrency to prevent different load errors. #848

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

tilmanschweitzer
Copy link

Add parameter --maxConcurrency to prevent different load errors.

Depending on the content different load errors like 'Bandwidth exhausted'
or 'Received RST_STREAM with code 2' occur and cause the import to be incomplete.
Limiting the concurrent imported batches seems to prevents issues.
The default behaviour is still without any limit.

see #411

Tilman Schweitzer added 2 commits December 7, 2021 22:09
Depending on the content different load errors like 'Bandwidth exhausted'
or 'Received RST_STREAM with code 2' occur and cause the import to be incomplete.
Limiting the concurrent imported batches seems to prevents issues.
The default behaviour is still without any limit.

see jloosli#411
@nonoumasy
Copy link

does this fix the bandwidth exhausted issue?

@tilmanschweitzer
Copy link
Author

@nonoumasy A max concurrency of 45 worked for my use case. Around a concurrency of 48 the issue started again. But I don't know if there are other aspects that contribute to the issue.

@nonoumasy
Copy link

nonoumasy commented Dec 19, 2022

@nonoumasy A max concurrency of 45 worked for my use case. Around a concurrency of 48 the issue started again. But I don't know if there are other aspects that contribute to the issue.

Thanks @tilmanschweitzer.
btw how do I use this parameter?
Do I just do something like?

npx -p node-firestore-import-export firestore-import -a credentials.json -b -maxConcurrency=45 backup.json

Also, when you get this issue after importing, is there possible data loss(some data doesn't get imported into firestore)? is data corruption a possibility?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants