-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
QGreenland is very slow to open in QGIS #60
Comments
Is there anything useful in the logs (~/.jupyter-server.txt I think)? Also, how big is the QGreenland file? Do smaller files start faster? |
QGreenland as a whole contains 3GB of data that need to be loaded as layers in QGIS (the project file itself is negligible in size, it's just XML that points to the real data). If we eliminate layers, the project does start faster. Will re-test and check the logs! Thanks :) |
could also check CPU usage with |
Unfortunately, I can't find such a log file, and Memory and CPU usage is consistently 2-4% CPU and 2% CPU the whole time the project is opening! Is the read from disk possibly the bottleneck?
|
I took some timings. Rsyncing all of QGreenland from
EDIT: Realized the above is NFS -> NFS, also tested
Opening QGreenland from Opening QGreenland from |
Given that our workshop is next week, do you have ideas for working around this? We considered writing a script and having attendees run that script to copy data they'll need in to the container's "local disk", but (a) lack of persistence, and (b) possibility of filling up the shared node's disk are possible dealbreakers. |
@MattF-NSIDC aaah, I had thought the data was coming from the outside world via HTTP or something of that sort. Makes total sense it's NFS being the bottleneck. When there are many users reading it, they might all be much slower too. The larger scheme of things, I think finding a way to get QGis to load this via HTTP might be the way to go. However, I don't understand enough about QGis to even know if this is possible. In the meantime, I've opened 2i2c-org/infrastructure#2562 to give each user a dedicated, 10Gi non-NFS disk at ~/qgis-data. I tested copying the greenland files here, took about 13s. And opening took about 15s. I think this is acceptable? If it is, we will deploy this when the week comes. For events specifically, I would like you to consider Finally, as someone involved with jupyter-desktop-server from the start, I JUST ABSOLUTELY LOVE THAT QGIS IS GOING TO BE USED IN A WORKSHOP VIA A HUB! It makes me so incredibly happy! Would you be interested in writing a blog post for the Jupyter blog on how it went after? :) |
We should probably clean up these extra disks after your workshop is over, and remove this feature to reduce costs. Otherwise 10Gi is paid for by all users regardless of wether it is used or not. |
Wonderful, thank you! This will be very helpful :)
Absolutely can do. Can you give an example of an issue that is appropriate and one that's better suited for emailing support? I've been fairly fuzzy on that. For example, should the
It makes me incredibly happy that we have the ability to do this, so thank you :) I'd love to write a blog post! Thanks for the invitation to do so :) |
@yuvipanda I'd like to take your answer to the above question and add it to an issues template for this repo so when someone goes to open a new issue they'll be prompted to reconsider if their issue should be an email. What do you think? |
@MattF-NSIDC will try to provide answers soon :) A short version is that 'anything requiring changes outside the image' definitely needs to go to [email protected]. In the meantime though, we've deployed the change. Test it out and let us know? Also when is your event? |
Thank you, Yuvi! Our event is tomorrow at 9AM MT through the end of the week. We'll be active ~3 hours per day, maybe using the hub for half that time. |
@yuvipanda Thank you again for hooking us up with the EBS volumes so quickly last week ❤️ Sorry for the last-minute nature of this issue! We set up a script at I can't begin to describe the stress relief from not having to deal with weird problems with end-user hardware/software installations. Excited to do a write-up after this week! :) |
@MattF-NSIDC thanks! I opened 2i2c-org/infrastructure#2575 to track the blog post! Can you open an issue here to track what should be here vs sent to support@? |
#68 :) |
It takes something like 5 minutes to open QGreenland (can be found at
public/QGreenland
) in QGIS in CryoCloud. I tested this with a full 32GB node share and saw more or less the same results as a 4GB share. On my local machine (i7-1255U), the project opens in less than 5 seconds within the locally-builtrepo2docker
container. Any idea what the cause might be? Is it just the type of virtual machine we're deploying onto?The text was updated successfully, but these errors were encountered: