Data download consumes too much memory

[manycoding]

Downloading 3m of items consumes about 8gb memory. Not much, but could it be lower?

Possible solution with chunks - scrapinghub/python-scrapinghub#121

[manycoding]

1. Read raw data to numpy in chunks, using coroutines - yield. Then collate so the index stays in order.
2. Same with DataFrame.

We cannot go from max DataFrame size yet, but possibly its memory consumption is much lower than when it's being initialized.

[manycoding]

The raw data is stored in a numpy array and has certain size. When downloading, there's no memory overhead (the memory consumption is smaller then the result array).
Thus, I don't see any room for improvement memory-wise as long as raw is kept (used for JSON schema validation), no matter how we get the data.

Pandas looks more memory efficient.

Code
https://jupyter.scrapinghub.com/user/v/lab/tree/shared/Arche/Memory%20consumption.ipynb
Bench notebook (to accuratelly measure memory consumption) https://jupyter.scrapinghub.com/user/v/lab/tree/shared/Arche/Memory%20consumption%20bench.ipynb

[manycoding]

@andersonberg @felipefin @alexandr1988 Have you hit the wall with memory recently, in particular in Kosmonaut high memory profile?
As I haven't found a way to optimize memory in the library, I can increase memory for our server if needed.

[felipefin]

I remember having issues downloading more than 500k items, however I was not using Kosmonaut.
I'll keep that in mind next time I face this kind of situations.

[nerevaryn]

I think I didn't have a task like this. Will let you know if I have

…

On Mon, Oct 14, 2019 at 5:10 PM Felipe Lunkes Fin ***@***.***> wrote: I remember having issues downloading more than 500k items, however I was not using Kosmonaut. I'll keep that in mind next time I face this kind of situations. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#140?email_source=notifications&email_token=ABWRSRYPOF6CNRGLAVQKE5TQOR4WNA5CNFSM4IBOEUN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBE3LBY#issuecomment-541701511>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWRSR2NWFIEIUZCSEYQPPTQOR4WNANCNFSM4IBOEUNQ> .

[andersonberg]

No, I didn't use Kosmonaut with high memory yet. Adding to that, I didn't face any memory problems in the Arche profile

[manycoding]

Closing because the most straightforward way is to up the server memory.