High memory utilisation per chain during fitting

[mattmoo]

I hate to spam the discussions with my big data problems, but I wonder whether this is expected behaviour.

I am struggling to increase the number of chains in an analysis. N is about 60000, currently I am looking at 1500 iterations (but that's too low, RHat ~ 1.8 on some parameters), with 8 chains (running parallel on a 16 core Linux machine). The size of the resulting mcpfit is about as expected, 5.2GB (~60000 * 1500 * 8chains * 8bytes).

What I didn't expect is that each chain generates a worker which is about 5GB, which seems to quite rapidly build from a few 100MB each towards the end of the analysis (so it fails right at the end of the hour-long analysis!).

That means that I can't increase the number of iterations without significantly cutting down the number of chains. Of course this may be a jags problem, but do you think that there would be a way to address it? (Or maybe alternatively do some rough calculations and warn the user if memory is likely to be an issue).

[lindeloev]

I think this will be solved with the next release. Possibly, the memory inflation comes when JAGS transfers the result to R (via rjags). This result will be around 90% smaller in v0.4. Keeping the issue open so that we can track whether this is indeed solved.

[mattmoo]

Sounds good, please let me know when there's a working dev version!

[lindeloev]

It's in branch "v0.4". Plotting does not work yet and I haven't written up tests for the new features. Install using remotes::install_github("lindeloev/[email protected]") (I think). Also, to fix the loo problem in #110, you could look at the code for the add_loo function which calls fitted. You could simply re-write that line to fitted(..., nsamples = some_lower_number) to use fewer samples for loo computation as a workaround, until I implement the by-data-row solution.

Let me know if you try this out and learn anything! It's busy days here, but I'm looking forward to getting v0.4 past the goal post when I get a bit of free time!

[mattmoo]

Thanks looks to work! (At least for a Gaussian model, I didn't get a chance to try a logit model yet).

I did have to run it twice serially for reasons of stupidity, but it took far longer the second time (Linux, RStudio Server). I used 10 cores, but it seems that there are workers on 16 cores are, even now. Do the workers need to be explicitly released?

[lindeloev]

That sounds great! So did this approach (using v0.4 + fewer samples for loo) fix both this issue and #110?

mcp uses the future and future,apply packages to handle parallel processing. Once it has launched the processes, they keep hanging around, waiting for more jobs. But it should not spawn more than cores processes ever.

[mattmoo]

I haven't got the loo solution sorted yet, by modifying add_loo, do you mean modifying (or maybe masking) it in the brms package?

[lindeloev]

Oh, I see now it's in add_loglik() but it requires edits at multiple lines and thorough testing that it's correct.

…

On Fri, Mar 19, 2021, 20:11 mattmoo ***@***.***> wrote: I haven't got the loo solution sorted yet, by modifying add_loo, do you mean modifying (or maybe masking) it in the brms package? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#112 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK7NOKGYMWVU47YMHDFID3TEOOVTANCNFSM4ZHLWZZA> .

[mattmoo]

I see, yep am getting a loo error 'Error: Number of rows in matrix not equal to length(chain_id).', which I guess is down to it not being divided into separate tables for each chain. Is that strictly necessary? Or can I divide them arbitrarily?