Evaluating individuals after deadline

[leightonvg]

I was running GAMA multiple times and sometimes it happens that the search phase does not end within the max_total_time. I have to interrupt GAMA to stop the search. The logs show the following behavior:

I notice that the errors are raised in gama.genetic_programming.operator_set. I suppose that within Gama._search_phase() it happens that gama.genetic_programming.compilers.scikitlearn.evaluate_individual is called beyond the deadline.

Causing timeout issues in gama.genetic_programming.compilers.scikitlearn.evaluate_individual because the timeout is negative.

I am not sure whether this is the cause of the errors and the non-stopping search space execution, and why it happens exactly. But do note that in the logs the timeout values are negative, and seem to correspond to when I would have expected the deadline based on a 120 second run. Therefore I believe sometimes individuals are evaluated after the deadline, causing GAMA to go on and eventually crash with memory issues.

[PGijsbers]

Thanks for the report! I was under the impression that this issue had been resolved.
The error is originally raised here.My memory is a bit fuzzy on this, but I think the deadline was a workaround because sometimes the stopit module would not stop runtime evaluation in time, so I added a thin layer of manual checks (the deadline stuff). Ideally I'd like to fix the core issue by either using a different module to accomplish the same thing as stopit in a more reliable manner, or figure out why stopit wasn't reliable in the first place. But a quick fix to the deadline logic to patch things over would also be OK for now. I am a bit busy right now, but if that sounds like something you'd like to work on I am open to PRs :)

[simonprovost]

@leightonvg Any advancements in this direction?