Introduce two citation matching related fixes for issues discovered during the last provision round on BETA #1429
Conversation
…emory limits Introducing new `citationmatchingDirectSparkExecutorOverhead` parameter set to `2048` by default.
…emory limits Fixing fuzzy citation matching failure due to an executor exceeding memory limits by propagating `sparkExecutorOverhead` to the `citation-matching-input-transformer` spark action. Until now it was propagated to `citation-matching` action only.
…o not remove output dir at startup
| @@ -276,6 +276,11 @@ | |||
| <value>${sparkExecutorMemory}</value> | |||
| <description>memory for individual executor</description> | |||
| </property> | |||
| <property> | |||
| <name>citationmatchingDirectSparkExecutorOverhead</name> | |||
| <value>2048</value> | |||
There was a problem hiding this comment.
I don't understand why this value has to be specified both here and in the config-default.template. And to they have to be always kept in sync?
There was a problem hiding this comment.
This is in line with other *SparkExecutorOverhead properties defined in workflow.xml file and it does not require keeping in sync values declared in workflow.xml file with the ones from the default-config.xml files.
I think this duplication was mostly kept as safety measure: if the value is not declared in default-config.xml it will be taken from the workflow.xml as a default one. It is quite useful when performing development runs (no default-config.xml involved) but I agree it does not look well having some properties defined in two places therefore I have created a dedicated issue to consider duplicates removal: #1430.
Namely:
Usually I will create two separate PRs but those are pretty small fixes so we could cover them with a single PR (will be merged with the master branch with separate commits though).