/api/remoteEnvironment/compare creates unbounded number of concurrent HTTP requests

[piotrp]

Descrie the bug
On environment with large number of processess calling /api/remoteEnvironment/compare fails with:

16:41:53.126 [nussknacker-ui-akka.actor.default-dispatcher-22] ERROR akka.actor.ActorSystemImpl - Error during processing of request: 'Exceeded configured max-open-requests value of [128]. This means that the request queue of this pool (HostConnectionPoolSetup(flink1,8080,ConnectionPoolSetup(ConnectionPoolSettings(4,0,5,128,1,Duration.Inf,100 milliseconds,2 minutes,30 seconds,ClientConnectionSettings(Some(User-Agent: akka-http/10.1.8),10 seconds,1 minute,512,None,WebSocketSettings(<function0>,ping,Duration.Inf,akka.http.impl.settings.WebSocketSettingsImpl$$$Lambda$568/0x0000000800690840@7d4c4bed),List(),ParserSettings(2048,16,64,64,8192,64,83886080,8388608,256,1048576,Strict,RFC6265,true,Set(),Full,Error,Map(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 0, Content-MD5 -> 0, Date -> 0, If-Match -> 0, If-None-Match -> 0, User-Agent -> 32),false,true,akka.util.ConstantFun$$$Lambda$548/0x000000080064a840@4f3fe11,akka.util.ConstantFun$$$Lambda$548/0x000000080064a840@4f3fe11,akka.util.ConstantFun$$$Lambda$549/0x000000080064a040@68d2c3d4),None,TCPTransport),New,1 second),akka.http.scaladsl.HttpConnectionContext$@64e053d9,akka.event.MarkerLoggingAdapter@7b6c6f8b))) has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See http://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information.'. Completing with 500 Internal Server Error response. To change default exception handling behavior, provide a custom ExceptionHandler.
akka.stream.BufferOverflowException: Exceeded configured max-open-requests value of [128]. This means that the request queue of this pool (HostConnectionPoolSetup(flink1,8080,ConnectionPoolSetup(ConnectionPoolSettings(4,0,5,128,1,Duration.Inf,100 milliseconds,2 minutes,30 seconds,ClientConnectionSettings(Some(User-Agent: akka-http/10.1.8),10 seconds,1 minute,512,None,WebSocketSettings(<function0>,ping,Duration.Inf,akka.http.impl.settings.WebSocketSettingsImpl$$$Lambda$568/0x0000000800690840@7d4c4bed),List(),ParserSettings(2048,16,64,64,8192,64,83886080,8388608,256,1048576,Strict,RFC6265,true,Set(),Full,Error,Map(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 0, Content-MD5 -> 0, Date -> 0, If-Match -> 0, If-None-Match -> 0, User-Agent -> 32),false,true,akka.util.ConstantFun$$$Lambda$548/0x000000080064a840@4f3fe11,akka.util.ConstantFun$$$Lambda$548/0x000000080064a840@4f3fe11,akka.util.ConstantFun$$$Lambda$549/0x000000080064a040@68d2c3d4),None,TCPTransport),New,1 second),akka.http.scaladsl.HttpConnectionContext$@64e053d9,akka.event.MarkerLoggingAdapter@7b6c6f8b))) has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See http://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information.
        at akka.http.impl.engine.client.PoolInterfaceActor.<init>(PoolInterfaceActor.scala:88) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.http.impl.engine.client.PoolInterfaceActor$.$anonfun$props$1(PoolInterfaceActor.scala:34) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.actor.TypedCreatorFunctionConsumer.produce(IndirectActorProducer.scala:87) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.actor.Props.newActor(Props.scala:212) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.actor.ActorCell.newActor(ActorCell.scala:628) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.actor.ActorCell.create(ActorCell.scala:654) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:527) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:549) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:283) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[nussknacker-ui-assembly.jar:0.3.1-preview_parallel_metrics_util_v3-2021-07-20-3070-b1db3f0a5da59cb5ba53a3d1fd99530868da3580-SNAPSHOT]

To work around this we changed our configuration to:

akka {
  http {
    host-connection-pool.max-open-requests = 512
  }
}

Environment
Nussknacker: staging@360bcb1b775e384de9cf2d11af15fcbbca72eb02