You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have an issue with one of the pods in cluster became unreachable due to the fact that Eureka client did not update the service registry cache. Initially this pod started to log WARNs that the thread pool is full:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3201d694[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@7ae6c02f[Wrapped task = com.netflix.discovery.DiscoveryClient$CacheRefreshThread@6c874d5e]] rejected from java.util.concurrent.ThreadPoolExecutor@60790f93[Running, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 20397]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:66)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Then after 4 days, the dependent service was redeployed, which caused hosts to change, however the affected pod failed to get the updated list of hosts, and started to fail all the requests, since the dependent service is used in every request.
I/O exception (java.net.NoRouteToHostException) caught when processing request to ->http://10.164.92.119:8080: No route to host (Host unreachable)
This caused a temporary outage on production system until we noticed the pod under issue and rebooted it.
All other pods of the same system did not have this issue and were stable.
After some investigation, it seems that this issue was reproducible couple of times on different versions of eureka-client:
For reference: #907
The version of eureka-client is at the moment: 1.10.17
Spring Cloud version: 3.1.2
java.util.concurrent.TimeoutException: null
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:68)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
We have an issue with one of the pods in cluster became unreachable due to the fact that Eureka client did not update the service registry cache. Initially this pod started to log WARNs that the thread pool is full:
Then after 4 days, the dependent service was redeployed, which caused hosts to change, however the affected pod failed to get the updated list of hosts, and started to fail all the requests, since the dependent service is used in every request.
This caused a temporary outage on production system until we noticed the pod under issue and rebooted it.
All other pods of the same system did not have this issue and were stable.
After some investigation, it seems that this issue was reproducible couple of times on different versions of eureka-client:
For reference: #907
The version of eureka-client is at the moment: 1.10.17
Spring Cloud version: 3.1.2
Also we noticed that eureka-client was updated only in major release of Spring Cloud:
https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-netflix-eureka-client/4.0.0
The text was updated successfully, but these errors were encountered: