Getting "possibly out of memory or process/resource limits reached" followed by "BoundedExecutor is in a failed state" after upgrading to version 446

[sravyanu-kore]

Hi Team,

We are getting "possibly out of memory or process/resource limits reached" followed by "BoundedExecutor is in a failed state" after upgrading to version 446. Our previous version was 430. Downgrading to 430 version is solving the issue. However, we are getting these errors once we upgrade to 446 in our higher environments. Any help is appreciated.

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:1553)
at java.base/java.lang.System$2.start(System.java:2577)
at java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1375)
at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:640)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:73)
at io.trino.execution.executor.scheduler.FairScheduler.submit(FairScheduler.java:155)
at io.trino.execution.executor.dedicated.TaskEntry.runSplit(TaskEntry.java:136)
at io.trino.execution.executor.dedicated.ThreadPerDriverTaskExecutor.enqueueSplits(ThreadPerDriverTaskExecutor.java:158)
at io.trino.execution.SqlTaskExecution.enqueueDriverSplitRunner(SqlTaskExecution.java:402)
at io.trino.execution.SqlTaskExecution.scheduleDriversForTaskLifeCycle(SqlTaskExecution.java:391)
at io.trino.execution.SqlTaskExecution.start(SqlTaskExecution.java:207)
at io.trino.execution.SqlTask.tryCreateSqlTaskExecution(SqlTask.java:572)
at io.trino.execution.SqlTask.updateTask(SqlTask.java:514)
at io.trino.execution.SqlTaskManager.doUpdateTask(SqlTaskManager.java:558)
at io.trino.execution.SqlTaskManager.lambda$updateTask$9(SqlTaskManager.java:490)
at io.trino.$gen.Trino_446____20240921_071353_2.call(Unknown Source)

ERROR http-worker-713220 com.google.common.util.concurrent.AbstractFuture RuntimeException while executing runnable CallbackListener{io.airlift.jaxrs.AsyncResponseHandler$1@57de90b0} with executor io.airlift.concurrent.BoundedExecutor@3311db54
java.util.concurrent.RejectedExecutionException: BoundedExecutor is in a failed state
at io.airlift.concurrent.BoundedExecutor.execute(BoundedExecutor.java:55)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:762)
at com.google.common.util.concurrent.FluentFuture$TrustedFuture.addListener(FluentFuture.java:114)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1103)
at io.airlift.jaxrs.AsyncResponseHandler.bindAsyncResponse(AsyncResponseHandler.java:49)
at io.trino.server.TaskResource.getResults(TaskResource.java:351)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:263)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)

[sriharshaj]

It can be related to this:
https://trinodb.slack.com/archives/CGB0QHWSW/p1721671769818799

You may want to try 454, and see if it fixes the issue.

[hashhar]

unable to create native thread: possibly out of memory or process/resource limits reached

This suggests you have reached nproc limits possibly. What does ulimits -a return for the user as which Trino process is running?

[Rakeshya-Kore]

Hi Trino Team,

Upon investigation, it was observed that the number of splitRunner threads increased by more than 2.5x times in version 446, even with dynamic-filtering turned off. In comparison, the thread count was significantly lower in version 430.

Findings:
From various sources, including the Trino issue #22642, it was identified that the experimental.thread-per-driver-scheduler-enabled configuration flag has been enabled by default since version 438.

To test the impact of this flag, we disabled it in version 446. After doing so, the number of threads observed was consistent with the thread count seen in version 430.

Requesting confirmation:
We are requesting confirmation from the Trino team as to whether the observed issue is solely caused by the experimental.thread-per-driver-scheduler-enabled flag, and whether disabling it resolves the problem in the long term?

[hashhar]

Yes, this is correct analysis.

[Rakeshya-Kore]

Hi Team,
Can someone respond on this.