Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I have a REST API written in Scala with Play framework and Akka, deployed on the production server. Periodically the container with the API crashes with the following error:

[ERROR] [01/18/2023 13:14:02.762] [application-akka.actor.internal-dispatcher-3] [akka://application/system/IO-TCP/selectors/$a/0] Accept error: could not accept new connection
java.io.IOException: Too many open files
        at java.base/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at akka.io.TcpListener.acceptAllPending(TcpListener.scala:128)
        at akka.io.TcpListener$$anonfun$bound$1.applyOrElse(TcpListener.scala:96)
        at akka.actor.Actor.aroundReceive(Actor.scala:537)
        at akka.actor.Actor.aroundReceive$(Actor.scala:535)
        at akka.io.TcpListener.aroundReceive(TcpListener.scala:36)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579)
        at akka.actor.ActorCell.invoke(ActorCell.scala:547)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
I have tried to increase a ulimit. It is now set to 1000000, didn't help at all. I assume it is connected to the amount of TCP connections but I cannot pinpoint the exact cause, nor can I reproduce it on my machine.
Would greatly appreciate any tips on how to handle this.
                I checked it with ulimit -n and it shows correct numbers.  So I don't think that's the case. It is an Azure VM so it's slightly different from the simple server machines I guess, but there is no start up scripts or anything, it should be up and running permanently.
– Igor Molchanov
                Jan 30 at 15:50
                It is applied globally, but can be overridden per process. It's gotta be ulimit ... Unless you are leaking connections. That's also a possibility I suppose.
– Dima
                Jan 31 at 13:39
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.