AM5.5 crashes after 2-3 hours

Tagged: , ,

This topic has 3 replies, 3 voices, and was last updated 4 years, 8 months ago by mohammedanask.

  • Author
    Posts
  • #20541
     mohammedanask
    Participant

    Hello,

    I have an AM5.5 service running with external OpenDJ and OpenIG. After couple of hours, I notice this error in /var/lib/tomcat7/logs/catalina.out log:

    
    Error occurred while enforcing retention policy Free Number Retention Policy cn=File Count Retention Policy,cn=Log Retention Policies,cn=config for logger cn=Replication Repair Logger,cn=Loggers,cn=config: DirectoryException: Error occurred while listing log files named by policy with initial file name /usr/share/tomcat7/openam/opends/logs/replication (FileNumberRetentionPolicy.java:69 MultifileTextWriter.java:370)
    Error occurred while enforcing retention policy Free Number Retention Policy cn=File Count Retention Policy,cn=Log Retention Policies,cn=config for logger cn=File-Based Error Logger,cn=Loggers,cn=config: DirectoryException: Error occurred while listing log files named by policy with initial file name /usr/share/tomcat7/openam/opends/logs/errors (FileNumberRetentionPolicy.java:69 MultifileTextWriter.java:370)
    Jan 14, 2018 6:33:54 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
    SEVERE: Socket accept failed
    java.net.SocketException: Too many open files (Accept failed)
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
            at java.net.ServerSocket.implAccept(ServerSocket.java:545)
            at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:205)
            at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:219)
            at java.lang.Thread.run(Thread.java:748)
    
    Jan 14, 2018 6:33:55 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
    SEVERE: Socket accept failed
    java.net.SocketException: Too many open files (Accept failed)
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
            at java.net.ServerSocket.implAccept(ServerSocket.java:545)
            at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:205)
            at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:219)
            at java.lang.Thread.run(Thread.java:748)
    
    Jan 14, 2018 6:33:57 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
    SEVERE: Socket accept failed
    java.net.SocketException: Too many open files (Accept failed)
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
            at java.net.ServerSocket.implAccept(ServerSocket.java:545)
            at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:205)
            at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:219)
            at java.lang.Thread.run(Thread.java:748)
    

    I have to restart tomcat7 service to fix the issue. But I see the problem again after 2-3 hours.
    Any help is appreciated

    #20544
     Andy Cory
    Participant

    Have you looked at the docs on setting file descriptor limits?

    https://backstage.forgerock.com/docs/am/5.5/install-guide/#prerequisites-file-descriptors

    -Andy

    #20551
     gary.rasmussen
    Participant

    Increase ulimit on your server for the user running tomcat.
    To check ulimit use the ulimit -n command
    To increase for current session use with a value you want to increase to like “ulimit -n 102400”
    And you can make that permanent changing the /etc/security/limits.conf.

    There are recommended settings in the installation guide for the limits:

    https://backstage.forgerock.com/docs/am/5.5/install-guide/#prerequisites-file-descriptors

    #20567
     mohammedanask
    Participant

    Thank you, above fixes looks to have resolved the issue.

Viewing 4 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.

©2022 ForgeRock - we provide an identity and access platform to secure every online relationship for the enterprise market, educational sector and even entire countries. Click to view our privacy policy and terms of use.

Log in with your credentials

Forgot your details?