serverinfo page stops responding

This topic contains 6 replies, has 2 voices, and was last updated by Profile photo of Peter Major Peter Major 3 months ago.

  • Author
    Posts
  • #15616
    Profile photo of hanibey hanibey 
    Participant

    Hello,

    I have installed OpenAM v13 on JBoss EAP 6.4 with OpenDJ v3 as the data and config store.
    If i login to OpenAM, leave it for a while then try to open another page inside the admin XUI, it hangs and i never receive a response. If i try to open a page that is still using the old interface, it works fine. I’m using the server name as DNS and logging in to the top level realm.
    While troubleshooting, i noticed, using the browser developer tools, that the cause of this is a call to /json/serverinfo/* that is hanging and never returns.
    I had to restart the app server for OpenAM to start responding again.

    Any ideas as to the cause of this?

    #15828
    Profile photo of hanibey hanibey 
    Participant

    This is becoming much more frequent. Whenever i do some action on OpenAM then leave it for a couple of minutes, all calls to /json/serverinfo/* stop responding.
    I’m using Oracle JDK1.8 btw.

    #15842
    Profile photo of Peter Major Peter Major 
    Moderator

    You should run a jstack against the hanging JVM and have a look at which threads are blocked and why.

    #15853
    Profile photo of hanibey hanibey 
    Participant

    Hi Peter,

    Thank you for your response.
    I have noticed that whenever i open an Authorization Policy Set (or after 1 or 2 clicks inside), OpenAM will just hang. This started happening after i started adding more and more policies to the set.

    So I tried your recommendation and noticed the following blocked threads:

    "http-10.0.x.x:8080-27" #238 daemon prio=5 os_prio=0 tid=0x00007fb53003e800 nid=0x2657 waiting for monitor entry [0x00007fb54cac7000]
       java.lang.Thread.State: BLOCKED (on object monitor)
            at com.google.inject.Scopes$1$1.get(Scopes.java:63)
            - waiting to lock <0x0000000701985a90> (a java.lang.Class for com.google.inject.internal.InternalInjectorCreator)
            at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
            at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
            at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
            at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
            at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1009)
            at org.forgerock.guice.core.InjectorHolder.getInstance(InjectorHolder.java:93)
            at org.forgerock.openam.http.HandlerProvider.handle(HandlerProvider.java:48)
            at org.forgerock.openam.http.HttpRoute$3.handle(HttpRoute.java:142)
            at org.forgerock.http.routing.Router.handle(Router.java:92)
            at org.forgerock.http.handler.Chain.handle(Chain.java:57)
            at org.forgerock.http.filter.TransactionIdInboundFilter.filter(TransactionIdInboundFilter.java:60)
            at org.forgerock.http.handler.Chain.handle(Chain.java:55)
            at org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:222)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
            at org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
            at com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:106)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
            at org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:51)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
            at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
            at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:559)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854)
            at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:656)
            - locked <0x00000007b8cee3b0> (a org.apache.coyote.http11.Http11Processor)
            at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
            at java.lang.Thread.run(Thread.java:745)

    There are usually around 30 http threads that are blocked waiting on a lock that is locked by one of the http threads:

    "http-10.0.x.x:8080-2" #213 daemon prio=5 os_prio=0 tid=0x00007fb5300e5000 nid=0x260b in Object.wait() [0x00007fb5619f4000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            at java.lang.Object.wait(Object.java:502)
            at org.forgerock.util.promise.PromiseImpl.await(PromiseImpl.java:572)
            - locked <0x0000000798023b90> (a org.forgerock.opendj.ldap.LDAPConnectionFactory$ConnectionImpl$LdapPromiseImplWrapper$1)
            at org.forgerock.util.promise.PromiseImpl.getOrThrow(PromiseImpl.java:143)
            at org.forgerock.opendj.ldap.spi.LdapPromiseWrapper.getOrThrow(LdapPromiseWrapper.java:90)
            at org.forgerock.opendj.ldap.spi.LdapPromiseImpl.getOrThrow(LdapPromiseImpl.java:44)
            at org.forgerock.opendj.ldap.AbstractAsynchronousConnection.blockingGetOrThrow(AbstractAsynchronousConnection.java:109)
            at org.forgerock.opendj.ldap.AbstractAsynchronousConnection.search(AbstractAsynchronousConnection.java:104)
            at org.forgerock.opendj.ldap.AbstractConnection.searchSingleEntry(AbstractConnection.java:398)
            at org.forgerock.opendj.ldap.CachedConnectionPool$PooledConnection.searchSingleEntry(CachedConnectionPool.java:544)
            at com.sun.identity.sm.ldap.SMSLdapObject.read(SMSLdapObject.java:275)
            at com.sun.identity.sm.SMSEntry.read(SMSEntry.java:699)
            at com.sun.identity.sm.SMSEntry.read(SMSEntry.java:676)
            at com.sun.identity.sm.SMSEntry.<init>(SMSEntry.java:469)
            at com.sun.identity.sm.CachedSMSEntry.getInstance(CachedSMSEntry.java:383)
            at com.sun.identity.sm.ServiceConfigImpl.checkAndUpdatePermission(ServiceConfigImpl.java:646)
            at com.sun.identity.sm.ServiceConfigImpl.getInstance(ServiceConfigImpl.java:529)
            at com.sun.identity.sm.ServiceConfigImpl.getInstance(ServiceConfigImpl.java:484)
            at com.sun.identity.sm.ServiceConfigManagerImpl.getGlobalConfig(ServiceConfigManagerImpl.java:204)
            at com.sun.identity.sm.ServiceConfigManager.getGlobalConfig(ServiceConfigManager.java:253)
            at org.forgerock.openam.rest.VersionBehaviourConfigListener.updateSettings(VersionBehaviourConfigListener.java:72)
            at org.forgerock.openam.rest.VersionBehaviourConfigListener.register(VersionBehaviourConfigListener.java:63)
            at org.forgerock.openam.rest.VersionBehaviourConfigListener.<init>(VersionBehaviourConfigListener.java:51)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
            at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:85)
            at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
            at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
            at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
            at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
            at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
            at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
            at com.google.inject.Scopes$1$1.get(Scopes.java:65)
            - locked <0x0000000701985a90> (a java.lang.Class for com.google.inject.internal.InternalInjectorCreator)
            at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
            at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
            at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
            at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
            at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1009)
            at org.forgerock.guice.core.InjectorHolder.getInstance(InjectorHolder.java:93)
            at org.forgerock.openam.http.HandlerProvider.handle(HandlerProvider.java:48)
            at org.forgerock.openam.http.HttpRoute$3.handle(HttpRoute.java:142)
            at org.forgerock.http.routing.Router.handle(Router.java:92)
            at org.forgerock.http.handler.Chain.handle(Chain.java:57)
            at org.forgerock.http.filter.TransactionIdInboundFilter.filter(TransactionIdInboundFilter.java:60)
            at org.forgerock.http.handler.Chain.handle(Chain.java:55)
            at org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:222)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
            at org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
            at com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:106)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
            at org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:51)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
            at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
            at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:559)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854)
            at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:656)
            - locked <0x00000007b8d96458> (a org.apache.coyote.http11.Http11Processor)
            at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
            at java.lang.Thread.run(Thread.java:745)

    I saw the reference to LDAP, and in my current setup i have 2 OpenDJ servers doing replication and OpenAM is pointing to the load balancer IP in front of them but i don’t know if that has anything to do with this issue.

    Thank you.

    #15854
    Profile photo of Peter Major Peter Major 
    Moderator
    #15858
    Profile photo of hanibey hanibey 
    Participant

    Thanks a lot for that, i’m trying to do a quick fix for it on my installation.
    If i call the policy evaluation API, would that trigger this bug as well? Because that would explain a lot of the issues i’m encountering.

    #15861
    Profile photo of Peter Major Peter Major 
    Moderator

    It’s very possible. I would strongly recommend to patch this in your environment.

Viewing 7 posts - 1 through 7 (of 7 total)

You must be logged in to reply to this topic.

©2017 ForgeRock - we provide an identity and access platform to secure every online relationship for the enterprise market, educational sector and even entire countries. Click to view our privacy policy and terms of use.

Log in with your credentials

Forgot your details?