OpenDJ 6.0 – JMX Monitoring

[matthewponzio Participant]

Hi everyone,

My question here is regarding JMX Monitoring on DJ 6.0. I have a correctly configured JMX Handler, WITH the appropriate permissions. We are using Datadog Agent v6 to collect JMX monitoring statistics. The DD agent is correctly configured, with correct permission and can see the JMX Mbeans – as such:

[root@dj6-rs1 ~]# datadog-agent jmx list everything | grep ds-mon-product-name
2019-02-06 18:46:30 UTC | INFO | (jmxfetch.go:209 in func1) |        Not Matching: Bean name: org.opends.server:Name=rootDSE,Rdn1=cn-monitor - Attribute name: ds-mon-product-name  - Attribute type: java.lang.String

Take out the grep part of that command, and you can see thousands of Mbeans. My point here is that JMX and Datadog are working and there are no permission problems.

So with that said…

The problem I am having is that I CANNOT see the all important Replication monitoring Mbean – ds-mon-current-delay (among others – I also cannot see ds-mon-backend-entry-count)

[root@dj6-rs1 ~]# datadog-agent jmx list everything | grep ds-mon-current-delay
[root@dj6-rs1 ~]#

ds-mon-current-delay is just not there.

However, I ABSOLUTELY CAN query this attribute via LDAP:

[wsfr@dj6-rs1 root]$ /prod/dsd/apps/opendj/bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -w admin1234  --baseDN cn=monitor "(ds-mon-backend-entry-count=*)" ds-mon-backend-entry-count

dn: ds-cfg-backend-id=__config.ldif__,cn=backends,cn=monitor
ds-mon-backend-entry-count: 188

dn: ds-cfg-backend-id=adminRoot,cn=backends,cn=monitor
ds-mon-backend-entry-count: 33

dn: ds-cfg-backend-id=ads-truststore,cn=backends,cn=monitor
ds-mon-backend-entry-count: 22

dn: ds-cfg-backend-id=backup,cn=backends,cn=monitor
ds-mon-backend-entry-count: 1

dn: ds-cfg-backend-id=changelog,cn=backends,cn=monitor
ds-mon-backend-entry-count: -1

dn: ds-cfg-backend-id=monitor,cn=backends,cn=monitor
ds-mon-backend-entry-count: 125

dn: ds-cfg-backend-id=rootUser,cn=backends,cn=monitor
ds-mon-backend-entry-count: 1

dn: ds-cfg-backend-id=schema,cn=backends,cn=monitor
ds-mon-backend-entry-count: 1

dn: ds-cfg-backend-id=ssousers,cn=backends,cn=monitor
ds-mon-backend-entry-count: 9

dn: ds-cfg-backend-id=tasks,cn=backends,cn=monitor
ds-mon-backend-entry-count: 3

dn: ds-cfg-backend-id=userRoot,cn=backends,cn=monitor
ds-mon-backend-entry-count: 14

So, what’s going on here? I can see myriad JMX attributes, but not some of the ones I need. Is there some additional JMX configuration I need to add in order for these attributes to show up?

[Ludo Moderator]

I’m sorry, I’m not familiar with DataDog-agent, but DJ JMX access is secured with an LDAP user that must have the jmx-read and monitor-read privilege.
Also you need to connect to the Directory Server using a service URL, as documented in https://backstage.forgerock.com/docs/ds/6/admin-guide/#jmx-access
Otherwise, when connecting directly to the process, you are not authorised to see the content of the DS specific MBeans.

[matthewponzio Participant]

Hi Ludo, thanks for the reply. Yes, we most definitely have the correct permissions. We’re using cn=Directory Manager user and we also are using the correct service URL.

As I said, I am able to see most all of the JMX attributes/metrics under cn=montior, so I’m all but certain we have the correct permissions and JMX URL config.

The Datadog agent is configured via a YAML file and here is the relevant portion:

instances:
    - host: localhost
      port: 1689
      user: cn=Directory Manager
      password: <adminpass>

      conf:
          - include:
              domain: org.opends.server
              bean:
                  - org.opends.server:Name=rootDSE,Rdn1=cn-monitor,Rdn2=cn-ssousers_Backend
              attribute:
                  - ds-backend-entry-count
         
           
      jmx_url: "service:jmx:rmi:///jndi/rmi://localhost:1689/org.opends.server.protocols.jmx.client-unknown"
      name: jmx_instance

I nevertheless attempted to add the permissions for cn=Directory Manager as directed in the documentation, but as you can see, the permissions are already present:

/prod/dsd/apps/opendj/bin/ldapmodify --port 1389 --bindDN "cn=Directory Manager" --bindPassword <admin_pass> <<'EOF'
> dn: cn=Directory Manager
> changetype: modify
> add: ds-privilege-name
> ds-privilege-name: jmx-notify
> ds-privilege-name: jmx-read
> ds-privilege-name: monitor-read
> EOF
# The LDAP modify request failed: 20 (Attribute or Value Exists)
# Additional Information:  Entry cn=Directory Manager cannot be modified because it would have resulted in one or more duplicate values for attribute ds-privilege-name: jmx-notify, jmx-read, monitor-read

Thanks,
Matt

[matthewponzio Participant]

I’ve shortened the output as much as possible, but this is the list of all the JMX Attributes/MBeans that the Datadog agent can see. It’s quite a lot, so that’s why I’m fairly sure my permissions and configuration are correct…but conspicuously missing from this list are ds-mon-current-delay and ds-mon-backend-entry-count

Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-schema - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-schema - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-adstruststore - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-adstruststore - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-tasks - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-tasks - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-ssousers - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-ssousers - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-ssousers - Attribute name: ds-mon-backend-ttl-is-running  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-ssousers - Attribute name: ds-mon-db-version  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-monitor - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-monitor - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-JMX - Attribute name: ds-mon-protocol  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-JMX - Attribute name: ds-mon-listen-address  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-server-state  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-connected-to-server-hostport  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-ssl-encryption  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-ds-mon-updates-totals-per-replay-thread  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-status  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-java-version  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-java-vendor  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-java-home  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-version  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-vendor  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-architecture  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-arguments  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-supported-tls-protocols  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-supported-tls-ciphers  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-class-path  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-backup - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-backup - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-configldif - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-configldif - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-disk_space_monitor,Rdn3=dsmondiskroot-proddata - Attribute name: ds-mon-disk-state  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-disk_space_monitor,Rdn3=dsmondiskroot-proddata - Attribute name: ds-mon-disk-dir  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-disk_space_monitor,Rdn3=dsmondiskroot-proddata - Attribute name: ds-mon-disk-root  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-LDAP - Attribute name: ds-mon-protocol  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-LDAP - Attribute name: ds-mon-listen-address  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-server-state  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-connected-to-server-hostport  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-ssl-encryption  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-ds-mon-updates-totals-per-replay-thread  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-status  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-schema - Attribute name: ds-mon-connected-to-server-hostport  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-schema - Attribute name: ds-mon-ssl-encryption  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-schema - Attribute name: ds-mon-ds-mon-updates-totals-per-replay-thread  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-schema - Attribute name: ds-mon-status  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-adminRoot - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-adminRoot - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-Certificates,Rdn3=cn-Default_Key_Manager,Rdn4=dsmonalias-servercert - Attribute name: ds-mon-alias  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-HTTP - Attribute name: ds-mon-protocol  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-HTTP - Attribute name: ds-mon-listen-address  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-rootUser - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-rootUser - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-Administration_Connector - Attribute name: ds-mon-protocol  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-Administration_Connector - Attribute name: ds-mon-listen-address  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-product-name  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-short-name  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-vendor-name  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-full-version  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-compact-version  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-revision  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-build-time  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-version-qualifier  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-fix-ids  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-working-directory  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-os-name  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-os-version  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-os-architecture  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-system-name  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-install-path  - Attribute type: java.lang.String
Rdn1=cn-monitor - Attribute name: ds-mon-instance-path  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-dc-capitalonedc-com - Attribute name: ds-mon-connected-to-server-hostport  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-dc-capitalonedc-com - Attribute name: ds-mon-ssl-encryption  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-dc-capitalonedc-com - Attribute name: ds-mon-ds-mon-updates-totals-per-replay-thread  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-dc-capitalonedc-com - Attribute name: ds-mon-status  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-userRoot - Attribute name: ds-mon-backend-is-private  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-userRoot - Attribute name: ds-mon-backend-writability-mode  - Attribute type: java.lang.String
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-userRoot - Attribute name: ds-mon-backend-ttl-is-running  - Attribute type: boolean
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-userRoot - Attribute name: ds-mon-db-version  - Attribute type: java.lang.String

[JnRouvignac Participant]

Hello,

According to your output, there are only booleans and Strings appearing.
It means everything else is not showing up including LDAP integers. They are currently mapped to java.lang.BigInteger.
It looks like the Datadog agent only understands java primitive types + String type.
In the past I have been looking at best practices for the “type” of the MBeanAttributeInfo, but I could not find anything.

Thanks to your report, I found the list of types supported by the Datadog JMX agent: https://github.com/DataDog/jmxfetch/blob/master/src/main/java/org/datadog/jmxfetch/Instance.java#L32-L53
For the LDAP integers, I think we can simply map to java.lang.Number instead of java.lang.BigInteger. That will solve your problem.
We have a number of other types which needs to be remapped and these may need a bit more work.

I have raised https://bugster.forgerock.org/jira/browse/OPENDJ-6007 to track this work.

Thanks again for your very useful report.

Jean-Noel

 

PS:
Looking at the datadog documentation for the configuration (https://docs.datadoghq.com/integrations/java/), it is possible to a map a metric_type to each attribute with either gauge a counter, and this is exactly what we have internally.
Please refer to https://ea.forgerock.com/docs/ds/monitoring-guide/chap-monitoring.html#monitoring-metrics-ldap to identify the correct metric_type.
Attributes using the Counter metric syntax can be mapped to counter, while attributes using the Integer syntax can be mapped to gauge.

I can also see mentions of OneMinuteRate, 75thPercentile, 95thPercentile and 99thPercentile.
Good news is that we have these too! See https://ea.forgerock.com/docs/ds/monitoring-guide/chap-monitoring.html#monitoring-types.
Bad news is that they are mapped to a JMX type of org.forgerock.json.JsonValue, thus not collected by the agent. We need to find a better way to expose them to JMX.

[matthewponzio Participant]

Wow, awesome. Thanks so much for the diligent follow up! So yeah, looking the JMX MBeans from my DJ 3.5 servers, I see that ds-backend-entry-count was java.lang.String.

I’ll definitely take a look at the docs you mentioned in your PS. Thanks again!!

[matthewponzio Participant]

Hey, looks like I don’t have access to https://ea.forgerock.com/docs..

Forbidden
You don't have permission to access /docs/ds/monitoring-guide/chap-monitoring.html on this server.

Could be because my Backstage account isn’t linked to a paid Support Agreement? The company I’m working at does have a support agreement, but I’m a consultant, and so I’m not attached to it.

[JnRouvignac Participant]

Aw no that’s my fault.

I should not have linked to early access docs when final docs exist.
Here is the same content in existing final docs:
– https://backstage.forgerock.com/docs/ds/6/reference/#monitoring-metrics-ldap
– https://backstage.forgerock.com/docs/ds/6/reference/#monitoring-types

[JnRouvignac Participant]

By the way, I can see that datadog has support for prometheus exposition format of metrics:
https://www.datadoghq.com/blog/monitor-prometheus-metrics/

If you have the possibility, I would strongly advise you to consider using this instead of JMX. Granted, it does not support string metrics, but that’s the only downside. For all the rest, JMX is a very limited subset of the prometheus exposition format. See https://backstage.forgerock.com/docs/ds/6/reference/#monitoring-metrics-prometheus for what DS exposes in this format. It was one of the big features for the whole ForgeRock stack in the 6.0 release.

[matthewponzio Participant]

Ah, cool. So I’m trying to configure prometheus monitoring, but I can’t seem to find the prometheus endpoint url for OpenDJ in the docs? I tried http://localhost:8080/json/metrics/prometheus and http://localhost/json/metrics/prometheus, but no good:

    prometheus (2.0.0)
    ------------------
        Instance ID: prometheus:a054d854bd81b5a3 [ERROR]
        Total Runs: 1
        Metric Samples: 0, Total: 0
        Events: 0, Total: 0
        Service Checks: 1, Total: 1
        Average Execution Time : 5ms
        Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /json/metrics/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e22d6c0d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

[matthewponzio Participant]

Oh..never mind, I found it…under HTTP-Based monitoring — http://localhost:8080/metrics/prometheus Now just need to figure out how to authenticate to it properly.

[JnRouvignac Participant]

Have a look at https://backstage.forgerock.com/docs/ds/6/admin-guide/index.html#setup-admin-endpoint

I think you have already setup the monitor-read privileges, so all you need to do now is to authenticate using the appropriate authentication mechanism (HTTP Basic Auth by default).

[Author]

Posts