Tagged: #openDJ, jmx, monitoring, replication
This topic has 11 replies, 3 voices, and was last updated 3 years, 4 months ago by JnRouvignac.
-
AuthorPosts
-
February 6, 2019 at 8:02 pm #24706
matthewponzio
ParticipantHi everyone,
My question here is regarding JMX Monitoring on DJ 6.0. I have a correctly configured JMX Handler, WITH the appropriate permissions. We are using Datadog Agent v6 to collect JMX monitoring statistics. The DD agent is correctly configured, with correct permission and can see the JMX Mbeans – as such:
[[email protected] ~]# datadog-agent jmx list everything | grep ds-mon-product-name 2019-02-06 18:46:30 UTC | INFO | (jmxfetch.go:209 in func1) | Not Matching: Bean name: org.opends.server:Name=rootDSE,Rdn1=cn-monitor - Attribute name: ds-mon-product-name - Attribute type: java.lang.String
Take out the
grep
part of that command, and you can see thousands of Mbeans. My point here is that JMX and Datadog are working and there are no permission problems.So with that said…
The problem I am having is that I CANNOT see the all important Replication monitoring Mbean –
ds-mon-current-delay
(among others – I also cannot seeds-mon-backend-entry-count
)[[email protected] ~]# datadog-agent jmx list everything | grep ds-mon-current-delay [[email protected] ~]#
ds-mon-current-delay
is just not there.However, I ABSOLUTELY CAN query this attribute via LDAP:
[[email protected] root]$ /prod/dsd/apps/opendj/bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -w admin1234 --baseDN cn=monitor "(ds-mon-backend-entry-count=*)" ds-mon-backend-entry-count dn: ds-cfg-backend-id=__config.ldif__,cn=backends,cn=monitor ds-mon-backend-entry-count: 188 dn: ds-cfg-backend-id=adminRoot,cn=backends,cn=monitor ds-mon-backend-entry-count: 33 dn: ds-cfg-backend-id=ads-truststore,cn=backends,cn=monitor ds-mon-backend-entry-count: 22 dn: ds-cfg-backend-id=backup,cn=backends,cn=monitor ds-mon-backend-entry-count: 1 dn: ds-cfg-backend-id=changelog,cn=backends,cn=monitor ds-mon-backend-entry-count: -1 dn: ds-cfg-backend-id=monitor,cn=backends,cn=monitor ds-mon-backend-entry-count: 125 dn: ds-cfg-backend-id=rootUser,cn=backends,cn=monitor ds-mon-backend-entry-count: 1 dn: ds-cfg-backend-id=schema,cn=backends,cn=monitor ds-mon-backend-entry-count: 1 dn: ds-cfg-backend-id=ssousers,cn=backends,cn=monitor ds-mon-backend-entry-count: 9 dn: ds-cfg-backend-id=tasks,cn=backends,cn=monitor ds-mon-backend-entry-count: 3 dn: ds-cfg-backend-id=userRoot,cn=backends,cn=monitor ds-mon-backend-entry-count: 14
So, what’s going on here? I can see myriad JMX attributes, but not some of the ones I need. Is there some additional JMX configuration I need to add in order for these attributes to show up?
February 12, 2019 at 12:27 pm #24772Ludo
ModeratorI’m sorry, I’m not familiar with DataDog-agent, but DJ JMX access is secured with an LDAP user that must have the
jmx-read
andmonitor-read
privilege.
Also you need to connect to the Directory Server using a service URL, as documented in https://backstage.forgerock.com/docs/ds/6/admin-guide/#jmx-access
Otherwise, when connecting directly to the process, you are not authorised to see the content of the DS specific MBeans.February 12, 2019 at 6:19 pm #24777matthewponzio
ParticipantHi Ludo, thanks for the reply. Yes, we most definitely have the correct permissions. We’re using cn=Directory Manager user and we also are using the correct service URL.
As I said, I am able to see most all of the JMX attributes/metrics under cn=montior, so I’m all but certain we have the correct permissions and JMX URL config.
The Datadog agent is configured via a YAML file and here is the relevant portion:
instances: - host: localhost port: 1689 user: cn=Directory Manager password: <adminpass> conf: - include: domain: org.opends.server bean: - org.opends.server:Name=rootDSE,Rdn1=cn-monitor,Rdn2=cn-ssousers_Backend attribute: - ds-backend-entry-count jmx_url: "service:jmx:rmi:///jndi/rmi://localhost:1689/org.opends.server.protocols.jmx.client-unknown" name: jmx_instance
I nevertheless attempted to add the permissions for cn=Directory Manager as directed in the documentation, but as you can see, the permissions are already present:
/prod/dsd/apps/opendj/bin/ldapmodify --port 1389 --bindDN "cn=Directory Manager" --bindPassword <admin_pass> <<'EOF' > dn: cn=Directory Manager > changetype: modify > add: ds-privilege-name > ds-privilege-name: jmx-notify > ds-privilege-name: jmx-read > ds-privilege-name: monitor-read > EOF # The LDAP modify request failed: 20 (Attribute or Value Exists) # Additional Information: Entry cn=Directory Manager cannot be modified because it would have resulted in one or more duplicate values for attribute ds-privilege-name: jmx-notify, jmx-read, monitor-read
Thanks,
MattFebruary 12, 2019 at 6:39 pm #24778matthewponzio
ParticipantI’ve shortened the output as much as possible, but this is the list of all the JMX Attributes/MBeans that the Datadog agent can see. It’s quite a lot, so that’s why I’m fairly sure my permissions and configuration are correct…but conspicuously missing from this list are
ds-mon-current-delay
andds-mon-backend-entry-count
Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-schema - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-schema - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-adstruststore - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-adstruststore - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-tasks - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-tasks - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-ssousers - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-ssousers - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-ssousers - Attribute name: ds-mon-backend-ttl-is-running - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-ssousers - Attribute name: ds-mon-db-version - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-monitor - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-monitor - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-JMX - Attribute name: ds-mon-protocol - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-JMX - Attribute name: ds-mon-listen-address - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-server-state - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-connected-to-server-hostport - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-ssl-encryption - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-ds-mon-updates-totals-per-replay-thread - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-ou-ssousersdc-capitalonedc-com - Attribute name: ds-mon-status - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-java-version - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-java-vendor - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-java-home - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-version - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-vendor - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-architecture - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-arguments - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-supported-tls-protocols - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-supported-tls-ciphers - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-jvm - Attribute name: ds-mon-jvm-class-path - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-backup - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-backup - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-configldif - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-configldif - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-disk_space_monitor,Rdn3=dsmondiskroot-proddata - Attribute name: ds-mon-disk-state - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-disk_space_monitor,Rdn3=dsmondiskroot-proddata - Attribute name: ds-mon-disk-dir - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-disk_space_monitor,Rdn3=dsmondiskroot-proddata - Attribute name: ds-mon-disk-root - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-LDAP - Attribute name: ds-mon-protocol - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-LDAP - Attribute name: ds-mon-listen-address - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-server-state - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-connected-to-server-hostport - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-ssl-encryption - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-ds-mon-updates-totals-per-replay-thread - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-admin_data - Attribute name: ds-mon-status - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-schema - Attribute name: ds-mon-connected-to-server-hostport - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-schema - Attribute name: ds-mon-ssl-encryption - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-schema - Attribute name: ds-mon-ds-mon-updates-totals-per-replay-thread - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-cn-schema - Attribute name: ds-mon-status - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-adminRoot - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-adminRoot - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-Certificates,Rdn3=cn-Default_Key_Manager,Rdn4=dsmonalias-servercert - Attribute name: ds-mon-alias - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-HTTP - Attribute name: ds-mon-protocol - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-connection_handlers,Rdn3=cn-HTTP - Attribute name: ds-mon-listen-address - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-rootUser - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-rootUser - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-Administration_Connector - Attribute name: ds-mon-protocol - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-Administration_Connector - Attribute name: ds-mon-listen-address - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-product-name - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-short-name - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-vendor-name - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-full-version - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-compact-version - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-revision - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-build-time - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-version-qualifier - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-fix-ids - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-working-directory - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-os-name - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-os-version - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-os-architecture - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-system-name - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-install-path - Attribute type: java.lang.String Rdn1=cn-monitor - Attribute name: ds-mon-instance-path - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-dc-capitalonedc-com - Attribute name: ds-mon-connected-to-server-hostport - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-dc-capitalonedc-com - Attribute name: ds-mon-ssl-encryption - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-dc-capitalonedc-com - Attribute name: ds-mon-ds-mon-updates-totals-per-replay-thread - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-replication,Rdn3=cn-replicas,Rdn4=dsmondomainname-dc-capitalonedc-com - Attribute name: ds-mon-status - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-userRoot - Attribute name: ds-mon-backend-is-private - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-userRoot - Attribute name: ds-mon-backend-writability-mode - Attribute type: java.lang.String Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-userRoot - Attribute name: ds-mon-backend-ttl-is-running - Attribute type: boolean Rdn1=cn-monitor,Rdn2=cn-backends,Rdn3=dscfgbackendid-userRoot - Attribute name: ds-mon-db-version - Attribute type: java.lang.String
February 13, 2019 at 12:45 pm #24784JnRouvignac
ParticipantHello,
According to your output, there are only booleans and Strings appearing.
It means everything else is not showing up including LDAP integers. They are currently mapped tojava.lang.BigInteger
.
It looks like the Datadog agent only understands java primitive types + String type.
In the past I have been looking at best practices for the “type” of theMBeanAttributeInfo
, but I could not find anything.Thanks to your report, I found the list of types supported by the Datadog JMX agent: https://github.com/DataDog/jmxfetch/blob/master/src/main/java/org/datadog/jmxfetch/Instance.java#L32-L53
For the LDAP integers, I think we can simply map tojava.lang.Number
instead ofjava.lang.BigInteger
. That will solve your problem.
We have a number of other types which needs to be remapped and these may need a bit more work.I have raised https://bugster.forgerock.org/jira/browse/OPENDJ-6007 to track this work.
Thanks again for your very useful report.
Jean-Noel
PS:
Looking at the datadog documentation for the configuration (https://docs.datadoghq.com/integrations/java/), it is possible to a map ametric_type
to each attribute with eithergauge
acounter
, and this is exactly what we have internally.
Please refer to https://ea.forgerock.com/docs/ds/monitoring-guide/chap-monitoring.html#monitoring-metrics-ldap to identify the correct metric_type.
Attributes using theCounter metric
syntax can be mapped tocounter
, while attributes using theInteger
syntax can be mapped togauge
.I can also see mentions of
OneMinuteRate
,75thPercentile
,95thPercentile
and99thPercentile
.
Good news is that we have these too! See https://ea.forgerock.com/docs/ds/monitoring-guide/chap-monitoring.html#monitoring-types.
Bad news is that they are mapped to a JMX type oforg.forgerock.json.JsonValue
, thus not collected by the agent. We need to find a better way to expose them to JMX.February 13, 2019 at 3:50 pm #24787matthewponzio
ParticipantWow, awesome. Thanks so much for the diligent follow up! So yeah, looking the JMX MBeans from my DJ 3.5 servers, I see that
ds-backend-entry-count
wasjava.lang.String
.I’ll definitely take a look at the docs you mentioned in your PS. Thanks again!!
February 13, 2019 at 3:58 pm #24788matthewponzio
ParticipantHey, looks like I don’t have access to https://ea.forgerock.com/docs..
Forbidden You don't have permission to access /docs/ds/monitoring-guide/chap-monitoring.html on this server.
Could be because my Backstage account isn’t linked to a paid Support Agreement? The company I’m working at does have a support agreement, but I’m a consultant, and so I’m not attached to it.
February 13, 2019 at 4:29 pm #24789JnRouvignac
ParticipantAw no that’s my fault.
I should not have linked to early access docs when final docs exist.
Here is the same content in existing final docs:
– https://backstage.forgerock.com/docs/ds/6/reference/#monitoring-metrics-ldap
– https://backstage.forgerock.com/docs/ds/6/reference/#monitoring-typesFebruary 13, 2019 at 4:34 pm #24790JnRouvignac
ParticipantBy the way, I can see that datadog has support for prometheus exposition format of metrics:
https://www.datadoghq.com/blog/monitor-prometheus-metrics/If you have the possibility, I would strongly advise you to consider using this instead of JMX. Granted, it does not support string metrics, but that’s the only downside. For all the rest, JMX is a very limited subset of the prometheus exposition format. See https://backstage.forgerock.com/docs/ds/6/reference/#monitoring-metrics-prometheus for what DS exposes in this format. It was one of the big features for the whole ForgeRock stack in the 6.0 release.
February 13, 2019 at 5:44 pm #24792matthewponzio
ParticipantAh, cool. So I’m trying to configure prometheus monitoring, but I can’t seem to find the prometheus endpoint url for OpenDJ in the docs? I tried
http://localhost:8080/json/metrics/prometheus
andhttp://localhost/json/metrics/prometheus
, but no good:prometheus (2.0.0) ------------------ Instance ID: prometheus:a054d854bd81b5a3 [ERROR] Total Runs: 1 Metric Samples: 0, Total: 0 Events: 0, Total: 0 Service Checks: 1, Total: 1 Average Execution Time : 5ms Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /json/metrics/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e22d6c0d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
February 13, 2019 at 5:53 pm #24795matthewponzio
ParticipantOh..never mind, I found it…under HTTP-Based monitoring —
http://localhost:8080/metrics/prometheus
Now just need to figure out how to authenticate to it properly.February 13, 2019 at 11:29 pm #24797JnRouvignac
ParticipantHave a look at https://backstage.forgerock.com/docs/ds/6/admin-guide/index.html#setup-admin-endpoint
I think you have already setup the
monitor-read
privileges, so all you need to do now is to authenticate using the appropriate authentication mechanism (HTTP Basic Auth by default). -
AuthorPosts
You must be logged in to reply to this topic.