JMX Monitoring of OpenDJ Replication

This topic contains 2 replies, has 2 voices, and was last updated by  matthewponzio 4 months, 1 week ago.

  • Author
    Posts
  • #24174
     matthewponzio 
    Participant

    Hi all,

    We are using JMX to monitor our OpenDJ environment, including replication. We are seeing a discrepancy between what the JMX Mbean is reporting for missing_changes (jmx.org.opends.server.missing_changes) vs what the dsreplication status commands reports.

    Could this be because the JMX Mbean statistic for missing_changes is cumulative? Perhaps when the environment was deployed, there were missing_changes that spiked the JMX Mbean value, which then does not reset to 0 after the missing_changes have been replicated?

    Also notice that the number of missng_changes reported by the JMX Mbean (364801.0) is actually more than the total number of Entries (357196) as reported by dsreplication status.

    Here is the output from dsreplication status – It’s edited for formatting and security, but the important values are there, specifically M.C 0 for ALL SERVERS:

    Server : Entries : Replication enabled : DS ID : RS ID : RS Port (1) : M.C. : A.O.M.C.
    iam-fr-dev2-blue-udj-ds0 : 357196 : true : 29576 : (5) : : 0 :
    iam-fr-dev2-blue-udj-ds1 : 357196 : true : 18308 : (5) : : 0 :
    iam-fr-dev2-blue-udj-ds2 : 357196 : true : 23537 : (5) : : 0 :
    iam-fr-dev2-blue-udj-rs1 : 357196 : true : 6581 : 611 : 20059 : 0 :
    iam-fr-dev2-blue-udj-rs2 : 357196 : true : 10659 : 28878 : 20059 : 0 :

    And here is the output of our JMX Mbeans for missing_changes:

    jmx.org.opends.server.missing_changes[instance:jmx_instance,jmx_domain:org.opends.server,
    Rdn2:cn-Replication,
    Rdn3:cn-oussousersdc<our_domain>dccom,
    Rdn1:cn-monitor,
    Rdn4:cn-Replication_server_RS28878_iamfrdev2blueudjrs220059,
    Name:rootDSE] – 1544120059 = 364801.0
    jmx.org.opends.server.missing_changes[instance:jmx_instance,jmx_domain:org.opends.server,
    Rdn2:cn-Replication,
    Rdn3:cn-oussousersdc<our_domain>dccom,
    Rdn1:cn-monitor,
    Rdn4:cn-Replication_server_RS28878_iamfrdev2blueudjrs220059,
    Name:rootDSE,
    Rdn5:cn-Connected_replication_server_RS611_iamfrdev2blueudjrs120059] – 1544120059 = 364801.0

    jmx.org.opends.server.missing_changes[instance:jmx_instance,jmx_domain:org.opends.server,
    Rdn2:cn-Replication,
    Rdn3:cn-oussousersdc<our_domain>dccom,
    Rdn1:cn-monitor,
    Rdn6:cn-Connected_directory_server_DS6581_iamfrdev2blueudjrs148616,
    Rdn4:cn-Replication_server_RS28878_iamfrdev2blueudjrs220059,
    Name:rootDSE,
    Rdn5:cn-Connected_replication_server_RS611_iamfrdev2blueudjrs120059] – 1544120059 = 0.0
    jmx.org.opends.server.missing_changes[instance:jmx_instance,jmx_domain:org.opends.server,
    Rdn2:cn-Replication,
    Rdn3:cn-oussousersdc<our_domain>dccom,
    Rdn1:cn-monitor,
    Rdn4:cn-Replication_server_RS28878_iamfrdev2blueudjrs220059,
    Name:rootDSE,
    Rdn5:cn-Connected_directory_server_DS10659_iamfrdev2blueudjrs246226] – 1544120059 = 0.0
    jmx.org.opends.server.missing_changes[instance:jmx_instance,jmx_domain:org.opends.server,
    Rdn2:cn-Replication,
    Rdn3:cn-oussousersdc<our_domain>dccom,
    Rdn1:cn-monitor,
    Rdn4:cn-Replication_server_RS28878_iamfrdev2blueudjrs220059,
    Name:rootDSE,
    Rdn5:cn-Connected_directory_server_DS18308_iamfrdev2blueudjds143742] – 1544120059 = 0.0
    jmx.org.opends.server.missing_changes[instance:jmx_instance,jmx_domain:org.opends.server,
    Rdn2:cn-Replication,
    Rdn3:cn-oussousersdc<our_domain>dccom,
    Rdn1:cn-monitor,
    Rdn6:cn-Connected_directory_server_DS29576_iamfrdev2blueudjds056348,
    Rdn4:cn-Replication_server_RS28878_iamfrdev2blueudjrs220059,
    Name:rootDSE,
    Rdn5:cn-Connected_replication_server_RS611_iamfrdev2blueudjrs120059] – 1544120059 = 0.0
    jmx.org.opends.server.missing_changes[instance:jmx_instance,jmx_domain:org.opends.server,
    Rdn2:cn-Replication,
    Rdn3:cn-oussousersdc<our_domain>dccom,
    Rdn1:cn-monitor,
    Rdn6:cn-Connected_directory_server_DS23537_iamfrdev2blueudjds235424,
    Rdn4:cn-Replication_server_RS28878_iamfrdev2blueudjrs220059,
    Name:rootDSE,
    Rdn5:cn-Connected_replication_server_RS611_iamfrdev2blueudjrs120059] – 1544120059 = 0.0

    #24196
     Ludo 
    Moderator

    Hi,

    Which version of OpenDJ ?
    This seems to be a bug or an initialisation issue, but we would need more data to understand exactly.

    Note that with DS 6 and higher, we’ve stopped using the missing changes (and age) as a way to measure replication, because missing changes needs to be computed with distributed systems, and iterate through the changelog DB, and this was too costly and inaccurate.

    #24197
     matthewponzio 
    Participant

    Hi Ludo, thanks so much for the reply. Sorry, totally forgot to include the version…it’s 3.5. I’m hopeful that we will be able to be upgrade to 6 sometime fairly soon, so this may be a moot point indeed. I did see the note in the docs that missing_changes was deprecated in v6. Probably not worth pursuing a resolution at this stage then, thanks again.

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.

©2019 ForgeRock - we provide an identity and access platform to secure every online relationship for the enterprise market, educational sector and even entire countries. Click to view our privacy policy and terms of use.

Log in with your credentials

Forgot your details?