OpenDJ 3.0 – Replication Issue

This topic has 10 replies, 3 voices, and was last updated 2 years, 5 months ago by dwindau.

  • Author
    Posts
  • #9993
     buddhadeb.das
    Participant

    Hi Experts,

    We have setup two OpenDJs in replication mode and it was working fine initially but it stopped working after performing the below steps.

    1. Shutdown the OpenDJ server1.
    2. Update an entry in the OpenDJ server2.
    3. Start the OpenDJ server1.

    Error On starting the server1:
    ======================================

    [21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log ‘/app/opendj/changelogDb/2.dom/13293.server’ must be released but it is not referenced.”
    [21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the database /app/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN ‘dc=example,dc=com’, serverId ‘13293’, generationId ‘38425’
    [21/Apr/2016:07:04:24 -0400] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.204 msg=Replication server RS(7593) started listening for new connections on address 0.0.0.0 port 8989
    [21/Apr/2016:07:04:24 -0400] category=SYNC severity=NOTICE msgID=null.-1 msg=Rejecting append to log ‘/app/opendj/changelogDb/3.dom/16130.server’ for record: [Record [0000015435b587b03f0200000001:ModifyMsg content: protocolVersion: 8 dn: uid=openam_cts,ou=admins,dc=cts,dc=com csn: 0000015435b587b03f0200000001 uniqueId: 8eccedb1-8630-4841-b00d-8616466eae59 assuredFlag: false assuredMode: SAFE_DATA_MODE safeDataLevel: 1 size: 133]], last key appended: [0000015435b587b03f0200000001]
    [21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log ‘/app/opendj/changelogDb/3.dom/25174.server’ must be released but it is not referenced.”
    [21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.26 msg=Error trying to use the underlying database. The Replication Server is going to shut down: ChangelogException: Could not get or create replica DB for baseDN ‘dc=cts,dc=com’, serverId ‘25174’, generationId ‘76815’ (ReplicationEnvironment.java:387 FileReplicaDB.java:151 FileReplicaDB.java:129 FileChangelogDB.java:278 FileChangelogDB.java:199 FileChangelogDB.java:812 ReplicationServerDomain.java:513 ReplicationServerDomain.java:337 ServerHandler.java:1140 ServerReader.java:113)

    Thanks,
    Deb

    #9996
     Chris Ridd
    Participant

    What’s the contents of your server’s config/buildinfo file? This includes the server version and git revision string.

    #10004
     buddhadeb.das
    Participant

    Hi Chris,

    Below is the config/buildinfo content:

    3.0.0.185acee3ba68d8da1782007eebacb3701dc996d6

    Thanks,
    Deb

    #10007
     Chris Ridd
    Participant

    Thanks, that looks like the 3.0.0 release.

    Do you have any steps to follow to reproduce this problem?

    Chris

    #10008
     buddhadeb.das
    Participant

    we have encountered this problem when we tried to test failover for openDJ servers in cluster.
    Here are the high level steps the we have done :-

    Setup replication between DJ1 and DJ2,
    Test to confirm replication is working fine
    shutdown DJ1
    make changes on DJ2
    Try to bring up DJ1
    this is when we started observing these errors in the startup logs of DJ1,When we restarted DJ2 we could see similar errors in its startup logs also.

    Thanks,
    Deb

    #10009
     buddhadeb.das
    Participant

    To add to above we tried to disable replication on both the nodes and reconfigure replication again which completes successfully however I start facing the same issue when i restart the servers.

    #10038
     Chris Ridd
    Participant

    Sorry for the delay.

    Engineering comment that this sort of procedure is carried out regularly in our functional tests (and never fails!), so what you are seeing is not expected. They suggest looking at things like file permissions or disk space, or something environmental…

    #10329
     buddhadeb.das
    Participant

    Hi Chris,

    Apologies for the late update the issue was a one of issue where the ChangeLog DB files of OpenDJ got corrupted :-

    The Resolution is to
    1>Disable Replication on both Nodes
    2>Stop both the OpenDJ services
    3>Delete the ChangeLog DB Folders from both the OpenDJ Node
    4>Restart the Servers
    5>Enable and Initialize Replication again.

    The Root cause is still unknown.

    Thanks,
    Buddhadeb

    #10468
     Chris Ridd
    Participant

    We may have found the problem – can you do an ls -lR of your changelogDb directory please? It seems to be related to file sizes.

    #22609
     dwindau
    Participant

    Hi,

    sorry for necromanzing this thread, but we do have an identical issue at the moment. Everytime the head.log in any replication server’s folder reaches a multiple of 256 bytes and the server gets stopped and started again, the server throws two errors:

    [27/Jul/2018:08:21:07 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/opendj/changelogDb/2.dom/29648.server' must be released but it is not referenced."
    [27/Jul/2018:08:21:07 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the database /opt/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN 'o=XXX', serverId '29648', generationId '20860917'

    There are two commits on github targetting this problem (this and this). But neither of it work for us.

    Is there someone that has tackled this problem successfully and may share his knowledge about this here?

    Greetings,

    dw

    #22651
     dwindau
    Participant

    For those who also suffer from this bug: Using OpenDJ 4.1.10 helped me

Viewing 11 posts - 1 through 11 (of 11 total)

You must be logged in to reply to this topic.

©2021 ForgeRock - we provide an identity and access platform to secure every online relationship for the enterprise market, educational sector and even entire countries. Click to view our privacy policy and terms of use.

Log in with your credentials

Forgot your details?