OpenDJ 3.0 – Replication Issue

[buddhadeb.das]

Hi Experts,

We have setup two OpenDJs in replication mode and it was working fine initially but it stopped working after performing the below steps.

1. Shutdown the OpenDJ server1.
2. Update an entry in the OpenDJ server2.
3. Start the OpenDJ server1.

Error On starting the server1:
======================================

[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log ‘/app/opendj/changelogDb/2.dom/13293.server’ must be released but it is not referenced.”
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the database /app/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN ‘dc=example,dc=com’, serverId ‘13293’, generationId ‘38425’
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.204 msg=Replication server RS(7593) started listening for new connections on address 0.0.0.0 port 8989
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=NOTICE msgID=null.-1 msg=Rejecting append to log ‘/app/opendj/changelogDb/3.dom/16130.server’ for record: [Record [0000015435b587b03f0200000001:ModifyMsg content: protocolVersion: 8 dn: uid=openam_cts,ou=admins,dc=cts,dc=com csn: 0000015435b587b03f0200000001 uniqueId: 8eccedb1-8630-4841-b00d-8616466eae59 assuredFlag: false assuredMode: SAFE_DATA_MODE safeDataLevel: 1 size: 133]], last key appended: [0000015435b587b03f0200000001]
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log ‘/app/opendj/changelogDb/3.dom/25174.server’ must be released but it is not referenced.”
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.26 msg=Error trying to use the underlying database. The Replication Server is going to shut down: ChangelogException: Could not get or create replica DB for baseDN ‘dc=cts,dc=com’, serverId ‘25174’, generationId ‘76815’ (ReplicationEnvironment.java:387 FileReplicaDB.java:151 FileReplicaDB.java:129 FileChangelogDB.java:278 FileChangelogDB.java:199 FileChangelogDB.java:812 ReplicationServerDomain.java:513 ReplicationServerDomain.java:337 ServerHandler.java:1140 ServerReader.java:113)

Thanks,
Deb

[Chris Ridd]

What’s the contents of your server’s config/buildinfo file? This includes the server version and git revision string.

[buddhadeb.das]

Hi Chris,

Below is the config/buildinfo content:

3.0.0.185acee3ba68d8da1782007eebacb3701dc996d6

Thanks,
Deb

[Chris Ridd]

Thanks, that looks like the 3.0.0 release.

Do you have any steps to follow to reproduce this problem?

Chris

[buddhadeb.das]

we have encountered this problem when we tried to test failover for openDJ servers in cluster.
Here are the high level steps the we have done :-

Setup replication between DJ1 and DJ2,
Test to confirm replication is working fine
shutdown DJ1
make changes on DJ2
Try to bring up DJ1
this is when we started observing these errors in the startup logs of DJ1,When we restarted DJ2 we could see similar errors in its startup logs also.

Thanks,
Deb

[buddhadeb.das]

To add to above we tried to disable replication on both the nodes and reconfigure replication again which completes successfully however I start facing the same issue when i restart the servers.

[Chris Ridd]

Sorry for the delay.

Engineering comment that this sort of procedure is carried out regularly in our functional tests (and never fails!), so what you are seeing is not expected. They suggest looking at things like file permissions or disk space, or something environmental…

[buddhadeb.das]

Hi Chris,

Apologies for the late update the issue was a one of issue where the ChangeLog DB files of OpenDJ got corrupted :-

The Resolution is to
1>Disable Replication on both Nodes
2>Stop both the OpenDJ services
3>Delete the ChangeLog DB Folders from both the OpenDJ Node
4>Restart the Servers
5>Enable and Initialize Replication again.

The Root cause is still unknown.

Thanks,
Buddhadeb

[Chris Ridd]

We may have found the problem – can you do an ls -lR of your changelogDb directory please? It seems to be related to file sizes.

[dwindau]

Hi,

sorry for necromanzing this thread, but we do have an identical issue at the moment. Everytime the head.log in any replication server’s folder reaches a multiple of 256 bytes and the server gets stopped and started again, the server throws two errors:

[27/Jul/2018:08:21:07 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/opendj/changelogDb/2.dom/29648.server' must be released but it is not referenced."
[27/Jul/2018:08:21:07 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the database /opt/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN 'o=XXX', serverId '29648', generationId '20860917'

There are two commits on github targetting this problem (this and this). But neither of it work for us.

Is there someone that has tackled this problem successfully and may share his knowledge about this here?

Greetings,

dw

[dwindau]

For those who also suffer from this bug: Using OpenDJ 4.1.10 helped me

[Author]

Posts