April 21, 2016 at 4:48 pm #9993
We have setup two OpenDJs in replication mode and it was working fine initially but it stopped working after performing the below steps.
1. Shutdown the OpenDJ server1.
2. Update an entry in the OpenDJ server2.
3. Start the OpenDJ server1.
Error On starting the server1:
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log ‘/app/opendj/changelogDb/2.dom/13293.server’ must be released but it is not referenced.”
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the database /app/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN ‘dc=example,dc=com’, serverId ‘13293’, generationId ‘38425’
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.204 msg=Replication server RS(7593) started listening for new connections on address 0.0.0.0 port 8989
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=NOTICE msgID=null.-1 msg=Rejecting append to log ‘/app/opendj/changelogDb/3.dom/16130.server’ for record: [Record [0000015435b587b03f0200000001:ModifyMsg content: protocolVersion: 8 dn: uid=openam_cts,ou=admins,dc=cts,dc=com csn: 0000015435b587b03f0200000001 uniqueId: 8eccedb1-8630-4841-b00d-8616466eae59 assuredFlag: false assuredMode: SAFE_DATA_MODE safeDataLevel: 1 size: 133]], last key appended: [0000015435b587b03f0200000001]
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log ‘/app/opendj/changelogDb/3.dom/25174.server’ must be released but it is not referenced.”
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.26 msg=Error trying to use the underlying database. The Replication Server is going to shut down: ChangelogException: Could not get or create replica DB for baseDN ‘dc=cts,dc=com’, serverId ‘25174’, generationId ‘76815’ (ReplicationEnvironment.java:387 FileReplicaDB.java:151 FileReplicaDB.java:129 FileChangelogDB.java:278 FileChangelogDB.java:199 FileChangelogDB.java:812 ReplicationServerDomain.java:513 ReplicationServerDomain.java:337 ServerHandler.java:1140 ServerReader.java:113)
DebApril 21, 2016 at 5:27 pm #9996
What’s the contents of your server’s config/buildinfo file? This includes the server version and git revision string.April 22, 2016 at 8:45 am #10004
Below is the config/buildinfo content:
DebApril 22, 2016 at 5:24 pm #10007
Thanks, that looks like the 3.0.0 release.
Do you have any steps to follow to reproduce this problem?
ChrisApril 22, 2016 at 5:50 pm #10008
we have encountered this problem when we tried to test failover for openDJ servers in cluster.
Here are the high level steps the we have done :-
Setup replication between DJ1 and DJ2,
Test to confirm replication is working fine
make changes on DJ2
Try to bring up DJ1
this is when we started observing these errors in the startup logs of DJ1,When we restarted DJ2 we could see similar errors in its startup logs also.
DebApril 22, 2016 at 5:54 pm #10009
To add to above we tried to disable replication on both the nodes and reconfigure replication again which completes successfully however I start facing the same issue when i restart the servers.April 26, 2016 at 10:39 am #10038
Sorry for the delay.
Engineering comment that this sort of procedure is carried out regularly in our functional tests (and never fails!), so what you are seeing is not expected. They suggest looking at things like file permissions or disk space, or something environmental…May 7, 2016 at 8:56 am #10329
Apologies for the late update the issue was a one of issue where the ChangeLog DB files of OpenDJ got corrupted :-
The Resolution is to
1>Disable Replication on both Nodes
2>Stop both the OpenDJ services
3>Delete the ChangeLog DB Folders from both the OpenDJ Node
4>Restart the Servers
5>Enable and Initialize Replication again.
The Root cause is still unknown.
BuddhadebMay 11, 2016 at 12:20 pm #10468
We may have found the problem – can you do an
ls -lRof your changelogDb directory please? It seems to be related to file sizes.July 27, 2018 at 1:57 pm #22609dwindauParticipant
sorry for necromanzing this thread, but we do have an identical issue at the moment. Everytime the head.log in any replication server’s folder reaches a multiple of 256 bytes and the server gets stopped and started again, the server throws two errors:
[27/Jul/2018:08:21:07 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/opendj/changelogDb/2.dom/29648.server' must be released but it is not referenced." [27/Jul/2018:08:21:07 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the database /opt/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN 'o=XXX', serverId '29648', generationId '20860917'
Is there someone that has tackled this problem successfully and may share his knowledge about this here?
dwAugust 1, 2018 at 9:57 am #22651
You must be logged in to reply to this topic.