Tagged: #openDJ, replication
This topic has 10 replies, 3 voices, and was last updated 3 years, 11 months ago by dwindau.
-
AuthorPosts
-
April 21, 2016 at 4:48 pm #9993
buddhadeb.das
ParticipantHi Experts,
We have setup two OpenDJs in replication mode and it was working fine initially but it stopped working after performing the below steps.
1. Shutdown the OpenDJ server1.
2. Update an entry in the OpenDJ server2.
3. Start the OpenDJ server1.Error On starting the server1:
======================================[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log ‘/app/opendj/changelogDb/2.dom/13293.server’ must be released but it is not referenced.”
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the database /app/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN ‘dc=example,dc=com’, serverId ‘13293’, generationId ‘38425’
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.204 msg=Replication server RS(7593) started listening for new connections on address 0.0.0.0 port 8989
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=NOTICE msgID=null.-1 msg=Rejecting append to log ‘/app/opendj/changelogDb/3.dom/16130.server’ for record: [Record [0000015435b587b03f0200000001:ModifyMsg content: protocolVersion: 8 dn: uid=openam_cts,ou=admins,dc=cts,dc=com csn: 0000015435b587b03f0200000001 uniqueId: 8eccedb1-8630-4841-b00d-8616466eae59 assuredFlag: false assuredMode: SAFE_DATA_MODE safeDataLevel: 1 size: 133]], last key appended: [0000015435b587b03f0200000001]
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log ‘/app/opendj/changelogDb/3.dom/25174.server’ must be released but it is not referenced.”
[21/Apr/2016:07:04:24 -0400] category=SYNC severity=ERROR msgID=org.opends.messages.replication.26 msg=Error trying to use the underlying database. The Replication Server is going to shut down: ChangelogException: Could not get or create replica DB for baseDN ‘dc=cts,dc=com’, serverId ‘25174’, generationId ‘76815’ (ReplicationEnvironment.java:387 FileReplicaDB.java:151 FileReplicaDB.java:129 FileChangelogDB.java:278 FileChangelogDB.java:199 FileChangelogDB.java:812 ReplicationServerDomain.java:513 ReplicationServerDomain.java:337 ServerHandler.java:1140 ServerReader.java:113)Thanks,
DebApril 21, 2016 at 5:27 pm #9996Chris Ridd
ParticipantWhat’s the contents of your server’s config/buildinfo file? This includes the server version and git revision string.
April 22, 2016 at 8:45 am #10004buddhadeb.das
ParticipantHi Chris,
Below is the config/buildinfo content:
3.0.0.185acee3ba68d8da1782007eebacb3701dc996d6
Thanks,
DebApril 22, 2016 at 5:24 pm #10007Chris Ridd
ParticipantThanks, that looks like the 3.0.0 release.
Do you have any steps to follow to reproduce this problem?
Chris
April 22, 2016 at 5:50 pm #10008buddhadeb.das
Participantwe have encountered this problem when we tried to test failover for openDJ servers in cluster.
Here are the high level steps the we have done :-Setup replication between DJ1 and DJ2,
Test to confirm replication is working fine
shutdown DJ1
make changes on DJ2
Try to bring up DJ1
this is when we started observing these errors in the startup logs of DJ1,When we restarted DJ2 we could see similar errors in its startup logs also.Thanks,
DebApril 22, 2016 at 5:54 pm #10009buddhadeb.das
ParticipantTo add to above we tried to disable replication on both the nodes and reconfigure replication again which completes successfully however I start facing the same issue when i restart the servers.
April 26, 2016 at 10:39 am #10038Chris Ridd
ParticipantSorry for the delay.
Engineering comment that this sort of procedure is carried out regularly in our functional tests (and never fails!), so what you are seeing is not expected. They suggest looking at things like file permissions or disk space, or something environmental…
May 7, 2016 at 8:56 am #10329buddhadeb.das
ParticipantHi Chris,
Apologies for the late update the issue was a one of issue where the ChangeLog DB files of OpenDJ got corrupted :-
The Resolution is to
1>Disable Replication on both Nodes
2>Stop both the OpenDJ services
3>Delete the ChangeLog DB Folders from both the OpenDJ Node
4>Restart the Servers
5>Enable and Initialize Replication again.The Root cause is still unknown.
Thanks,
BuddhadebMay 11, 2016 at 12:20 pm #10468Chris Ridd
ParticipantWe may have found the problem – can you do an
ls -lR
of your changelogDb directory please? It seems to be related to file sizes.July 27, 2018 at 1:57 pm #22609dwindau
ParticipantHi,
sorry for necromanzing this thread, but we do have an identical issue at the moment. Everytime the head.log in any replication server’s folder reaches a multiple of 256 bytes and the server gets stopped and started again, the server throws two errors:
[27/Jul/2018:08:21:07 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/opendj/changelogDb/2.dom/29648.server' must be released but it is not referenced." [27/Jul/2018:08:21:07 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the database /opt/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN 'o=XXX', serverId '29648', generationId '20860917'
There are two commits on github targetting this problem (this and this). But neither of it work for us.
Is there someone that has tackled this problem successfully and may share his knowledge about this here?
Greetings,
dw
August 1, 2018 at 9:57 am #22651dwindau
ParticipantFor those who also suffer from this bug: Using OpenDJ 4.1.10 helped me
-
AuthorPosts
You must be logged in to reply to this topic.