Opendj performance tuning for write/modify operations

This topic has 3 replies, 4 voices, and was last updated 6 years, 5 months ago by Ludo.

  • Author
    Posts
  • #10026
     jasonwhitener
    Participant

    edit: long story short, I discovered that opendj is able to create new people entries very quickly <50ms. The vendor code that was running was doing a bunch of extra stuff, which added up to 1-2 seconds per record. I am still interested in any tuning tips you all might have. Total number of records will be 1-2million. Write speeds are as important to me as read speeds.
    ———————–

    Original Post:

    I am running a proprietary vendor process that is creating new entries sequentially in opendj 2.5. That process is running on server A, and my opendj installation is on server B. The logs in server A show that it is taking 1-1.5 seconds to create a new ldap record. That process on server A also writes to other destinations, like an oracle database. Those other write operations take less than 50ms.

    Both servers are redhat 6.7 running on vmware 5.1. Each has 16GB ram and 4 fast cpu cores.

    I can’t see the code (requested it, waiting to get it), but I assume it is doing something simple like a series of ldapadd or ldapmodify statements. I don’t see those in the logs though. In the access log I just see searches.

    I’ve just started looking into tuning opendj. In the java.properties file I configured the heap to use 64bit, and 4GB ram, with a Xmn of 1GB. Re-ran my import, no improvement in speed. Tried this: https://ludopoitou.com/2011/12/16/an-important-tuning-flag-for-opendj-with-64bit-jvm/ No improvement.

    Are there other tuning changes I can make to increase write/modify speeds? And is there a way to increase the access log verbosity so I can see exactly what ldap statements server A is sending to my opendj on server B?

    I’m not yet sure if the problem is with our vmware environment, opendj, or the vendor process. According to other people I’ve talked with who use this vendor software, they are getting 2-3 records per second with default installations, so I’m not sure what is different about our setup.

    • This topic was modified 6 years, 5 months ago by jasonwhitener.
    #10034
     JnRouvignac
    Participant

    Hi Jason,

    Adding many entries (> 100.000) one by one is not the preferred way to import many entries.
    Adding them one by one to an existing database is creating a lot of garbage (inner nodes of the B-Tree) at the database level.
    If this is an initial import we strongly recommend to use import-ldif.

    Jean-Noel

    #10035
     Chris Ridd
    Participant

    I should also note that OpenDJ 2.5.0-Xpress1 is very old and has unfixed security vulnerabilities. You should be running at least OpenDJ 2.6.4 (only available to FR customers) or OpenDJ 3.0.0 (available to community and customers) to fix those vulnerabilities.

    It seems strange that you don’t see “ADD REQ” or “MODIFY REQ” operations in the access log.

    OpenDJ has an “audit” log which records exactly what changes are successfully written to the database. This is not enabled by default.

    Taking jstacks is a good way to observe what the server is doing during these 1-1.5s operations. If I were to guess I would say that it is spending time updating some expensive indexes. We provide tools to analyse index contents.

    #10059
     Ludo
    Moderator

    Hi Jason,
    Editing the first question is nice, but it doesn’t help with concluding the thread :)
    Glad to read that you found that OpenDJ was creating the entries quickly, and the initial issue was in the client tool.
    With regards to keep improving tuning and performances, we do have a little bit of documentation in the Administration Guide. But ADD processing performances is tightly coupled with the indexes and the types of indexes that apply to the entries (and contention when doing multiple ADD requests in parallel), as well as the performances of the underlying disks.
    I hope this helps.

Viewing 4 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.

©2022 ForgeRock - we provide an identity and access platform to secure every online relationship for the enterprise market, educational sector and even entire countries. Click to view our privacy policy and terms of use.

Log in with your credentials

Forgot your details?