OpenDJ Load testing

[rbhanot4739]

Hello,

I want to do some Load Testing on our OpenDJ server, from the Admin guide i read about OpenDJ LDAP toolkit. I downloaded and installed it however i could not find much information about the usage of the commands. While reading about the authrate command i was not able to figure out what would names.txt file contain, there was no sample given for that.

authrate -p 1389 -g “rand(names.txt)”
-D “uid=%s,ou=people,dc=example,dc=com” -w password -c 10 -f

So can anyone please help me out on this.

Thanks,
Rohit

[Mark Craig]

The reference information for the tools is in the LDAP SDK Developer’s Guide. For example, https://backstage.forgerock.com/#!/docs/opendj/2.6/dev-guide/dev-tools-ref.

[rbhanot4739]

Thanks for the reply, however i have already gone through that and that was the point i was talking about. Can you please help me what should be the contents of the names.txt look like ?

[Mark Craig]

In the explanation for -g, which generates an, “Argument used to evaluate the Java style format strings in program parameters (Base DN, Search Filter),” the reference says:

"rand({filename})"
  Random line from file

So when the argument generated replaces the %s in a -D "uid=%s,ou=people,dc=example,dc=com", you need a names.txt file with one UID per line.

If you generated the directory data for testing when you set up OpenDJ, for example, all the users have UIDs like user.0, user.1, user.2, etc.

So you’d need a names.txt file like:

user.0
user.1
user.2
... etc., etc....

• This reply was modified 6 years, 2 months ago by Mark Craig.

[rbhanot4739]

Hi Mark,

Thanks for your reply !! However i have some more questions for you for which i could not find the answers/information in OpenDJ documents or online references.

1. The OpenDJ SDK Developer’s guide talks about only one tool which is authrate and makes no reference to other three(addrate|modrate|searchrate) mentioned in the Admin Guide. So can you please point to me to any documentation for these as well ?

2. I ran authrate on some sample data in our DJ server and got some output results, but how would i know or make sure if these values which i received as result or good or bad ? By that what i mean to ask is that are they any Benchmarking stats against which we can compare our testing results ?

Below is the sample output for authrate command.

[[email protected] bin]# ./authrate -p 636 -g “rand(names.txt)” -D “uid=%s,ou=Users,ou=Internal,o=example” -w password -c 10 -f
—————————————————————–
Throughput Response Time
(ops/second) (milliseconds)
recent average recent average 99.9% 99.99% 99.999% err/sec
—————————————————————–
Error occurred on one or more connections: Result(resultCode=Server Connection
Closed, matchedDN=, diagnosticMessage=, referrals=[], controls=[])
27.2 27.2 13.301 13.235 187.00 187.00 187.00 27.2

• This reply was modified 6 years, 2 months ago by rbhanot4739.

[Mark Craig]

Regarding 1., you’ll need to scroll down the page to see the reference for the modrate and searchrate commands:

Here are the direct links:

• https://backstage.forgerock.com/#!/docs/opendj/2.6/dev-guide/dev-tools-ref#modrate-1
• https://backstage.forgerock.com/#!/docs/opendj/2.6/dev-guide/dev-tools-ref#searchrate-1

Regarding 2., I notice you have an error message there about the server connection being closed.

Also, if you look at the header column for the output, the column that says recent average ... err/sec, and then look at the output, you’ll notice that your average rate and your err/sec are the same. In other words, all you’re getting are errors.

Generally, to diagnose LDAP client access errors with OpenDJ directory server, you’ll want to start by trying to find the operations in the OpenDJ access log, /path/to/opendj/logs/access.

[rbhanot4739]

Hi,

Can you please let me know why i am getting error on port 636 and not on standard 389 ?

[[email protected] bin]# ./authrate -p 636 -D "cn=Directory Manager" -w xxxx$  -c 10 -f
-----------------------------------------------------------------
     Throughput                            Response Time
   (ops/second)                           (milliseconds)
recent  average  recent  average  99.9%  99.99%  99.999%  err/sec
-----------------------------------------------------------------
Error occurred on one or more connections: Result(resultCode=Server Connection
Closed, matchedDN=, diagnosticMessage=, referrals=[], controls=[])
[[email protected] bin]./authrate -p 389 -D "cn=Directory Manager" -w xxxx$  -c 10 -f
-----------------------------------------------------------------
     Throughput                            Response Time
   (ops/second)                           (milliseconds)
recent  average  recent  average  99.9%  99.99%  99.999%  err/sec
-----------------------------------------------------------------
35609.6  35609.6   0.295    0.295  10.00   20.00    27.00      0.0
37634.8  36622.4   0.276    0.285  10.00   19.00    26.00      0.0
38200.6  37149.6   0.270    0.280   9.60   19.00    27.00      0.0
37790.6  37310.1   0.275    0.278   9.60   19.00    27.00      0.0

[Ian Packer]

You need to add –useSSL if you’re using an SSL port (and potentially –trustAll)

[rbhanot4739]

Great Thank you !! One more query which i asked earlier as well in my posts were
1. How do we ensure if the results optimal or not ?
2. Are they any Benchmarks with which we can compare our values to make sure we are good ?
3. What does the output columns actually mean ? Any reference which explains these output?

[Ludo]

1. How do we ensure if the results optimal or not ?
Assume they are not, try to change the load and see if you have better results.
When running searchrate, the server should maximize CPU.
When running modrate, the server should maximize disk IO.
Note that if you are running the tools on the same machine as the server, you will not get optimal results, seen the tool is consuming CPU and memory resources. If they are on separate machines, you should monitor the network and make sure it is not a bottleneck.

2. Are they any Benchmarks with which we can compare our values to make sure we are good ?
No, there are no public benchmarks, as there are no standard for LDAP Benchmarks from SPEC.org.
This said, depending on HW OpenDJ can do up to 100 000 searches per seconds. On my laptop I can do 45 000 with the client on the same machine. In term of Modifications, with SSD, the server can reach 25 000 modifications / seconds (Unindexed modification). A good value is around 12 500 mod/sec.

3. What does the output columns actually mean ? Any reference which explains these output?
The columns have a header which is explicit (1st 2 columns are throughput, the next 5 ones are response times, last one is the number of errors).

[rbhanot4739]

Thanks for the response Ludo. Can you help me analyzing below output on a system with 20G of RAM and 8 CPU cores.

./searchrate -p 636 --usessl --trustAll -h testbox -b "o=example" -A -F -c 4 -t 4  -g "rand(names.txt)" "(uid=%s)"
     Throughput                            Response Time
   (ops/second)                           (milliseconds)
recent  average  recent  average  99.9%  99.99%  99.999%  err/sec  req/res  Entries/Srch
----------------------------------------------------------------------------------------
  16.6     16.6  2728.193  2727.723  4600.00  4700.00  4700.00      0.0     66.6           1.2
  18.6     17.6  7640.731  5323.801  10000.00  10000.00  10000.00      0.0      2.0           1.0
  17.6     17.6  12625.909  7757.837  14500.00  15000.00  15000.00      0.0      0.0           1.1
  18.8     17.9  17590.713  10339.654  19500.00  20000.00  20000.00      0.0      0.0           1.0

16.6 ops/second seems very less as compared to 100000 or 45000 searches/second.Also i would like to know what does 99.9% and 99.99.% columns mean ?

Also do you mean that server should be running high on CPU when running searchrate as you mentioned above ?

• This reply was modified 6 years, 2 months ago by rbhanot4739.

[Ludo]

Response times are in milliseconds… A recent response time of 17 seconds seems to indicate that your server is either under heavy stress or completely lack resources.
Remember that OpenDJ is built on Java, and it doesn’t matter how much memory that system has. What matters is how much memory was allocated to the Java process. Please read the Tuning Section of our Administration Guide for guidance:
https://backstage.forgerock.com/#!/docs/opendj/3/admin-guide#chap-tuning

[rbhanot4739]

Hi Ludo,

I am little confused here in understanding and interpreting the output which i posted in my previous posts. As much as i can figure out, does a throughput(ops/second) of 16.6 means our OpenDj server instance is completing/processing 16.6 searches per second ? If i suppose that to be true then that is way too bad when compared with your said values of 100,000 or 45000 searches per seconds !!!

We have allocated 2.5G of memory to JVM for ldap search operation and the names.txt file has almost 400,000 entries.

Also we like to know what does Average column mean ?

[Ludo]

Hi Rohit,

Yes, your tests are showing that the server is returning 16.6 entries per second, which is low, but probably an indication that the search is not indexed.
For each line, the output contains the recent result and the average result since the beginning of the load.
Recent and Average number of operation per seconds, and then the recent and average response time in milliseconds.
So in average, your server starts with responding in 2 seconds then over time takes more than 17 seconds to respond to each request.
There is definitely something wrong in your settings. It could either be the client (searchrate), the OpenDJ server or the network. I suggest that you spend a little bit more time to analyse each of them, look at GC, netstat and CPU, and try to understand which part is not working correctly.

[tangudukishore]

Hi Ludo,
Hope you are doing good. I have read your responses in multiple articles, which are very useful.
I have a doubt regarding openDJ addrate(openDJ3.0 ldap toolkit).

Request::

./addrate -p 1389 -D “cn=Directory Manager” -w password -f -c 4 -s 50 users.template

Response::

———————————————————————–
Throughput Response Time
(ops/second) (milliseconds)
recent average recent average 99.9% 99.99% 99.999% err/sec Add%
———————————————————————–
Exception in thread “Worker Thread” Exception in thread “Worker Thread” Exception in thread “Worker Thread” java.util.NoSuchElementException
at org.forgerock.opendj.ldif.EntryGenerator.readEntry(EntryGenerator.java:285)
at com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner$AddDeleteWorkerThread.doAdd(AddRate.java:236)
at com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner$AddDeleteWorkerThread.performOperation(AddRate.java:177)
at com.forgerock.opendj.ldap.tools.PerformanceRunner$WorkerThread.run(PerformanceRunner.java:603)
Exception in thread “Worker Thread” java.util.NoSuchElementException
at org.forgerock.opendj.ldif.EntryGenerator.readEntry(EntryGenerator.java:285)
at com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner$AddDeleteWorkerThread.doAdd(AddRate.java:236)
at com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner$AddDeleteWorkerThread.performOperation(AddRate.java:177)
at com.forgerock.opendj.ldap.tools.PerformanceRunner$WorkerThread.run(PerformanceRunner.java:603)
java.util.NoSuchElementException
at org.forgerock.opendj.ldif.EntryGenerator.readEntry(EntryGenerator.java:285)
at com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner$AddDeleteWorkerThread.doAdd(AddRate.java:236)
at com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner$AddDeleteWorkerThread.performOperation(AddRate.java:177)
at com.forgerock.opendj.ldap.tools.PerformanceRunner$WorkerThread.run(PerformanceRunner.java:603)
java.util.NoSuchElementException
at org.forgerock.opendj.ldif.EntryGenerator.readEntry(EntryGenerator.java:285)
at com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner$AddDeleteWorkerThread.doAdd(AddRate.java:236)
at com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner$AddDeleteWorkerThread.performOperation(AddRate.java:177)
at com.forgerock.opendj.ldap.tools.PerformanceRunner$WorkerThread.run(PerformanceRunner.java:603)
10.0 10.0 4.140 4.140 15.00 15.00 15.00 10.0 0.00
Purge phase…

Description:: The fortunate thing is user is adding to the ldap. And I am unable to see the performance metrics for addrate. It is printing only one line in the last.
Can you please suggest me what to do?

[Author]

Posts