OpenIDM 4.5 includes support for integration with ElasticSearch. For those not familiar, ElasticSearch is part of what is commonly referred to as the ELK stack:
E – ElasticSearch: Processes incoming data and allows for it to be quickly and RESTfully analysed and searched.
L – LogStash: A logging processor that takes input data, puts it through a pipeline of operations and outputs it somewhere ( usually into ElasticSearch ).
K – Kibana: Visualisation engine that utilises the data provided by ElasticSearch to create visual charts of all different types that can be embedded into web pages.
Together they enable you to create slick visualisations of data.
In the latest 4.5 release of OpenIDM two new features have been introduced:
- An ElasticSearch audit handler, that can be used to push audit events into ElasticSearch.
- Configurable dashboard widgets in the admin interface. There are a number of widgets available OOTB, one of which enables the embedding of Kibana visualisations.
Installing ElasticSearch & Kibana
The first step is to download and install ElasticSearch from https://www.elastic.co/downloads/elasticsearch
Simply unzip ElasticSearch somewhere and start it up:
mv elasticsearch-2.4.1 elasticsearch
You can check that it is running by navigating to http://localhost.localdomain.com:9200, and you should see something like this:
Next, download and install Kibana from https://www.elastic.co/products/kibana
Again, just unzip it somewhere:
tar -xvf kibana-4.6.1-darwin-x86_64.tar.gz
But before starting it up check the following file:
And ensure it is pointing at your elasticsearch instance:
If you can see this kibana splash page, then everyone is on track.
Configuring OpenIDM 4.5 for ElasticSearch Audit
Note: In earlier builds there was a bug where to enable the handler you had to manually edit conf/audit.json. If you follow all the steps and something is not working, double check that the handler is set to “enabled” : true
Scroll down and configure the handler parameters as below:
host: your host
port: 9200 ( unless you changed it)
Press Submit then on the next page you must remember to scroll down and press Save
Now, put your elasticsearch process window somewhere you can see it, then log out and back in of OpenIDM and take a look. You should see some activity:
Processing Audit Data in Kibana
Now we need to configure Kibana to process and do things with all that juicy audit data from OpenIDM. Navigate to Kibana again and replace “logstash-*” with “audit, then hit Create:
We now have an Audit indices, and you should see the OpenIDM audit attributes have already been catalogued:
And if you click Discover you should be able to browse the data!
Creating a Visualisation
Adding the Visualisation to OpenIDM
Add a widget to your new dashboard: