Developing Tibco BW interfaces with Elasticsearch for trace logs – PART 2

Part 1 of this post was generally about elasticsearch. 

That part will explain how to send documents from TIBCO BW 5.13 to Elasticsearch.

Send docs from TIBCO to Elastic

There are two ways to send docs from BW to elastic:

1. Elastic REST API.

2. Elastic Java Client API (Downloaded from here: https://www.elastic.co/guide/en/elasticsearch/client/java-api/index.html).

This post will discuss the first option.

In order to create JSON docs + invoke REST easily - please install JSON & REST Plugin in tibco folder.

Create mapping and appropriate xsd 

  

Let's get back to elastic for a moment.

Open Sense and create a mapping for trace documents that will be sent from Tibco.
Please notice that there aren't any analyzed fields since each field contains single word\string:

POST monitors

{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "monitor": {
      "properties": {
        "ProcessGroup": {
          "type": "string",
          "index": "not_analyzed"
        },
        "ProcessName": {
          "type": "string",
          "index": "not_analyzed"
        },
        "OpName": {
          "type": "string",
          "index": "not_analyzed"
        },
        "Domain": {
          "type": "string",
          "index": "not_analyzed"
        },
        "TraceType": {
          "type": "string",
          "index": "not_analyzed"
        },
        "TraceDateTime": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss"
        },
        "PatientID": {
          "type": "string",
          "index": "analyzed"
        },
        "MessageDateTime": {
          "type": "string"
        },
        "ApplicationCode": {
          "type": "string",
          "index": "not_analyzed"
        },
        "SrcMessageID": {
          "type": "string",
          "index": "not_analyzed"
        },
        "ProcessID": {
          "type": "string",
          "index": "not_analyzed"
        },
        "OpID": {
          "type": "string",
          "index": "not_analyzed"
        },
        "OpParentID": {
          "type": "string",
          "index": "not_analyzed"
        },
        "HostName": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    }
  }
}

Go back to Tibco BW and create a schema which match the mapping: 

<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema"
     xmlns:tns="http://Integration.clalit.org.il/Schemas/TibSrv_Log"
     targetNamespace="http://Integration.clalit.org.il/Schemas/TibSrv_Log"
     elementFormDefault="qualified"
     attributeFormDefault="unqualified">
    <element name="WriteTrace_Request" type="tns:WriteTraceType"/>
    <complexType name="MonitorType">
        <sequence>
            <element name="ProcessGroup" type="string"/>
            <element name="ProcessName" type="string"/>
            <element name="OpName" type="string"/>
            <element name="Domain" type="string"/>
            <element name="TraceType">
                <simpleType>
                    <restriction base="string">
                        <enumeration value="Debug"/>
                        <enumeration value="Info"/>
                        <enumeration value="Warning"/>
                        <enumeration value="Error"/>
                        <enumeration value="Critical"/>
                    </restriction>
                </simpleType>
            </element>
            <element name="TraceDateTime" type="tns:string"/>
            <element name="PatientID" type="string" minOccurs="0"/>
            <element name="MessageDateTime" type="tns:string" minOccurs="0"/>
            <element name="ApplicationCode" type="string" minOccurs="0"/>
            <element name="SrcMessageID" type="string" minOccurs="0"/>
            <element name="ProcessID" type="string" minOccurs="0"/>
            <element name="OpID" type="string" minOccurs="0"/>
            <element name="OpParentID" type="string" minOccurs="0"/>
            <element name="HostName" type="string"/>
            <any namespace="##any" processContents="skip" minOccurs="0" maxOccurs="unbounded"/>
        </sequence>
    </complexType>
    <complexType name="MessageKeyDataType">
        <sequence>
            <element name="KeyValueData" maxOccurs="unbounded">
                <complexType>
                    <sequence>
                        <element name="DataFieldType" type="string"/>
                        <element name="DataFieldValue" type="string"/>
                    </sequence>
                </complexType>
            </element>
        </sequence>
    </complexType>
    <complexType name="ExceptionType">
        <sequence>
            <element name="ProcessID" type="string"/>
            <element name="Class" type="string"/>
            <element name="ProcessStack" type="string"/>
            <element name="MsgCode" type="string"/>
            <element name="Msg" type="string"/>
            <element name="StackTrace" type="string" minOccurs="0"/>
            <element name="Data" minOccurs="0">
                <complexType>
                    <sequence>
                        <any namespace="##any" processContents="skip" minOccurs="0"/>
                    </sequence>
                </complexType>
            </element>
        </sequence>
    </complexType>
    <complexType name="WriteTraceType">
        <sequence>
            <element name="Monitor" type="tns:MonitorType" nillable="true"/>
            <element name="MessageKeyData" type="tns:MessageKeyDataType" nillable="true" minOccurs="0"/>
            <element name="Exception" type="tns:ExceptionType" nillable="true" minOccurs="0"/>
        </sequence>
    </complexType>
    <element name="Monitor" type="tns:MonitorType"/>
    <element name="MessageKeyData" type="tns:MessageKeyDataType"/>
    <element name="Exception" type="tns:ExceptionType"/>
    <simpleType name="string">
        <restriction base="string"/>
    </simpleType>
    <simpleType name="anySimpleType">
        <restriction base="string"/>
    </simpleType>
</schema>

Take notice on:

After HostName element, any element can be placed.
Elasticsearch allows to insert fields which are not included in mapping.

For instance:

  <HostName>Machine1</HostName>

  <NewElem>Hi</NewElem>

  <Another>Again</Another>

Develop BW process 

 Take a look at the whole process:

 

First step is to map to the xsd created above.

It's important to make sure each datetime field is converted to UTC time.

Here is the mapping I've used:

Second, render from xml to JSON is easy using the plugin.

In addition, I found "Remove root" option very useful.

Here is an example at runtime after rendering to JSON:

  

Last step is to invoke REST API using POST method.

Convert JSON string (render output) to base64 and place it in "binary" node "content" element.

That's it! 

All that's left to do is to call this process asynchronously (you surely do NOT want traces to disturb your processes) and bomb elastic search with docs!
Don't forget to use kibana to create nice dashboards for maintenance.