Extracting Valuable Data from log4j logs with Virtual Fields

In our recent upgrade to XpoLog V6 we enhanced the features of log4j analysis. In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs. We have also prepared a hands-on-guide-in-one if you prefer to read all the posts in one go.

Once your log4j logs have been transferred to and properly defined in XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own AppTags for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum log analysis.

This post will show you how to define log4j logs in XpoLog, to create the most readable data and thus allow for XpoLog to perform highly detailed analysis of your logs. I will also show you an example of how you can virtually extract specific data from your message using Regular Expression to allow for XpoLog to perform a more refined parsing of your data.

If you want to follow my steps as you read along, you can download XpoLog V6 for free.

Defining Patterns in XpoLog Center

If you are letting XpoLog access and pull data from your files, define the logger with a name, pattern and data pattern, and then define the log patterns in XpoLog Center.

For example:

#Logger definition



#Appender data for mylog


log4j.appender.mylog.layout.ConversionPattern=[%d] [%t] [%p] [%c] [%l] %m%n

(d = date, t = thread, p = priority, c = class, l = method, m = message, and n = new line)

Defining the log pattern in XpoLog Center:

  1. In XpoLog Center, add a new log. (See the instructions in my previous post.) Once you have filled in the details, click Next to get to the Log Pattern screen.
  2. In the Wizard of the Pattern Editor, define the log pattern.image 1

Click Manual in the Pattern Editor and edit the XpoLog data pattern to comply with the log4j layout:

    1. [%d] = [{date:Date,locale=en,yyyy-MM-dd HH:mm:ss,SSS}]
    2. [%t] = [{text:Thread}]
    3. [%p]= [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}]
    4. [%c]= [{string:Class}]
    5. [%l]= [{string:Method}({text:Source}:{number:LineNumber})]
    6. %m = {string:Message}
    7. %n =  new lineimage 2

The XpoLog pattern in our example will be:

[{date:Date,locale=en,yyyy-MM-dd HH:mm:ss,SSS}] [{text:Thread}] [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}] [{string:Class}] [{string:Method}({text:Source}:{number:LineNumber})] {string:Message}

  3. Click Save.

You can also edit the pattern after you have added the log, of which I will speak more of in my next post.

Virtually Extract Specific Data from your Message

XpoLog can also extract data from within the message if you use Regular Expression prior to the data transfer.

This is what the message might look like in the Log Pattern section of the Add Log screen without using Regular Expression:

In the Pattern Editor, all you see is {string:Message}.

image 3

If you use Regular Expression to extract any word that appears after the word “Manager”, the Log Pattern section of the Add Log screen would look as follows:

In the Pattern Editor, you will now see:

{regexp:HTMLManager state,refName=Message,HTMLManager: (w+)}{string:Message}

In the Log records analysis result section below XpoLog has added the column HTMLManager state for the data you wished to extract.

image 4

In the Manager Interface of XpoLog Center, where you view your logs, you will also see this extra column, HTMLManager state, for the extracted data:

image 5

By extracting the HTMLManager state into a new virtual field we can now measure and monitor the HTMLManager state performance and activity.

In the next post, I will show how to define and edit the log4j patterns when sending log events and log messages to XpoLog through SysLog. Stay tuned, or check out our “spoiler” for the full tutorial.

Developer Games: RegExp and log4j Parsing

About The Author

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>