Parsing request and response time in a log file

Question

I'm trying to parse a log file. I need help in checking the time difference between a request and response. Essentially number of requests that took more than 10 seconds. Following is a sample excerpt from the file:

16/08/29-03:20:39.538 << 1250 REQUEST COUNTER="1"
16/08/29-03:20:39.542 << 1250 REQUEST COUNTER="2"
16/08/29-03:20:39.656 >> 2250 RESPONSE COUNTER="1"
16/08/29-03:20:39.655 >> 2250 RESPONSE COUNTER="2"

This file can have around 3 million records. I'm thinking of reading it using a BufferedReader, reading a line then checking to see if "COUNTER" is present in the line, storing request time and the value of counter, then searching the entire file for the same counter value to get the response time and then checking the difference.

I need help in storing the time and checking the difference. Also, is there any better approach?

    BufferedReader reader = new BufferedReader(new FileReader(new File(fileName)));
        String line;
        while((line = reader.readLine()) != null) {
            if(line.contains("COUNTER")) {
                String[] tokens = line.split("\\s+");
                Date date = new SimpleDateFormat("yy/mm/dd-HH:mm:ss.SSS").parse("16/08/29-12:42:48.167");
                //this is not storing the correct value                 
            }

Note that you may use the same `SimpleDateFormat` object for each matching line, instead of creating a new one each time. — Arnaud, Aug 29 '16 at 13:27
As long as you dont have multiple threads working on your file in parallel that is! Then each thread would need its own formatter. — GhostCat, Aug 29 '16 at 13:38
While reading the log file, I would record the "REQUEST"s I encounter in a list. Once I get to a "RESPONSE" line, I would remove the appropriate REQUEST from the list and check if the delay is over 10 seconds. — Aaron, Aug 29 '16 at 13:43

score 2 · Accepted Answer · answered Aug 29 '16 at 17:31

I would take the approach of reading the log file one time and keeping track of request/response using a HashMap. The map key would be the counter and the map value would be the request's date. To save memory, remove items from the map when the response is received. Use Java's pattern matching to determine if it's a request or response and for getting the counter ID and date.

Here's an example:

final static long EXPIRED_MILLIS = 1000 * 10; // ten seconds
final static Pattern REQUEST_PATTERN = Pattern.compile("(.* )(<< \\d{4} )(REQUEST COUNTER=\")(\\d)(\")");
final static Pattern RESPONSE_PATTERN = Pattern.compile("(.* )(<< \\d{4} )(RESPONSE COUNTER=\")(\\d)(\")");
final static SimpleDateFormat DATE_FORMAT = new SimpleDateFormat("yy/MM/dd-HH:mm:ss.SSS");

List<String> getExpiredCounterList(File file) throws Exception {
    Map<String,Long> requestMap = new HashMap<>();
    List<String> expiredCounterList = new ArrayList<>();

    try( FileReader fileReader = new FileReader(file);
         BufferedReader buffReader = new BufferedReader(fileReader) )
    {
        String line;
        while((line = buffReader.readLine()) != null) {
            Matcher requestMatcher = REQUEST_PATTERN.matcher(line);
            if(requestMatcher.matches()) {
                addRequestToMap(line, requestMatcher, requestMap);
                continue;
            }

            Matcher responseMatcher = RESPONSE_PATTERN.matcher(line);
            if(requestMatcher.matches()) {
                String expiredCounter = checkIfExpiredAndRemoveFromMap(line, responseMatcher, requestMap);
                if(expiredCounter != null)
                    expiredCounterList.add(expiredCounter);
            }
        }
    }
    return expiredCounterList;
}

void addRequestToMap(String request, Matcher requestMatcher, Map<String,Long> requestMap) {
    String counter = getCounter(request, requestMatcher, 4);
    long date = getDate(request, requestMatcher, 1);
    requestMap.put(counter, date);
}

String checkIfExpiredAndRemoveFromMap(String response, Matcher responseMatcher, Map<String,Long> requestMap) {
    String counter = getCounter(response, responseMatcher, 4);
    if( requestMap.containKey(counter) ) {
        long date = getDate(response, responseMatcher, 1);
        long elapsedMillis = date - requestMap.remove(counter);
        if(elapsedMillis > EXPIRED_MILLIS)
            return counter;
     }
     return null;
}

String getCounter(String line, Matcher matcher, int group) {
    return line.substring(matcher.start(group), matcher.end(group));
}

long getDate(String line, Matcher matcher, int group) throws ParseException {
    return DATE_FORMAT.parse(line.substring(matcher.start(group), matcher.end(group))).getTime();
}

score 1 · Answer 2 · answered Aug 29 '16 at 13:45

You have made one mistake in code that you used mm to pass the month but you should use MM to pass the value of month. Since m is used to represent minute in hour, You should use M which will represent month. For more Information, Refer this document.

Corrected Code:

BufferedReader reader = new BufferedReader(new FileReader(new File(fileName)));
        String line;
        while((line = reader.readLine()) != null) {
            if(line.contains("COUNTER")) {
                String[] tokens = line.split("\\s+");
                Date date = new SimpleDateFormat("yy/MM/dd-HH:mm:ss.SSS").parse("16/08/29-12:42:48.167");                
            }

When I am printing the output of date, I got the output Mon Aug 29 12:42:48 IST 2016 Hope this is the useful output for your code.

score 1 · Answer 3 · edited May 23 '17 at 12:00

Use java.time

You are using troublesome old legacy date-time classes, now supplanted by the java.time classes.

Instantiate a DateTimeFormatter. Save somewhere; no need to instantiate a new one on each parsing. Thread-safe, unlike the old date-time classes.

DateTimeFormatter f = DateTimeFormatter.ofPattern( "uu/MM/dd-HH:mm:ss.SSS" );

Parse into LocalDateTime since your input data has no information about time zone or offset-from-UTC.

LocalDateTime start = LocalDateTime.parse( input , f );

By the way, your inputs use a poor format for serializing date-time. Instead they should always be in UTC with an explicit notation, and should always use standard ISO 8601 formats such as (note the Z at the end):
2016-01-02T12:34:56.987654321Z

If you know the intended offset or time zone, apply it. Otherwise your date-time math will be done with generic 24-hour days, ignoring any anomalies such as Daylight Saving Time (DST).

The Duration class represents a length of time.

Duration duration = Duration.between( start , stop );

The Duration objects can be compared and sorted, implementing Comparable.

About java.time

The java.time framework is built into Java 8 and later. These classes supplant the troublesome old date-time classes such as java.util.Date, .Calendar, & java.text.SimpleDateFormat.

The Joda-Time project, now in maintenance mode, advises migration to java.time.

To learn more, see the Oracle Tutorial. And search Stack Overflow for many examples and explanations.

Much of the java.time functionality is back-ported to Java 6 & 7 in ThreeTen-Backport and further adapted to Android in ThreeTenABP (see How to use…).

The ThreeTen-Extra project extends java.time with additional classes. This project is a proving ground for possible future additions to java.time. You may find some useful classes here such as Interval, YearWeek, YearQuarter, and more.

Parsing request and response time in a log file

3 Answers3

Use java.time

About java.time