Creating a hit counter using the cache or application scope

Question

I would like to create a hit counter for my ColdFusion app. I don't want the database hits table to be updated on each page hit. Ideally, I would like to aggregate the hits in the app scope, or the cache in some type of struct, then save them intermittently.

I have to ideas so far:.

Idea 1

Create an app or cache struct to store pageId and pageHits key value pair.
On each page hit, check for the existence of pageId/ip address cookie.
a. If it exists, do nothing.
b. If it does not, set it and update the hits struct in the app or cache.

Problem: I don't know how to get the struct data into the database intermittently/after a certain time has elapsed. Scheduled event?

Idea 2

No cookies; save pageId/ip address in their own cache/app struct.

Problem I'm not sure how to structure the data structs/arrays to store the info. Seems complicated with array loops and struct finds. And I still have the same problem as idea one.

Any advice, ideas, criticism? In particular, I need some help figuring out the data structures for idea 2. I'm more interested in performance than data integrity and happy with CF9 only solution. I should add I'm looking to denormalize data into a page hits column for every page id. I don't need normalized data tables.

orangepips · Accepted Answer · 2011-05-31T18:32:18.440

Trying to capture this data in the ways you've describe introduces scaling problems with cache expiration to avoid OOM or long iteration times as the number of entries grows when you ultimately want to persist to a database. The information you want to aggregate is already captured in web server logs. Parsing these is a straightforward activity, you just need to write an algorithm that's aware of how and when these rollover.

Use log = fileRead('log.txt', 'read') and iterate over line fileReadLine(log) on a periodic basis via <cfschedule/>. For each batch you can in the fashion you've already described and then use whatever database approach for INSERT/UPDATE (usually uses the MERGE keyword, but MySQL is different) based on IP address if that's what you're after. If it's a very high traffic site, consider altering the data type of the IP address column to an integer datatype to speed index lookups.

Update

Use the following code to setup a file object reference:

<cflock name="logparser" type="exclusive" timeout="1" throwontimeout="false">
    <cfif NOT structKeyExists(application, "logFile")>
        <cfset application.logFile =
            fileOpen('/path/to/log.txt', 'read')>
    </cfif>
    <cfloop condition="NOT FileisEOF(application.logFile)">
        <!--- replace with an appropriate algorithm --->
        <cfoutput>
            #fileReadLine(application.logFile)#
            <br />
        </cfoutput>
    </cfloop>
</cflock>

Doing this sets up the reference once (i.e. fileOpen()), which keeps track of what line it's on. Then when <cfschedule/> hits this code again it uses the existing reference if present and iterates forward from there. The lock ensures only one thread executes the algorithm - so you don't need to to worry about a time constraint just let it keep on going as long as it needs to.

thanks for clearing this up. One thing I forgot to mention: I will denormalize hit data into a single page hit column since the pages themselves are saved in the db): if I run this service every 60 seconds, and the log files become really large, how do I read only 60 seconds worth of log lines? If my log file has 10,000 lines, but I only need the lines created in the the last 60 seconds... would this approach be practical? I'm not looking for normalized data. — Mohamad, May 31 '11 at 17:57
@Mohamad: see my update. Short answer is don't worry about the time constraint, use `` with a `fileOpen()` reference instead. — orangepips, May 31 '11 at 18:32
I haven't started implementing this, but I will give it a go this weekend. I thinking this morning, though, and it occurred to me: How would you track unique hits! If each visitor counted as a unique hit after 3 hours, the algorithm would have to parse each line and figure out if there's a duplicated value with the last x time.. would that be practical? — Mohamad, Jun 03 '11 at 15:36
@Mohamad: I would probably put each hit into a database table including IP address, URL and timestamp. Then run a DISTINCT over those lines and use a WHERE clause to constrain the time period. That should give you the information you want. — orangepips, Jun 03 '11 at 16:06

score 1 · Answer 2 · answered May 31 '11 at 17:23

1

Idea 1: yes, <cfschedule> is your friend

Idea 2: save pageId/ip address? I guess this is better to use Set from Java, but struct would work too if you assign it an empty value...

pageIDs["1"]["192.0.0.1"] = "";
hitsOfPage1 = structCount(pageIDs[1]));

However, if I visit the page again tmr, I won't be counted?

answered May 31 '11 at 17:23

Henry

32,689
19
120
221

why wouldn't I do something like `hits[1].page = { pageId = 1, ipAddress = 192.0.0.0 }` or what ever? I'm having a hard time visualizing the struct of the data in such a scenario! Also, in the cookie scenario, I can have it expire after a few hours. In the second idea, I have no clue how it would work yet. – Mohamad May 31 '11 at 17:26
1

because the most frequent answer you want for a hit counter is... how many hits are there for pageID = X. You want a data structure that supports that. Your purposed data structure can't answer the "where pageId = 1" part easily. – Henry May 31 '11 at 17:36

Creating a hit counter using the cache or application scope

2 Answers2