How to efficiently count HTTP Calls in asp.net core?

Question

I have an abstract class called HttpHelper it has basic methods like, GET, POST, PATCH, PUT
What I need to achieve is this:

Store the url, time & date in the database each time the function is called GET, POST, PATCH, PUT

I don't want to store directly to the database each time the functions are called (that would be slow) but to put it somewhere (like a static queue-memory-cache) which must be faster and non blocking, and have a background long running process that will look into this cache-storage-like which will then store the values in the database.

I have no clear idea how to do this but the main purpose of doing so is to take the count of each calls per hour or day, by domain, resource and url query.

I'm thinking if I could do the following:

1. Create a static class which uses ConcurrentQueue<T> to store data and call that class in each function inside HttpHelper class
2. Create a background task similar to this: Asp.Net core long running/background task
3. Or use Hangfire, but that might be too much for simple task

Or is there a built-in method for this in .netcore?

Answer 1

Both Hangfire and background tasks would do the trick as consumers of the queue items.

Hangfire was there before long running background tasks (pre .net core), so go with the long running tasks for net core implementations.

There is a but here though.

How important is to you that you will not miss a call? If it is, then neither can help you.

The Queue or whatever static construct you have will be deleted the time your application crashes/machine restarts or just plain recycling of the application pools.

You need to consider some kind of external Queuing mechanism like rabbit mq with persistence on.

You can also append to a file, but that might also cause some delays as read/write.

Answer 2

I do not know how complex your problem is but I would consider two solutions.

First is calling Async Insert Method which will not block your main thread but will start task. You can return response without waiting for your log to be appended to database. Since you want it to be implemented in only some methods, I would do it using Attributes and Middleware.

Simplified example:

public IActionResult SomePostMethod()
{
    LogActionAsync("This Is Post Method");

    return StatusCode(201);
}

public static Task LogActionAsync(string someParameter)
{
    return Task.Run(() => {
        // Communicate with database (X ms)
    });
}

Better solution is creating buffer which will not communicate with database each time but only when filled or at interval. It would look like this:

public IActionResult SomePostMethod()
{
    APILog.Log(new APILog.Item() { Date = DateTime.Now, Item1 = "Something" });

    return StatusCode(201);
}

public partial class APILog
{
    private static List<APILog.Item> _buffer = null;
    private cont int _msTimeout = 60000; // Timeout between updates
    private static object _updateLock = new object();

    static APILog()
    {
        StartDBUpdateLoopAsync();
    }

    private void StartDBUpdateLoopAsync()
    {
        // check if it has been already and other stuff
        Task.Run(() => {
            while(true) // Do not use true but some other expression that is telling you if your application is running.
            {
                Thread.Sleep(60000);
                
                lock(_updateLock)
                {
                    foreach(APILog.Item item in _buffer)
                    {
                        //Import into database here
                    }
                }
            }
        });
    }
    
    public static void Log(APILog.Item item)
    {
        lock(_updateLock)
        {
            if(_buffer == null)
                _buffer = new List<APILog.Item>();
                
            _buffer.Add(item);
        }
    }
}

public partial class APILog
{
    public class Item
    {
        public string Item1 { get; set; }
        public DateTime Date { get; set; }
    }
}

Also in this second example I would not call APILog.Log() each time but use Middleware in combination with Attribute