logging in the same application with different log files

Question

I have an application that processes tasks. each task goes through allot of stuff. from all kinds of queues, some stuff in parallel. logging can happen naturally on every part of the way. I need each task to have a different log file, with the log name as a unique id of this task. I have start examining log4net, but is looks like it is not the best tool for the job. (maybe it is) Does anybody have an idea, for a tool that this kind of logging is built in it's design. Thank you very much.

Possible duplicate of [Configure Log4net to write to multiple files](http://stackoverflow.com/questions/1372435/configure-log4net-to-write-to-multiple-files) — christutty, Mar 14 '16 at 11:57
I need lots of different loggers (one for each task, and the tasks keeps coming), that why I think log4net may not be the right solution for me. — tal, Mar 14 '16 at 12:02
Can you explain why you need the log file for each task to be unique? It will help with the assessing of alternative solutions. — christutty, Mar 14 '16 at 12:05
It will be easier for he user to look at a specific task (a task is all kind of processing on a specific set of images, every few minutes a new set arrives and get processed), and see the successes and failures of it. — tal, Mar 14 '16 at 12:08

score 1 · Accepted Answer · edited May 23 '17 at 11:45

In my opinion logging to file is the wrong approach. Most task processing frameworks I've seen use a structured data store (such as a databaseor a single structured log file) as you don't have the file processing overhead of creating a file per task and can view filtered task status while multiple tasks are writing to the logs. It's also more scalable, easier to wrap an interface around and you can search across multiple task logs ("find me all critical errors in the last hour) so you can attach a downstream critical alert module, for instance.

The task engines that I'm most familiar with use a database to manage the work queue so a database is the natural choice for logging. I've generally written a simple logging class to output messages but I think you could still use log4net configured for database logging.

That said, if you're sure that a file per task is the approach that you want this question Logging to an individual log file for each individual thread covers creating a log file per thread that I think should be convertible to a log file per task. The answers include log4net config (using a pattern with a thread id in the filename that for you could be the task id)...

<appender name="XmlAppender" type="log4net.Appender.FileAppender">
  <file type="log4net.Util.PatternString" value="D:\Temp\Logs\%property{LogName}.log" />
  <immediateFlush value="true"/>
  <appendToFile value="true" />
  <layout type="log4net.Layout.SimpleLayout" />
</appender>

... and C# code to open the log file...

ILoggerRepository loggerRepository = LogManager.CreateRepository(logFileName + "Repository");
ThreadContext.Properties["LogName"] = logFileName;
log4net.Config.XmlConfigurator.Configure(loggerRepository);
ILog logger = LogManager.GetLogger(logFileName + "Repository", "ProductionLogger");

Credit to https://stackoverflow.com/users/21682/adam for the specific code above.

logging in the same application with different log files

1 Answers1