Is there a way to stream a log to a Spring Boot application?

Question

I have a Spring Boot app that is used as an event logger. Each client sends different events via a REST api, which are then saved in a database. But apart from simple events, I need the clients to also send their execution logs to Spring Boot.

Now, uploading a log after a client finishes executing is easy, and there are plenty examples for it out there. What I need is to stream the log as the client is executing, line by line, and not wait until the client has finished.

I've spent quite some time googling for a possible answer and I couldn't find anything that fits my needs. Any advice how to do this using Spring Boot (future releases included)? Is it feasible?

Can you be more clear about what kind of clients are involved? Assuming you are writing the clients and control their behavior, and they're also written in Java, you could create your own Appender for whichever logging framework you're using, and that Appender could hit your REST api for each logging statement, instead of writing to a file. I would recommend doing the actual REST calls asynchronously by spawning another thread in the Appender implementation. — John, Aug 31 '17 at 17:37
@John the clients are Java SE, and yes I am writing the code for them. [Andrew](https://stackoverflow.com/users/1898535/andrew-kaluzniacki) already suggested something similar so appenders will be first thing I am going to try. Thanks for the reply! — Igorski, Aug 31 '17 at 20:06

score 0 · Accepted Answer · edited Mar 22 '19 at 15:53

0

I see a couple of possibilities here. First, consider using a logback (the default Spring Boot logging implementation) SocketAppender or ServerSocketAppender in your client. See: https://logback.qos.ch/manual/appenders.html. This would let you send log messages to any logging service. But I might suggest that you not log to your Spring Boot Event App as I suspect that will add complexity to your app unnecessarily, and I can see a situation where there is some bug in the Event App that then causes clients to log a bunch of errors which in turn all go back to the event app making it difficult to determine the initial error.

What I would respectfully suggest is that you instead log to a logging server - logstash: https://www.elastic.co/products/logstash for example, or if you already have a db that you are saving the event to, then maybe use the logbook DBAppender and write the logs directly to a db.

edited Mar 22 '19 at 15:53

Slava Semushin

14,904
7
53
69

answered Aug 31 '17 at 17:45

Andrew Kaluzniacki

89
9

I already considered logstash (and the whole elk stack), I just didn't know how it would fit my security setup. And I didn't want to add an additional level of complexity to the project. The appenders on the other hand, seem like a much straight forward solution so I will be trying that one first. Thank you for the answer! – Igorski Aug 31 '17 at 20:09
You might be able to find a logback http appender if you look: https://github.com/kewang/logback-http-appender https://stackoverflow.com/questions/38271359/logback-appender-to-post-message-as-http-message – Andrew Kaluzniacki Aug 31 '17 at 22:52
In the end I did chose Logstash. I added a log controller in my Spring Boot app that accepts the requests from my client's http log appenders. All the content the controller receives from the clients it just logs into a separate log file. I then simply hooked Logstash to that log file. I don't know if it is the ideal setup, but it at least allowed me to use my existing Spring Boot security. – Igorski Sep 03 '17 at 11:16

Mihaita Tinta · Answer 2 · 2021-08-26T13:13:09.483

I wrote here an example on how to stream file updates in a spring boot endpoint. The only difference is that the code uses the Java WatchService API to trigger file updates on a given file.

However, in your situation, I would also choose the log appender to directly send messages to the connected clients (with sse - call template.broadcast from there) instead of watching for changes like I described.

The endpoint:

@GetMapping(path = "/logs", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public SseEmitter streamSseMvc() {
    return sseService.newSseEmitter();
}

The service:

public class LogsSseService {
private static final Logger log = LoggerFactory.getLogger(LogsSseService.class);

private static final String TOPIC = "logs";
private final SseTemplate template;
private static final AtomicLong COUNTER = new AtomicLong(0);

public LogsSseService(SseTemplate template, MonitoringFileService monitoringFileService) {
    this.template = template;
    monitoringFileService.listen(file -> {

        try {
            Files.lines(file)
                    .skip(COUNTER.get())
                    .forEach(line ->
                            template.broadcast(TOPIC, SseEmitter.event()
                                    .id(String.valueOf(COUNTER.incrementAndGet()))
                                    .data(line)));
        } catch (IOException e) {
            e.printStackTrace();
        }
    });
}

public SseEmitter newSseEmitter() {
    return template.newSseEmitter(TOPIC);
}

}

The custom appender (which you have to add to your logger - check here):

public class StreamAppender extends UnsynchronizedAppenderBase<ILoggingEvent> implements SmartLifecycle {
public static final String TOPIC = "logs";

private final SseTemplate template;

public StreamAppender(SseTemplate template) {
    this.template = template;
}

@Override
protected void append(ILoggingEvent event) {
    template.broadcast(TOPIC, SseEmitter.event()
            .id(event.getThreadName())
            .name("log")
            .data(event.getFormattedMessage()));
}

@Override
public boolean isRunning() {
    return isStarted();
}

}

Is there a way to stream a log to a Spring Boot application?

2 Answers2