3

I have a web application written in Java using the Spring framework. I would like to store the users activities like, page visits, actions, interactions etc. I read that usually this is done by creating a table for each tracked aspect. I was wondering if there is a better way to do it using Spring framework, like a way to intercept all the requests and trigger some actions. What kind of technology do you recommend to store all these information? Right know I’m using a MySql database interacting with it through JPA. But, since I’m really really new to these kind of things I don’t know if I should go with a NoSql database or stay with my already existing MySql database. This wonder comes from the idea that this kind of data flow will be much bigger than a normal data flow coming from more traditional actions such as signin, creation, deletion etc. Hope to have explained myself... if not just tell me and I’ll try to add more details.

[EDIT 1]

The web app is an e-commerce. So far it does not have So many users but it will (in the order of thousands).

The goal of the user tracking it’s just to profile them in order to give them a better and more custom service. For instance, if a see that a user is taking a look to a lot of items of a precise category I can show him more items of that kind.

I do no care that much about the performance, I mean, it does not have to be so fast.

Right know I have just one database and everything is stored inside it. I don’t know if charging it with this kind of data flow would slow down its performance.

The application is running on AWS ElasticBeanstalk and the database is on AWS RDS.

Stefano Sambruna
  • 767
  • 2
  • 10
  • 29

2 Answers2

3

In general its a very broad topic.

The following considerations come to my mind:

  1. How many requests pass to the microservice per some period of time? If its a small number of users (which translates to the number of records to the database) - then its ok to go with the MySQL approach - the table won't be large. Note however, that sometimes it should be cleaned anyway

  2. Is the latency important? Sometimes requests have to be served very quickly, adding a hop to the database to save the user preference can be problematic

  3. How do you want to consume this kind of information? Are you planning to use dashboards (in this case micrometer + Prometheus / InfluxDB and Grafana can be a good solution). Are you planning to actually charge the users per number of requests with an ability to send the monthly bill to their email in PDF or provide a web access to such an information (like AWS does for example)?

  4. How about Rate limiter? Are you planning to deny some requests if they're frequent and coming from the same user?

  5. How many instance will "add" this kind of information? What if you have thousands of microservices that now will have to write to MySQL - it might not survive such a load (in addition to the regular load its set up for)?

The range of solutions can vary. You can Batch the requests per user in memory and send once in while a message into Kafka and then use kafka streams to provide aggregations on it. With this approach you'll minimize the impact of the added processing on the existing solution and will deploy some other service that will be able to process this pretty large amount of data.

Another option: maybe you can create an asynchronously populated log file and store the information there. Then you might want to add some "agent" / side-car container like logstash and stream the data into some storage. Yet Another project that might be relevant in this field is Apache Flume which will allow you to construct a pipeline.

For billing you might use specialized systems AFAIK spring doesn't have anything like this usually these are ready products that you can integrate with.

For Rate Limiting you might consider: Resilience4j or solve it with redis

Mark Bramnik
  • 39,963
  • 4
  • 57
  • 97
0

Yeah , That's possible , Here below are the three approaches with some sample snippets which would help you in the implementation , Moreover it depends on the data you store when you log the activity and when do you consider the activity data as obsolete and there are many factors which can decides your data store.

Approach 1: You can keep track of the login user using Spring-Security

You can write a HTTPSessionBindingListener and track the actions something like this

@Component
public class LoggedUser implements HttpSessionBindingListener {

    private String username; 
    private ActiveUserStore activeUserStore;

    public LoggedUser(String username, ActiveUserStore activeUserStore) {
        this.username = username;
        this.activeUserStore = activeUserStore;
    }

    public LoggedUser() {}

    @Override
    public void valueBound(HttpSessionBindingEvent event) {
        List<String> users = activeUserStore.getUsers();
        LoggedUser user = (LoggedUser) event.getValue();
        if (!users.contains(user.getUsername())) {
            users.add(user.getUsername());
        }
    }

    @Override
    public void valueUnbound(HttpSessionBindingEvent event) {
        List<String> users = activeUserStore.getUsers();
        LoggedUser user = (LoggedUser) event.getValue();
        if (users.contains(user.getUsername())) {
            users.remove(user.getUsername());
        }
    }

    // standard getter and setter
}

and for login and logout you can track using AuthenticationSuccessHandler

@Component("myAuthenticationSuccessHandler")
public class MySimpleUrlAuthenticationSuccessHandler implements AuthenticationSuccessHandler {

    @Autowired
    ActiveUserStore activeUserStore;

    @Override
    public void onAuthenticationSuccess(HttpServletRequest request, 
      HttpServletResponse response, Authentication authentication) 
      throws IOException {
        HttpSession session = request.getSession(false);
        if (session != null) {
            LoggedUser user = new LoggedUser(authentication.getName(), activeUserStore);
            session.setAttribute("user", user);
        }
    }
}

Approach 2 : The other method if you want to make it very simple is that you can write a OncePerRequestFilter

@Component
@Ordered(Ordered.LOWEST_PRECEDENCE)
public class LogFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
        throws ServletException, IOException {
        // Log the info you need
        // ...
        filterChain.doFilter(request, response);
    }
}

Approach 3 : Implement using Spring AOP.

@Aspect
@Component
public class WebMethodAuditor {

protected final Log logger = LogFactory.getLog(getClass());

public static final String DATE_FORMAT_NOW = "yyyy-MM-dd HH:mm:ss";

@Autowired
AuditRecordDAO auditRecordDAO; 

@Before("execution(* com.mycontrollers.*.*(..))")
public void beforeWebMethodExecution(JoinPoint joinPoint) {
    Object[] args = joinPoint.getArgs();
    String methodName = joinPoint.getSignature().getName();
    User principal = (User)SecurityContextHolder.getContext().getAuthentication().getPrincipal();
    Timestamp timestamp = new Timestamp(new java.util.Date().getTime());
    // only log those methods called by an end user
    if(principal.getUsername() != null) {
        for(Object o : args) {
            Boolean doInspect = true;
            if(o instanceof ServletRequestDataBinder) doInspect = false;
            if(o instanceof ExtendedModelMap) doInspect = false;
            if(doInspect) {
                if(o instanceof BaseForm ) {
                    // only show form objects
                    AuditRecord ar = new AuditRecord();
                    ar.setUsername(principal.getUsername());
                    ar.setClazz(o.getClass().getCanonicalName());
                    ar.setMethod(methodName);
                    ar.setAsString(o.toString());
                    ar.setAudit_timestamp(timestamp);
                    auditRecordDAO.save(ar);
                }
            }
        }
    }
}

}

Source and More details :

https://www.baeldung.com/spring-security-track-logged-in-users

Spring / AOP: Best way to implement an activities log in the database

What is the best way to log Activity in Spring Boot with Thymeleaf?

redhatvicky
  • 1,912
  • 9
  • 8