4

I'm very impressed from the new x-pack ML of the elastic stack. It seems their technique learns data patterns over time and can predict anomalies in multiple domains.

enter image description here

Zoomed in: enter image description here

I was wondering what approach and network topology could be used, in order to create a similar feature. Is it fair to assume that since x-pack works on time series data, RNN would be a good start?

Interested in your opinions and references.

Shlomi Schwartz
  • 8,693
  • 29
  • 109
  • 186
  • duplicate:https://stackoverflow.com/questions/47213120/machine-learning-algorithms-used-by-elastic-x-pack-plugin/51632375 – walterra Aug 01 '18 at 11:47

2 Answers2

2

From what I read, I'd be surprised if they're using neural networks. Here's how they say they detect anomalies:

Detect outliers in a population by building a profile of a “typical” user or machine to know when one starts to stray from the pack.

Doing anomaly detection like that requires nothing more than a statistical test of whether or not observed behavior is within 2-3 standard deviations of the expected behavior.

If you want to use neural networks for some reason, you could go with CNNs, RNNs, or attention-only networks. Google recently showed that you don't need RNNs or CNNs to do state-of-the-art translation.

Brian Bartoldson
  • 884
  • 9
  • 20
  • The attention mechanism proposed in the paper, is just a part of the NN "The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, positionwise fully connected feed-forward network" – Roni Gadot Jul 06 '17 at 06:44
  • @RoniGadot, my point was that the OP doesn't necessarily need something complicated like an RNN. Google's network is a feed-forward (no recurrences) network that operates on sequences. – Brian Bartoldson Jul 06 '17 at 16:26
0

I've been looking at their Anomaly Detection. I'm not a math major, but the results have given me goosebumps. (If only it could send me an alert! See here: Do Elastic Anomaly Detection Rules Actually Work?)

I am pretty sure they are not using neural net. From what I've read, that's a subset of Machine Learning.

McJava1967
  • 15
  • 2
  • This does not provide an answer to the question. Once you have sufficient [reputation](https://stackoverflow.com/help/whats-reputation) you will be able to [comment on any post](https://stackoverflow.com/help/privileges/comment); instead, [provide answers that don't require clarification from the asker](https://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead). - [From Review](/review/late-answers/34704315) – Koedlt Jul 22 '23 at 11:50