What does rules engine mean in Stream Analytics on Azure?

Question

I'm new to Azure and to analytics. I'm trying to understand streaming alert rules engine. I have used some sample data as input and have queries to filter data. However I'm not sure of what rules engine means, is it just queries or is there anything more to it and is there a way we can have all rules in one if yes, how?

Answer 1

The main way to define logic for ASA is to use SQL, which provide a way to define rules with SQL statements (e.g. SELECT DeviceID ... WHERE temperature>50). Multiple conditions can be used in the same query and multiple queries can be defined in the same job. This approach is pretty flexible, however the rules themselves need to be defined before the job start since ASA will compile the job. See here for some common query patterns.

Then, when we talk about rule engine, users often require more dynamicity of the rules. In order to provide this dynamicity, it is possible to dynamically inject rule conditions, or even full rules to a running job (the later one is for advanced users). Below some detailed info:

• Dynamically configure threshold rules using reference data: ASA can use reference data to get the latest conditions. See more details in this blog post
• Dynamically inject rules using JavaScript UDF (this is more advanced): users can create a new function using JavaScript Function using code snippets coming from reference data. Such technique is used by Azure Remote Monitoring microservices (see here on GitHub).

Let me know if you have any further question and how we can improve.

Thanks,

JS (Azure Stream Analytics)

Answer 2

Rules engines are frequently used in real time streaming for analytics, notifications, and firing rules when patterns in data are matched.

The typical format is

• When - a predicate used to determine whether pattern(s) are matched or not
• Then - action(s) to be triggered when the pattern is matched. Often, in event driven architecture, the action is simply to raise a new event.

These are especially useful in an environment where a stream of events can be monitored by the engine.

Simple rules can be constrained to the data in a single event (often called a 'fact' in rules terminology), e.g. a rule as per below only needs a single event:

1. Alert me when I receive a temperature reading above 50 degrees celcius

However, more sophisticated rules require knowledge of multiple facts (not necessarily homogeneous facts), and can also reason over and windows of time, e.g.

2. Alert me if the average temperature over the last 5 minutes exceeds 50 celcius

3. Alert me if the average temperature over the last 5 minutes exceeds 50 celcius but only if the fan has not switched on.

... these require the context of more than just a single event (and #3 needs 2 event types) in order to make a decision.

A naive rules engine could simply persist all events to storage, and then execute a predicate across a persisted data source, every time a new temperature event is received, e.g. (Pseudo SQL)

SELECT 
   CASE AVG(t.Temp) 
     WHEN > 50 THEN 'Fire!'
     ELSE 'OK'
   END
FROM 
   TempMeasures t
WHERE 
   t.TimeStamp > CURRENT_TIMESTAMP - 5 mins

However, this can be slow, and resource intensive, and does not scale well if there are a high frequency of events and a large number of rules.

More exotically, rules engines can also be used to detect the absence of an event, e.g.

4. Alert me if I haven't received a temperature measurement at all within the last 5 minutes.

In the naive scenario, this would require polling of the data source (polling frequency would be determined by the latency you'd be able to tolerate), again presenting inefficiency and lack of scalability.

More advanced techniques exist like the Rete Algorithm which are able to retain context without relying on a complete database / record of all prior events, and can infer which facts need to be retained, and for how long.

e.g. In the above rule examples, it can logically be deduced that

• any event older than 5 minutes can be discarded as it is not needed for these rules (all rules, #1-4)
• any time a temperature event is received, there is no need to check rule #4 for another 5 minutes

etc.