2

I have tested HTTP receiver with both WSO2 CEP 4.1.0 & 4.2.0. With same set of data, streams, execution plan(just event count wit 1 min window), publisher(logger) 4.1.0 is able to process 1M/minute but 4.2.0 could go up to 200K-300K/minute.

Have conducted the same test with my custom file receiver and found similar result as above.

By the way, I have tested it on my laptop, 3 MB file with 10 Millions records.

  • Is there anybody face same issue ?
  • Appreciate if anybody can explain the issues behind this ?

Thanks, Obaid

Community
  • 1
  • 1
Obaid
  • 237
  • 2
  • 14

2 Answers2

3

Just completed the test and have a feeling that logger publisher in 4.2.0 is the culprit here. Please find below outcomes and share your opinion (M=Million):

  1. HTTP Receiver only push to Event Stream:

    • Both 4.1 & 4.2 was able to push 2.25M events/minute.

  2. HTTP Receiver + below execution plan + logger:

    • 4.1: 1.25M events/minute.
    • 4.2: 250K events/minute.

  3. HTTP Receiver + below execution plan + "NO" logger:

    • 4.1: 2.25M events/minute.
    • 4.2: 2.25M events/minute.

  4. Custom File Receiver + below execution plan + "NO" logger:

    • 4.1: 1.5M events/minute.
    • 4.2: 3M events/minute (I am surprised).

Execution Plan:

@Import('subsApps:1.0.0')
define stream subsApps (meta_app_name string, correlation_imsi long, volume double);

@Export('subsCount:1.0.0')
define stream subsCount (meta_counts long);

from subsApps#window.time( 1 min ) 
select count() as meta_counts 
insert into subsCount;
Obaid
  • 237
  • 2
  • 14
0

This could not happen from HTTP receiver end, according to source codes of carbon-analyics-common packed with 4.1.0 [1] and 4.2.0 [2] there are no significant changes. The only change is if checking added for basic auth. This could be some other issue. Specially we can check on execution plan as well. its better if you could try to do same test by using wso2event adapter which does not have any changes in source codes of carbon-analyics-common packed with 4.1.0 [3] and 4.2.0 [4].

[1] https://github.com/wso2/carbon-analytics-common/tree/release-5.0.10/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http

[2] https://github.com/wso2/carbon-analytics-common/tree/release-5.1.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http

[3] https://github.com/wso2/carbon-analytics-common/tree/release-5.0.10/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.wso2event/src/main

[4] https://github.com/wso2/carbon-analytics-common/tree/release-5.1.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.wso2event/src/main/java/org/wso2/carbon/event/input/adapter/wso2event

Tharik Kanaka
  • 2,490
  • 6
  • 31
  • 54
  • Hi Tharik, I have done same test with my custom file event adapter. Same bundle having slower performance in 4.2. Anyway, I could test on wso2event, it may take some time. – Obaid Sep 26 '16 at 05:00
  • Before than you can simplify the problem by just testing HTTP receiver witha a passthrogh flow by eliminating execution plans in the flow. Then you can compare 4.1.0 and 4.2.0 and verify whether is a HTTP receiver or not – Tharik Kanaka Sep 26 '16 at 05:27
  • Sure, let me do it and share you the results – Obaid Sep 26 '16 at 09:28