Problem when querying Raw Data with STH-Comet - Returns empty

Question

I have Orion, Cygnus and STH-Comet(installed and configured in formal mode). Each component is in a container docker. I implemented the infrastructure with docker-compose.yml.

The Cygnus container is configured as follows:

    image: fiware/cygnus-ngsi:latest
    hostname: cygnus
    container_name: cygnus
    volumes:
      - /home/ubuntu/cygnus/multisink_agent.conf:/opt/fiware-cygnus/docker/cygnus-ngsi/multisink_agent.conf
    depends_on:
      - mongo
    networks:
      - default
    expose:
      - "5050"
      - "5080"
    ports:
      - "5050:5050"
      - "5080:5080" 
    environment:
      - CYGNUS_SERVICE_PORT=5050
      - CYGNUS_MONITORING_TYPE=http
      - CYGNUS_AGENT_NAME=cygnus-ngsi
      - CYGNUS_MONGO_SERVICE_PORT=5050
      - CYGNUS_MONGO_HOSTS=mongo:27017
      - CYGNUS_MONGO_USER=
      - CYGNUS_MONGO_PASS=
      - CYGNUS_MONGO_ENABLE_ENCODING=false
      - CYGNUS_MONGO_ENABLE_GROUPING=false
      - CYGNUS_MONGO_ENABLE_NAME_MAPPINGS=false
      - CYGNUS_MONGO_DATA_MODEL=dm-by-entity
      - CYGNUS_MONGO_ATTR_PERSISTENCE=column
      - CYGNUS_MONGO_DB_PREFIX=sth_
      - CYGNUS_MONGO_COLLECTION_PREFIX=sth_   
      - CYGNUS_MONGO_ENABLE_LOWERCASE=false
      - CYGNUS_MONGO_BATCH_TIMEOUT=30
      - CYGNUS_MONGO_BATCH_TTL=10
      - CYGNUS_MONGO_DATA_EXPIRATION=0
      - CYGNUS_MONGO_COLLECTIONS_SIZE=0
      - CYGNUS_MONGO_MAX_DOCUMENTS=0
      - CYGNUS_MONGO_BATCH_SIZE=1
      - CYGNUS_LOG_LEVEL=DEBUG
      - CYGNUS_SKIP_CONF_GENERATION=false 
      - CYGNUS_STH_ENABLE_ENCODING=false
      - CYGNUS_STH_ENABLE_GROUPING=false
      - CYGNUS_STH_ENABLE_NAME_MAPPINGS=false
      - CYGNUS_STH_DB_PREFIX=sth_
      - CYGNUS_STH_COLLECTION_PREFIX=sth_
      - CYGNUS_STH_DATA_MODEL=dm-by-entity
      - CYGNUS_STH_ENABLE_LOWERCASE=false
      - CYGNUS_STH_BATCH_TIMEOUT=30
      - CYGNUS_STH_BATCH_TTL=10
      - CYGNUS_STH_DATA_EXPIRATION=0
      - CYGNUS_STH_BATCH_SIZE=1

Obs: In the multisink_agent.conf file I changed the service and the servicepath:

cygnus-ngsi.sources.http-source-mongo.handler.default_service = tese

cygnus-ngsi.sources.http-source-mongo.handler.default_service_path = /iot

And the STH-Comet container looks like this:

    image: fiware/sth-comet:latest
    hostname: sth
    container_name: sth
    depends_on:
      - cygnus
      - mongo
    networks:
      - default
    expose:
      - "8666"
    ports:
      - "8666:8666"
    environment:
      - STH_HOST=0.0.0.0
      - STH_PORT=8666
      - DB_URI=mongo:27017
      - DB_USERNAME=
      - DB_PASSWORD=
      - LOGOPS_LEVEL=DEBUG

In the STH-Comet config.js file I enabled CORS and I changed the defaultService and the defaultServicePath. The file looks like this:

var config = {};

// STH server configuration
//--------------------------
config.server = {
    host: 'localhost',
    port: '8666',
    // Default value: "testservice".
    defaultService: 'tese',
    // Default value: "/testservicepath".
    defaultServicePath: '/iot',
    filterOutEmpty: 'true',
    aggregationBy: ['day', 'hour', 'minute'],
    temporalDir: 'temp',
    maxPageSize: '100'
};

// Cors Configuration
config.cors = {
    // The enabled is use to set CORS policy
    enabled: 'true',
    options: {
        origin: ['*'],
        headers: [
            'Access-Control-Allow-Origin',
            'Access-Control-Allow-Headers',
            'Access-Control-Request-Headers',
            'Origin, Referer, User-Agent'
        ],
        additionalHeaders: ['fiware-servicepath', 'fiware-service'],
        credentials: 'true'
    }
};

// Database configuration
//------------------------
config.database = {
    dataModel: 'collection-per-entity',
    user: '',
    password: '',
    authSource: '',
    URI: 'localhost:27017',
    replicaSet: '',
    prefix: 'sth_',
    collectionPrefix: 'sth_',
    poolSize: '5',
    writeConcern: '1',
    shouldStore: 'both',
    truncation: {
        expireAfterSeconds: '0',
        size: '0',
        max: '0'
    },
    ignoreBlankSpaces: 'true',
    nameMapping: {
        enabled: 'false',
        configFile: './name-mapping.json'
    },
    nameEncoding: 'false'
};

// Logging configuration
//------------------------
config.logging = {
   
    level: 'debug',
    format: 'pipe',
    proofOfLifeInterval: '60',
    processedRequestLogStatisticsInterval: '60'
};

module.exports = config;

I use Cygnus to persist historical data. STH-Comet is used only to query raw and aggregated data.

Cygnus' signature on Orion did this:

  "description": "A subscription All Entities",
  "subject": {
    "entities": [
      {
        "idPattern": ".*"
      }
    ],
    "condition": {
      "attrs": []
    }
  },
  "notification": {
    "http": {
      "url": "http://cygnus:5050/notify"
    },
    "attrs": [],
    "attrsFormat":"legacy"
  },
  "expires": "2040-01-01T14:00:00.00Z",
  "throttling": 5
}

The headers used for fiware-service and fiware-servicepath are:

Fiware-service: tese

Fiware-servicepath: /iot

The entities data are stored in orion-tese. I have the collection: entities

{
        "_id" : {
                "id" : "Tank1",
                "type" : "Tank",
                "servicePath" : "/iot"
        },
        "attrNames" : [
                "temperature"
        ],
        "attrs" : {
                "temperature" : {
                        "value" : 0.333,
                        "type" : "Float",
                        "mdNames" : [ ],
                        "creDate" : 1594334464,
                        "modDate" : 1594337770
                }
        },
        "creDate" : 1594334464,
        "modDate" : 1594337771,
        "lastCorrelator" : "f86d0d74-c23c-11ea-9c82-0242ac1c0005"
}

The raw and aggregated data are stored in sth_tese. I have the collections:

sth_/iot_Tank1_Tank.aggr

and

sth_/iot_Tank1_Tank

The sth_/iot_Tank1_Tank raw data is in mongoDB:

{
        "_id" : ObjectId("5f079d0369591c06b0fc981a"),
        "temperature" : 279,
        "recvTime" : ISODate("2020-07-09T22:41:05.670Z")
}
{
        "_id" : ObjectId("5f07a9eb69591c06b0fc981b"),
        "temperature" : 0.333,
        "recvTime" : ISODate("2020-07-09T23:36:11.160Z")
}

When I run: http://localhost:8666/STH/v1/contextEntities/type/Tank/id/Tank1/attributes/temperature?aggrMethod=sum&aggrPeriod=minute

or

http://localhost:8666/STH/v2/entities/Tank1/attrs/temperature?type=Tank&aggrMethod=sum&aggrPeriod=minute

I have the result: "sum": 279 and "sum": 0.333. I can recover ALL the aggregated data, max, min, sum, sum2.

The difficulty is with the STH-Comet when I try to retrieve the raw data, the return code is 200 and the value returns empty.

I've tried with APIs v1 and v2, to no avail.

request with v2:

http://sth:8666/STH/v2/entities/Tank1/attrs/temperature?type=Tank&lastN=10

Return

{
  "type": "StructuredValue",
  "value": []
}

request with v1:

http://sth:8666/STH/v1/contextEntities/type/Tank/id/Tank1/attributes/temperature?lastN=10

Return

{
  "contextResponses": [{
    "contextElement": {
      "attributes": [{
        "name": "temperature",
        "values": []
      }],
      "id": "Tank1",
      "isPattern": false,
      "type": "Tank"
    },
    "statusCode": {
      "code": "200",
      "reasonPhrase": "OK"
    }
  }]
}

The STH-Comet log shows that it is online and connects to the correct database:

time=2020-07-09T22:39:06.698Z | lvl=INFO | corr=n/a | trans=n/a | op=OPER_STH_DB_CONN_OPEN | from=n/a | srv=n/a | subsrv=n/a | comp=STH | msg=Establishing connection to the database at mongodb://@mongo:27017/sth_tese
time=2020-07-09T22:39:06.879Z | lvl=INFO | corr=n/a | trans=n/a | op=OPER_STH_DB_CONN_OPEN | from=n/a | srv=n/a | subsrv=n/a | comp=STH | msg=Connection successfully established to the database at mongodb://@mongo:27017/sth_tese
time=2020-07-09T22:39:07.218Z | lvl=INFO | corr=n/a | trans=n/a | op=OPER_STH_SERVER_START | from=n/a | srv=n/a | subsrv=n/a | comp=STH | msg=Server started at http://0.0.0.0:8666

The STH-Comet log with the api v2 request:

time=2020-07-09T23:46:47.400Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=GET /STH/v2/entities/Tank1/attrs/temperature?type=Tank&lastN=10
time=2020-07-09T23:46:47.404Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=Getting access to the raw data collection for retrieval...
time=2020-07-09T23:46:47.408Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=The raw data collection for retrieval exists
time=2020-07-09T23:46:47.412Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=No raw data available for the request: /STH/v2/entities/Tank1/attrs/temperature?type=Tank&lastN=10
time=2020-07-09T23:46:47.412Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=Responding with no points

According to the log, it establishes the connection to recover the raw data: msg=Getting access to the raw data collection for retrieval.... Confirms that the raw data exists: msg=The raw data collection for retrieval exists. But, it cannot recover this data and generates the message that the raw data is not available and does not return any points:msg=No raw data available for the request and msg=Responding with no points.

I already read the configuration part in the documentation. I've reinstalled everything, several times. I combed all settings and I can't find anything to justify this problem.

What could it be? Could someone with expertise in STH-Comet give any guidance?

Thanks!

Question: if you do the query based in date range (instead of using lastN), how it behaves? Does it work or the problem is the same? — fgalan, Jun 11 '20 at 18:06
Yes, it is the same I used: `http://sth:8666/STH/v1/contextEntities/type/Tank/id/Tank1/attributes/pressure?hLimit=3&hOffset=0&dateFrom=2020-06-11T13:00:00.000Z&dateTo=2020-06-11T15:59:59.999Z` returned: `{ "contextResponses": [{ "contextElement": { "attributes": [{ "name": "pressure", "values": [] }], "id": "Tank1", "isPattern": false, "type": "Tank" }, "statusCode": { "code": "200", "reasonPhrase": "OK" } }] } ` — Rosiberto, Jun 14 '20 at 16:21
Which fiware-service and fiware-servicepath headers did you used in your STH requests? — fgalan, Jun 16 '20 at 06:30
@fgalan **Fiware-service: iot** and **Fiware-servicepath: /** — Rosiberto, Jun 16 '20 at 15:34
A final piece of information that is missing in the question and that I think could add some light to the problem... Which STH configuration are you using? config.js, env vars, etc. Have you deployed using docker-compose.yml (in that case the docker-compose.yml file could include the relevant env vars)? Please edit your question to include that information. — fgalan, Jun 30 '20 at 14:43
@fgalan Yes, I am using a docker-compose. To env: `environment: - STH_HOST=0.0.0.0 - STH_PORT=8666 - DB_URI=mongo:27017 - DB_USERNAME= - DB_PASSWORD= - LOGOPS_LEVEL=DEBUG ` In config.js I enabled CORS. — Rosiberto, Jul 03 '20 at 00:43
Apart from enabling CORS, which other settings are you using in your config.js file? Please, don't answer in a new comment. Comments are not the best place to provide useful information. It's better to edit your question post and include that information (along with the one you have mentioned about docker-compose environment in the comment just above). Thanks! — fgalan, Jul 03 '20 at 13:55
@fgalan I put more information on the post as per your guidance. — Rosiberto, Jul 06 '20 at 16:16
With regards to config.js file, instead of telling what you have modified (about CORS and default service), could you edit your question post and provide the full content of the file, please? That way is clearer. Thanks! — fgalan, Jul 15 '20 at 11:17
@fgalan, I updated the post and included the contents of the config.js file, following your guidelines. — Rosiberto, Jul 16 '20 at 14:40

score 0 · Answer 1 · answered Jul 20 '20 at 09:03

0

Sometimes the way in which STH tries to recover information doesn't match to the way in wich Cygnus store it. However, that doesn't to be the case here. The datamodel used by STH is configured with config.database.dataModel and it seems to be correct: collection-per-entity (as you have collections like sth_/iot_Tank1_Tank, which correspondds to a single entity, i.e. the one with id Tank1 and type Tank).

Assuming that the setting in config.js is not being overridden by DATA_MODEL env var (although it would be wise to check that, looking to the env vars actuallly inyected to the docker container running STH, I guess that with docker inspect) the only way I think we can continue debugging is to inspect which actual query does STH on MongoDB to end in No raw data available for the request.

MongoDB has a profiler that allows to record every query done in the DB. Thus the procedure would be as follows:

Avoid (or minimize) any other usage of MongoDB instance, to avoid "noise" in the information recorded by the profiler
Start the profiler in "all queries" mode (i.e. profiling level 2)
Do the query at STH API
Stop the profiler
Check the queries recorded by the profiler as a consequence of the request done in step 3

Explaining the usage of the MongoDB profiler is out of the scope of this answer, but the reference I provided above is a good starting point if you don't know it already.

Once you have information about the queries, please provide feedback as comments to this answers. Thanks!

answered Jul 20 '20 at 09:03

fgalan

11,732
9
46
89

`"executionTimeMillisEstimate" : 0, "works" : 4, "advanced" : 0, "needTime" : 3, "needYield" : 0, "saveState" : 0, "restoreState" : 0, "isEOF" : 1, "invalidates" : 0, "direction" : "forward", "docsExamined" : 2 } } } }, "ts" : ISODate("2020-07-28T17:14:34.632Z"), "client" : "172.27.0.7", "allUsers" : [ ], "user" : "" }` – Rosiberto Jul 28 '20 at 17:44
the summary of the steps you suggested. – Rosiberto Jul 28 '20 at 22:31
I'm afraid is a bit hard to read... I'd suggest providing your feedback using some tool like https://pastebin.com, (including the link to your document as a comment here. Try to do your report in a structured manner (i.e. "When I do this request X I get this Y in the profiler") and use some prettier tool (e.g. https://jsonlint.com) to format the JSON fragments. Thanks! – fgalan Jul 30 '20 at 09:51
Hello @fgalan, Every STH configuration has been verified and the DATA_MODEL is correct. The command: docker inspect sth was executed in the STH-Comet container. The result can be seen at: [https://pastebin.com/w4bgUGZR]. – Rosiberto Aug 02 '20 at 21:54
MongoDB was configured for the type 2 profile with the command: db.setProfilingLevel (2). Then, a query was made at STH-Comet using api v1. Then the profiler was stopped. The result of the consultation can be seen at: [https://pastebin.com/i6SJP9Bh]. – Rosiberto Aug 02 '20 at 22:00
The complete log of the STH-Comet container is seen at: [https://pastebin.com/Hsc7EGC8]. This is the summary of all executions carried out with their respective results following your @fgalan guidance. – Rosiberto Aug 02 '20 at 22:04
I was having a look to the above pastebin.com links, but all them result in 404 Not Found error... – fgalan Jan 04 '21 at 12:16
Related issues at gihub: https://github.com/telefonicaid/fiware-sth-comet/issues/545 and https://github.com/telefonicaid/fiware-sth-comet/issues/519 – fgalan Jan 25 '21 at 09:13

Problem when querying Raw Data with STH-Comet - Returns empty

request with v2:

Return

request with v1:

Return

1 Answers1