Unable to process JSON input when ingesting logs to Oracle

Question

Error:

    send: b'{"specversion": "1.0", "logEntryBatches": [{"entries": [{"data": "{\\"hello\\": \\"oracle\\", \\"as\\": \\"aaa\\"}", "id": "ocid1.test.oc1..jkhjkhh23423fd", "time": "2021-04-01T12:19:28.416000Z"}], "source": "EXAMPLE-source-Value", "type": "remediationLogs", "defaultlogentrytime": "2021-04-01T12:19:28.416000Z"}]}'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Date: Fri, 02 Apr 2021 07:39:16 GMT
header: opc-request-id: ER6S6HDVTNWUOKCJ7XXZ/OpcRequestIdExample/770899C2C7CA6ABA11D996CC57E8EE8F
header: Content-Type: application/json
header: Connection: close
header: Content-Length: 79
Traceback (most recent call last):
  File "tool.py", line 45, in <module>
    put_logs_response = loggingingestion_client.put_logs(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/oci/loggingingestion/logging_client.py", line 172, in put_logs
    return self.base_client.call_api(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/oci/base_client.py", line 276, in call_api
    response = self.request(request)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/oci/base_client.py", line 388, in request
    self.raise_service_error(request, response)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/oci/base_client.py", line 553, in raise_service_error
    raise exceptions.ServiceError(
oci.exceptions.ServiceError: {'opc-request-id': 'ER6S6HDVTNWUOKCJ7XXZ/OpcRequestIdExample/770899C2C7CA6ABA11D996CC57E8EE8F', 'code': 'InvalidParameter', 'message': 'Unable to process JSON input', 'status': 400}

I am trying to send json data to Oracle logs, but getting the above error. I am using json.dumps(data) to convert the dict to string. Kindly let me know if any workaround available to this.

Code:

data = {'hello':'oracle', "as":"aaa"}
put_logs_response = loggingingestion_client.put_logs(
        log_id="ocid1.log.oc1.iad.<<Log OCID>>",
        put_logs_details=oci.loggingingestion.models.PutLogsDetails(
            specversion="1.0",
            log_entry_batches=[
                oci.loggingingestion.models.LogEntryBatch(
                    entries=[
                        oci.loggingingestion.models.LogEntry(
                            data= json.dumps(data),
                            id="ocid1.test.oc1..jkhjkhh23423fd",
                            time=datetime.strptime(
                                "2021-04-01T12:19:28.416Z",
                                "%Y-%m-%dT%H:%M:%S.%fZ"))],
                    source="EXAMPLE-source-Value",
                    type="Logs",
                    defaultlogentrytime=datetime.strptime(
                        "2021-04-01T12:19:28.416Z",
                        "%Y-%m-%dT%H:%M:%S.%fZ"))]),
        timestamp_opc_agent_processing=datetime.strptime(
            "2021-04-01T12:19:28.416Z",
            "%Y-%m-%dT%H:%M:%S.%fZ"),
        opc_agent_version="EXAMPLE-opcAgentVersion-Value",
        opc_request_id="ER6S6HDVTNWUOKCJ7XXZ/OpcRequestIdExample/")

score 3 · Answer 1 · answered Apr 05 '21 at 18:36

Your data looks fine , I think the issue is that Time precision is more than millisec. it should work fine if you loose the trailing zeros in time.
Format a datetime into a string with milliseconds

https://docs.oracle.com/en-us/iaas/api/#/en/logging-dataplane/20200831/LogEntry/

Life is complex · Accepted Answer · 2021-04-06T22:10:39.913

This exception indicates that you have an InvalidParameter in your JSON input.

oci.exceptions.ServiceError: {'opc-request-id': 'ER6S6HDVTNWUOKCJ7XXZ/OpcRequestIdExample/770899C2C7CA6ABA11D996CC57E8EE8F', 'code': 'InvalidParameter', 'message': 'Unable to process JSON input', 'status': 400}

The InvalidParameter is your timestamp, which is date - 2021-04-01T12:19:28.416Z.

According to Oracle's documentation you need to use a RFC3339-formatted date-time string with milliseconds precision when creating a LogEntry.

This code snippet is from oci-python-sdk - log_entry.py, but it doesn't mention the milliseconds precision like Oracle's documentation.

@time.setter
    def time(self, time):
        """
        Sets the time of this LogEntry.
        Optional. The timestamp associated with the log entry. An RFC3339-formatted date-time string.
        If unspecified, defaults to PutLogsDetails.defaultlogentrytime.
        :param time: The time of this LogEntry.
        :type: datetime
        """
        self._time = time

This code create a UTC RFC3339 complaint timestamp with milliseconds precision

from datetime import datetime
from datetime import timezone

current_utc_time_with_offset = datetime.now(timezone.utc).isoformat()
print(current_utc_time_with_offset)
#output 
2021-04-06T13:00:52.706040+00:00

current_utc_time_with_timezone = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
print(current_utc_time_with_timezone)
#output 
2021-04-06T13:09:10.053432Z

This Stack Overflow question is worth a read:

What's the difference between ISO 8601 and RFC 3339 Date Formats?

This article is also useful:

Understanding about RFC 3339 for Datetime and Timezone Formatting in Software Engineering

I altered my code to send utc time with millisecond precision, but no luck. It still gives me same error. I converted time to datetime object with strptime, as Oracle only accepts datetime objects. This also didn't worked. — Dushyant Bhardwaj, Apr 07 '21 at 11:53
Where is the documentation on this statement "Oracle only accepts datetime objects?" I don't see this requirement in the oci loggingingestion documentation or within the code. The documentation/code that I have read says "RFC3339 formatted datetime string." — Life is complex, Apr 08 '21 at 13:40
Yeah, realized a little late. In the example It was using datetime.strptime, and that was resulting in trailing zeros. Issue resolved with strings — Dushyant Bhardwaj, Apr 15 '21 at 10:47

score 1 · Answer 3 · answered Apr 06 '21 at 17:45

Your time is RFC3339 but precision is more than millisec

'{"specversion": "1.0", "logEntryBatches": [{"entries": [{"data": "{\"hello\": \"oracle\", \"as\": \"aaa\"}", "id": "ocid1.test.oc1..jkhjkhh23423fd", "time": "2021-04-01T12:19:28.416000Z"}], "source": "EXAMPLE-source-Value", "type": "remediationLogs", "defaultlogentrytime": "2021-04-01T12:19:28.416000Z"}]}'

See https://docs.oracle.com/en-us/iaas/api/#/en/logging-dataplane/20200831/LogEntry/

The timestamp associated with the log entry. An RFC3339-formatted date-time string with milliseconds precision. If unspecified, defaults to PutLogsDetails.defaultlogentrytime.

I am giving time 2021-04-01T12:19:28.416Z, but when i strptime to make it a datetime.datetime object It adds those trailing zeros. Any workaround this? — Dushyant Bhardwaj, Apr 07 '21 at 12:01

Unable to process JSON input when ingesting logs to Oracle

3 Answers3