1

I have the following JSON document being sent from an SNS HTTPs subscription to an API Gateway endpoint (backed by a Python2.7 Lambda function). In this case, I have a Cloudwatch Alarm that's configured to send to SNS, which then sends to this Gateway endpoint. SNS packages that alarm (which is JSON) into the "body" field of the message it sends to Gateway.

I need to extract the JSON document out of the "body" field, but by this point it's been properly mangled with escape characters and new lines and json.loads() is not liking it at all.

How can I read the value of "body" back into a JSON document using Python2.7 within a Lambda function? I've tried cleaning it by removing '\n' and '\', but I'm just striking out!

Here is the JSON as received by Lambda:

{
"body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"944c9xxx3-c98d636ff2c7\",\n  \"TopicArn\" : \"arn:aws:sns:us-west-2:xxx6xx:sxxxr-sns-topic\",\n  \"Subject\" : \"ALARM: \\\"hhh\\\" in US West (Oregon)\",\n  \"Message\" : \"{\\\"AlarmName\\\":\\\"hhh\\\",\\\"AlarmDescription\\\":null,\\\"AWSAccountId\\\":\\\"8xxx\\\",\\\"NewStateValue\\\":\\\"ALARM\\\",\\\"NewStateReason\\\":\\\"Threshold Crossed: 1 out of the last 1 datapoints [0.333370380661336 (13/06/18 18:06:00)] was greater than or equal to the threshold (0.1) (minimum 1 datapoint for OK -> ALARM transition).\\\",\\\"StateChangeTime\\\":\\\"2018-06-13T18:16:56.457+0000\\\",\\\"Region\\\":\\\"US West (Oregon)\\\",\\\"OldStateValue\\\":\\\"INSUFFICIENT_DATA\\\",\\\"Trigger\\\":{\\\"MetricName\\\":\\\"CPUUtilization\\\",\\\"Namespace\\\":\\\"AWS/EC2\\\",\\\"StatisticType\\\":\\\"Statistic\\\",\\\"Statistic\\\":\\\"AVERAGE\\\",\\\"Unit\\\":null,\\\"Dimensions\\\":[{\\\"name\\\":\\\"InstanceId\\\",\\\"value\\\":\\\"i-07bxxx26\\\"}],\\\"Period\\\":300,\\\"EvaluationPeriods\\\":1,\\\"ComparisonOperator\\\":\\\"GreaterThanOrEqualToThreshold\\\",\\\"Threshold\\\":0.1,\\\"TreatMissingData\\\":\\\"\\\",\\\"EvaluateLowSampleCountPercentile\\\":\\\"\\\"}}\",\n  \"Timestamp\" : \"2018-06-13T18:16:56.486Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"fFunXkjjxxxvF7Kmxxx\",\n  \"SigningCertURL\" : \"https://sns.us-west-2.amazonaws.com/SimpleNotificationService-xxx.pem\",\n  \"UnsubscribeURL\" : \"https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=axxxd\"\n}",
"resource": "/message",
"requestContext": {
    "requestTime": "13/Jun/2018:18:16:56 +0000",
    "protocol": "HTTP/1.1",
    "resourceId": "m4sxxxq",
    "apiId": "2v2cthhh",
    "resourcePath": "/message",
    "httpMethod": "POST",
    "requestId": "f41e8-8cbd-57ad9e625d12",
    "extendedRequestId": "xxx",
    "path": "/stage/message",
    "stage": "stage",
    "requestTimeEpoch": 1528913816627,
    "identity": {
        "userArn": null,
        "cognitoAuthenticationType": null,
        "accessKey": null,
        "caller": null,
        "userAgent": "Amazon Simple Notification Service Agent",
        "user": null,
        "cognitoIdentityPoolId": null,
        "cognitoIdentityId": null,
        "cognitoAuthenticationProvider": null,
        "sourceIp": "xxx",
        "accountId": null
    },
    "accountId": "xxx"
},
"queryStringParameters": {
    "id": "CBxxx69"
},
"httpMethod": "POST",
"pathParameters": null,
"headers": {
    "Content-Type": "text/plain; charset=UTF-8",
    "Via": "1.1 xxx.cloudfront.net (CloudFront)",
    "Accept-Encoding": "gzip,deflate",
    "CloudFront-Is-SmartTV-Viewer": "false",
    "x-amz-sns-subscription-arn": "arn:aws:sns:us-west-2:xxx:sxxx-nxxx-sns-topic:xxx",
    "CloudFront-Forwarded-Proto": "https",
    "X-Forwarded-For": "54.240.xxx, 54.182.xxx",
    "CloudFront-Viewer-Country": "US",
    "User-Agent": "Amazon Simple Notification Service Agent",
    "X-Amzn-Trace-Id": "Root=1-5b21xxx53acea6642317ed4",
    "x-amz-sns-topic-arn": "arn:aws:sns:us-west-2:xxxx:sxxxier-sns-topic",
    "Host": "2vxxx.execute-api.us-west-2.amazonaws.com",
    "X-Forwarded-Proto": "https",
    "X-Amz-Cf-Id": "xxx",
    "CloudFront-Is-Tablet-Viewer": "false",
    "X-Forwarded-Port": "443",
    "x-amz-sns-message-type": "Notification",
    "CloudFront-Is-Mobile-Viewer": "false",
    "x-amz-sns-message-id": "xxx",
    "CloudFront-Is-Desktop-Viewer": "true"
},
"stageVariables": null,
"path": "/message",
"isBase64Encoded": false
}
user212869
  • 11
  • 2
  • You can't just keep removing the garbage until it's clean? Like: body.replace('\"', '').replace('\n', '')...... – sehafoc Jun 13 '18 at 21:25
  • Check out this one, I think you need to convert the string first , then you an do normal json decoding: https://stackoverflow.com/questions/24242433/python-how-to-convert-a-raw-string-into-a-normal-string (maybe this answer is best: https://stackoverflow.com/a/24242596/399696). There's a python 3 version as well. – dmgig Jun 13 '18 at 21:27
  • @sehafoc you should never try to modify JSON-in-JSON with string replacement manipulation. That isn't garbage. – Michael - sqlbot Jun 13 '18 at 23:36
  • ["event – AWS Lambda uses this parameter to pass in event data to the handler. This parameter is usually of the Python dict type"](https://docs.aws.amazon.com/lambda/latest/dg/python-programming-model-handler-types.html). You said below that the output in your question is from `json.dumps(event)`, isn't `event` already a Python dict? – G_M Jun 14 '18 at 04:35
  • 1
    @Michael-sqlbot That's sound advice, The body JSON section looks like it's be grabbed as an output of a pretty printer or something, which would violate JSON rules. Though in my tests everything json.loads() cleanly anyways. Which lead me to believe there is something else there. – sehafoc Jun 15 '18 at 21:00
  • @sehafoc this is just a normal SNS event notification with multiple layers of JSON encoding. A surprising number of people are stumped by this. Cloudwatch creates the message, sends it to SNS, which expects *text*, not JSON (so it's stringified = pass 1). SNS sends an event notification, which is wraps the text into a JSON wrapper (pass 2). API Gateway passes the body to Lambda, inside a JSON object (pass 3). Peeling back the layers with `json.loads()` creates the correct objects. Pseudocode: parse(parse(parse(event).body).Message) gives you the inner message. – Michael - sqlbot Jun 15 '18 at 23:07
  • ...but since event is already parsed from JSON by Lambda, it's just parse(parse(event.body).Message). – Michael - sqlbot Jun 15 '18 at 23:08

1 Answers1

1

if i use your pasted sample as a raw string, it works well:

>>> j = r'''...your sample pasted here...'''
>>> data = json.loads(j)
>>> bodydata = json.loads(data['body'])
>>> bodydata['Type']
u'Notification'

seems, that what you pasted above is the repr form, printed out with Python

powo
  • 460
  • 1
  • 6
  • 17
  • Yeah, what I pasted was the json.dumps(event) output. Printing out event directly ports it out as unicode. Whenever I execute what you have, I get `ValueError: Expecting : delimiter: line 42 column 27 (char 2808)`. How would you suggest I convert the input variable to a raw string in the script? Thanks! – user212869 Jun 14 '18 at 02:14
  • Since `event` is already a python dict, all you may need is `json.loads(event['body'])` – powo Jun 14 '18 at 06:23