Unable to write item(s) to DynamoDB table utilizing DocumentClient - Nodejs

Question

I'm absolutely brand new to DynamoDb and I'm trying to simply write an object from a NodeJS Lambda. Based on what I've read and researched I should probably be using DocumentClient from the aws-sdk. I also found the following question here regarding issues with DocumentClient, but it doesn't seem to address my specific issue....which I can't really find/pinpoint unfortunately. I've set up a debugger to help with SAM local development, but it appears to be only providing some of the errors.

The code's implementation is shown here.

var params = {
     TableName: "March-Madness-Teams",
     Item: {
        "Id": {"S": randstring.generate(9)},
        "School":{"S": team_name},
        "Seed": {"S": seed},
        "ESPN_Id": {"S": espn_id}
      }
    }
  console.log(JSON.stringify(params))

   dynamodb.put(params, (error,data) => {
      if (error) {
         console.log("Error ", error)
      } else {
         console.log("Success! ", data)
      }
   })

Basically I'm scrubbing a website utilizing cheerio library and cherry picking values from the DOM and saving them into the json object shown below.

{
  "TableName": "March-Madness-Teams",
  "Item": {
    "Id": {
      "S": "ED311Oi3N"
    },
    "School": {
      "S": "BAYLOR"
    },
    "Seed": {
      "S": "1"
    },
    "ESPN_Id": {
      "S": "239"
    }
  }
}

When I attempt to push this json object to Dynamo, I get errors says

Error  MultipleValidationErrors: There were 2 validation errors:
* MissingRequiredParameter: Missing required key 'TableName' in params
* MissingRequiredParameter: Missing required key 'Item' in params

The above error is all good in well....I assume it didn't like the fact that I had wrapped those to keys in strings, so I removed the quotes and sent the following

{
  TableName: "March-Madness-Teams",
  Item: {
    "Id": {
      "S": "ED311Oi3N"
    },
    "School": {
      "S": "BAYLOR"
    },
    "Seed": {
      "S": "1"
    },
    "ESPN_Id": {
      "S": "239"
    }
  }
}

However, when I do that...I kind of get nothing.

Here is a larger code snippet.

return new Promise((resolve,reject) => {
            axios.get('http://www.espn.com/mens-college-basketball/bracketology')
            .then(html => {

                const dynamodb = new aws.DynamoDB.DocumentClient()

                let $ = cheerio.load(html.data)
                $('.region').each(async function(index, element){
                    var preregion = $(element).children('h3,b').text()
                    var region = preregion.substr(0, preregion.indexOf('(') - 1)

                    $(element).find('a').each(async function(index2, element2){
                        var seed = $(element2).siblings('span.rank').text()
                    if (seed.length > 2){
                        seed = $(element2).siblings('span.rank').text().substring(0, 2)
                    }

                    var espn_id = $(element2).attr('href').split('/').slice(-2)[0]
                    var team_name = $(element2).text()
                    var params = {
                        TableName: "March-Madness-Teams",
                        Item: {
                            "Id": randstring.generate(9),
                            "School":team_name,
                            "Seed": seed,
                            "ESPN_Id": espn_id
                        }
                    }
                    console.log(JSON.stringify(params))

                    // dynamodb.put(params)
                    //     .then(function(data) {
                    //         console.log(`Success`, data)
                    //     })
                })
              })
            })
        })

It seem fine, just check again your debug point and aws document example https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStarted.NodeJs.03.html#GettingStarted.NodeJs.03.01 — hoangdv, Feb 23 '20 at 16:46
Can you show more of the code? Based on one of your comments it sounds like you may be combining async with callbacks. — Jason Wadsworth, Feb 23 '20 at 17:35
There you go, @JasonWadsworth. Edited the original post with more of the source code. — RoflWaffle17, Feb 24 '20 at 03:32
it's just a lambda function right now but will be supporting a webpage. — RoflWaffle17, Feb 26 '20 at 02:02

score 0 · Answer 1 · answered Feb 23 '20 at 16:30

0

Can you try without the type?

Instead of

    "School":{"S": team_name},

for example, use

    "School": team_name,

answered Feb 23 '20 at 16:30

Horatiu Jeflea

7,256
6
38
67

No luck.....when debugging it goes to where I start to invoke the put API, but when I try to walk through, it just skips the callback entirely. Even when I have breakpoints throught the entire dynamodb api section of code :( – RoflWaffle17 Feb 23 '20 at 16:46

score 0 · Answer 2 · answered Feb 28 '20 at 06:21

0

From your code, I can see the mis promise on the dynamodb request. Try to change your lines :

dynamodb.put(params).then(function(data) {
    console.log(`Success`, data)
})

to be :

dynamodb.put(params).promise().then(function(data) {
    console.log(`Success`, data)
})

you can combine with await too :

await dynamodb.put(params).promise().then(function(data) {
    console.log(`Success`, data)
})

answered Feb 28 '20 at 06:21

Mahdi Ridho

264
1
8

Like Jason was saying I was missing some serious awaits...I took some time and restructured a lot of the code to make it more readable and potentially make sense. I will update post to reflect the changes and accept it. If you or anyone sees something that could/should change just let me know! – RoflWaffle17 Feb 29 '20 at 02:01

score 0 · Accepted Answer · answered Feb 29 '20 at 02:10

exports.lambdaHandler = async (event, context) => {
    const html = await axios.get('http://www.espn.com/mens-college-basketball/bracketology')
    let $ = cheerio.load(html.data)
    const schools = buildCompleteSchoolObject(html, $)
    try {
        await writeSchoolsToDynamo(schools)
        return { statusCode: 200 }
    } catch (error) {
        return { statusCode: 400, message: error.message }
    }
}

const writeSchoolsToDynamo = async (schools) => {
        const promises = schools.map(async school => {
            await dynamodb.put(school).promise()
        })

        await Promise.all(promises)
}

const buildCompleteSchoolObject = (html, $) => {
    const schools = []

    $('.region').each(loopThroughSubRegions(schools, $))
    return schools
}
const loopThroughSubRegions = (schools, $) => {
    return (index, element) => {
        var preregion = $(element).children('h3,b').text()
        var region = preregion.substr(0, preregion.indexOf('(') - 1)
        $(element).find('a').each(populateSchoolObjects(schools, $))
    }
}
const populateSchoolObjects = (schools, $) => {
    return (index, element) => {
        var seed = $(element).siblings('span.rank').text()
        if (seed.length > 2) {
            seed = $(element).siblings('span.rank').text().substring(0, 2)
        }
        var espn_id = $(element).attr('href').split('/').slice(-2)[0]
        var team_name = $(element).text()
        schools.push({
            TableName: "March-Madness-Teams",
            Item: {
                "Id": randstring.generate(9),
                "School": team_name,
                "Seed": seed,
                "ESPN_Id": espn_id
            }
        })
    }
}

I know this is drastically different from what I started with but I did some more digging and kind of kind of worked to this...I'm not sure if this is the best way, but I seemed to get it to work...Let me know if something should change!

score 0 · Answer 4 · answered Mar 01 '20 at 05:06

Oh I understand what you want.

Maybe you can see the code above works, but there is one concept you have to improve here about async - await and promise especially on lambda function.

I have some notes here from your code above, maybe can be your consideration to improve your lambda :

Using await for every promise in lambda is not the best approach because we know the lambda time limitation. But sometimes we can do that for other case.
Maybe you can change the dynamodb.put method to be dynamodb.batchWriteItem :

The BatchWriteItem operation puts or deletes multiple items in one or more tables.

Or If you have to use dynamodb.put instead, try to get improve the code to be like so :

    const writeSchoolsToDynamo = async (schools) => {
        const promises = schools.map(school => {
            dynamodb.put(school).promise()
        })

        return Promise.all(promises)
    }

Hey Mahdi, I attempted your code snippet, however it did not work. The function completed successfully, however none of the data was actually written to the table....thoughts? — RoflWaffle17, Mar 02 '20 at 00:26
Honestly, I didn't test my code above. I just deliver an idea run asynchronous/parallel multiple process (here you run multiple dynamodb.put). In your code you run synchronous process through await. it means if we have 5 put need 1s each, your code execute them serialize and needs 5s. By parallel, they can be processed avg +- 1s for 5 put. Maybe you would try change my code : const promises = schools.map(school => { dynamodb.put(school).promise() }) to be const promises = schools.map(school => { return dynamodb.put(school).promise() }) — Mahdi Ridho, Mar 03 '20 at 02:08

Unable to write item(s) to DynamoDB table utilizing DocumentClient - Nodejs

4 Answers4