How to insert from csv to dynamodb

Question

Dynamo db have one table employees primary key as id

data.csv is below which uploaded in the csvdynamo bucket

bucket_name = csvdynamo

id,name,co
20,AB,PC
21,CD,PC
22,EF,MC
23,GH,MC

Need to insert above csv into dynamodb

psuedo code

for emp in employees:
    emp_data= emp.split(',')
    print (emp_data)

    try:
        table.put_item(
            Item = {
                "emp_id": int(emp_data[0]),
                "Name": emp_data[1],
                "Company": emp_data[2]
            }
        )
    except Exception as e:
          pass

"Full code which is not working" - can you please clarify what do you mean? Do you get any error messages, timeouts, access denies ...? — Marcin, Jun 30 '20 at 05:02
@Marcin any alternate way without calling event['Records'] will be helpful. Deleting full code as it is unnecessary — aysh, Jun 30 '20 at 05:07
Sorry, hadn't had time to look into it. I will try a bit later today. Meantime, maybe someone else will be able to assist. — Marcin, Jun 30 '20 at 06:54

score 6 · Accepted Answer · answered Jun 30 '20 at 08:40

Here is an example of a lambda function which works, as I verified it using my own function, csv file and dynamodb. I think the code is self-explanatory. It should be a good start towards your end use-case.

          import boto3
          import json
          import os

          bucket_name = os.environ['BUCKET_NAME']
          csv_key = os.environ['CSV_KEY_NAME'] # csvdynamo.csv
          table_name = os.environ['DDB_TABLE_NAME']
          
          # temprorary file to store csv downloaded from s3
          tmp_csv_file = '/tmp/' + csv_key

          s3 = boto3.resource('s3')
          db_table = boto3.resource('dynamodb').Table(table_name)

          def save_to_dynamodb(id, name, co):

            return db_table.put_item(
                Item={
                  'emp_id': int(id),
                  'Name': name,
                  'Company': co
                })

          def lambda_handler(event, context):

              s3.meta.client.download_file(
                            bucket_name, 
                            csv_key, 
                            tmp_csv_file)

              with open(tmp_csv_file, 'r') as f:

                next(f) # skip header

                for line in f:

                  id, name, co = line.rstrip().split(',')

                  result = save_to_dynamodb(id, name, co)

                  print(result)

              return {'statusCode': 200}

can you check once https://stackoverflow.com/questions/63351210/how-to-extract-the-elements-from-csv-to-json-in-s3 — aysh, Aug 11 '20 at 04:16

score 1 · Answer 2 · answered Jun 30 '20 at 08:37

DynamoDB needs to be told the type of field being inserted.

From put_item(), the format for Item is:

    Item={
        'string': {
            'S': 'string',
            'N': 'string',
            'B': b'bytes',
            'SS': [
                'string',
            ],
            'NS': [
                'string',
            ],
            'BS': [
                b'bytes',
            ],
            'M': {
                'string': {'... recursive ...'}
            },
            'L': [
                {'... recursive ...'},
            ],
            'NULL': True|False,
            'BOOL': True|False
        }
    },

Therefore, you would need something like:

        table.put_item(
            Item = {
                "emp_id":  {'N': emp_data[0]},
                "Name":    {'S': emp_data[1]},
                "Company": {'S': emp_data[2]}
            }
        )

can you answer https://stackoverflow.com/questions/63618932/how-to-read-content-from-the-s3-bucket-as-url — aysh, Aug 27 '20 at 15:29

score 0 · Answer 3 · answered May 15 '23 at 21:27

Here is a script for those who just want to import a csv file that is locally on their computer to a DynamoDB table. (I just took the script from @Marcin and modified it a little bit, leaving out the S3 bucket code and making it more general.)

import boto3


csv_file_path = "your-csv-file.csv"
table_name = "your-dynamodb-table-name"
db_table = boto3.resource('dynamodb').Table(table_name)
line_seperator = ';'


def save_to_dynamodb(column_names, values):
    item = dict()

    for idx, column_name in enumerate(column_names):
        item[column_name.lower()] = values[idx]

    return db_table.put_item(
        Item=item
    )


def handler():

    with open(csv_file_path, 'r', encoding='utf-8-sig') as f:

        column_names = next(f).strip("\n").split(line_seperator)

        for line in f:
            values = line.strip("\n").split(line_seperator)

            result = save_to_dynamodb(column_names, values)

            print(result)

    print("FINISHED IMPORT")


handler()

How to insert from csv to dynamodb

3 Answers3