How to read and validate header in csv uploaded in s3 in aws lambda python function

Question

I need to upload CSV data to MySQL. To achieve this, I have utilized the AWS Lambda service. I am reading a CSV file from an S3 bucket in the Lambda function, which uses Python2.7. I want to validate the header of the CSV with values which I inserted in Environment Variables in the Lambda console. However, on printing the data, I am getting '\r' at the end of the value of the last column of each row.

I am able to read the data from CSV, and the values are also getting inserted to the MySQL DB.

def validateCSV(event,context):
    EXPECTED_HEADERS=os.environ['RM_EXPECTED_HEADERS']
    s3 = boto3.client("s3")
    file_obj = event["Records"][0]
    bucketname = str(file_obj['s3']['bucket']['name'])
    filename = str(file_obj['s3']['object']['key'])
    fileObj = s3.get_object(Bucket=bucketname, Key=filename)
    rows = fileObj["Body"].read().split('\n')
    print(rows)//(['Name,Age,PinCode\r', 'Apple,15,411001\r',''])
    fList=[]
    for line in rows:
       fList.append(line.split(','))
    print("fList Headers matched: ",fList[0]==EXPECTED_HEADERS)//this is giving me FALSE

I have added value in environment variable --key=RM_EXPECTED_HEADERS and value as ['Name','Age','PinCode']. But when I am printing fList[0], I am getting (['Name', 'Age', 'PinCode\r']).

How do I remove '\r' from fList[0]?

score 1 · Accepted Answer · answered Mar 30 '19 at 16:51

1

You can strip the \r manually from your rows:

  rows = [x.strip() for x in fileObj["Body"].read().split('\n'))

Alternativly include them into your split(...):

  rows = [x.strip() for x in fileObj["Body"].read().split('\r\n'))

I never had problems with remaining \r - normally python takes care of eiter \n (Linux) or \r\n (Windows) - problems might occure if you create textfiles using windows and splitting them under unix - not sure.

answered Mar 30 '19 at 16:51

Patrick Artner

50,409
9
43
69

Thanks @Patrick. I am able to remove \r using split('\r\n') but still I am getting FALSE when comparing both the lists fList[0]==EXPECTED_HEADERS. – AWS_Developer Mar 30 '19 at 18:10
@rumit [How to debug small programs](https://ericlippert.com/2014/03/05/how-to-debug-small-programs/) - print the content of both, make sure there are no invisible whitespaces inside your variables. `print("----|",rows[0].split(","),"|---",sep="")` same for your environment thing. See [https://stackoverflow.com/questions/9623114/check-if-two-unordered-lists-are-equal](https://stackoverflow.com/questions/9623114/check-if-two-unordered-lists-are-equal) – Patrick Artner Mar 30 '19 at 18:26

How to read and validate header in csv uploaded in s3 in aws lambda python function

1 Answers1