Best way to read and parse log file from AWS s3 using python

Asked Mar 18 '22 at 07:53

Active Mar 18 '22 at 08:30

Viewed 944 times

From Sagemaker

log file is read like follows:

sess = boto3.Session(region_name)
s3client = sess.client("S3")
log_obj = s3client.get_object(Bucket=bucket_name, Key=log_file_path)
log_obj["Body"].read()

which looks like this:

b'hello world this is log file1 \n arguments passed to function are \n [a,b,c,d] \n and so on...'

It is in one large text, unlike logs I see from other SO questions...

If \n writes next string into new line it would be much more easy to read, is there a way to let python NOT ignore escape character?

Or if there is better way to parse log of text from S3, let me know.

edited Mar 18 '22 at 08:30

John Rotenstein

241,921
22
380
470

asked Mar 18 '22 at 07:53

haneulkim

4,406
9
38
80

Try `log_obj["Body"].read().decode()` – Marcin Mar 18 '22 at 08:04
Still the same. – haneulkim Mar 18 '22 at 08:14
1

Have you tried it with a `for` loop: [Python: read all text file lines in loop](https://stackoverflow.com/a/17949545/174777) – John Rotenstein Mar 18 '22 at 08:30
How do you know it does not respect `\n`? In your code you are not using `log_obj["Body"].read()` at all. What do you do with it? Save to a file? Print onto a console? – Marcin Mar 18 '22 at 08:55
@Marcin there is log_obj["Body"].read(), last line. I'm trying to read it in sagemaker since simply downloading it from s3 has broken letters when opened. – haneulkim Mar 18 '22 at 09:04

Best way to read and parse log file from AWS s3 using python

0 Answers0