Importing a Text File with dates, times and data points and plotting into Matplotlib

Question

I'm starting out as a Research Assistant at a lab at UMD and I am having trouble with the coding aspect of the work. I am running python 2.7.12 w/ Anaconda 4.2.0. I was given a text file with the task of reading it into python and graphing it using matplotlib. The text file is in this format

20170109 001203 379.00 22824.13 1.00

where the last two columns can be ignored, the first is the date, the second is the time in HH:MM:SS. and the third column is ppm (parts per million).

I have been able to read the data into python but have not figured out how to differentiate the first two columns as dates and times. I am thinking of doing something using datetime but am not sure what inputs I should use. From there I would like to plot the data with time (both yyyy MMM dddd and HH MM SS) on the x-axis, and ppm on the y-axis, using matplotlib through numpy.

Look at [csv](https://docs.python.org/3.5/library/csv.html)! — Lucas, Jan 21 '17 at 00:23
You’re looking for the `strptime` method—there’s an example of how to use it in [this question](http://stackoverflow.com/questions/33397107/python-parsing-date-with-strptime) and the [documentation](https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior). — Noah Witherspoon, Jan 21 '17 at 00:23

Filip Kubicz · Answer 1 · 2017-01-25T17:27:00.350

To obtain the interesting part of the input, you can slice the input string

input = '20170109 001203 379.00 22824.13 1.00'

input_date = input[:15]
print(input_date)

input_ppm = input[16:23]
print(input_ppm)

Later, use strptime function to parse a string into date with time.

import datetime
dt = datetime.datetime.strptime(input_date, '%Y%m%d %H%M%S')
print(dt)

Now you can use dt as x-axis points. To see how to use dates in Matplotlib, you can check this Matplotlib example.

Good luck!

EDIT To read the file with multiple lines, you can use readlines() which will create a list. You can loop over that list to extract each line and parse it to date & time.

Whole code now will look like this:

import datetime

with open('filename.txt') as f:
    content = f.readlines()
# content is now a list of text line strings

# remove whitespaces, e.g. newline character
content = [x.strip() for x in content] 

for input in content:
    input_date = input[:15]
    print(input_date)

    input_ppm = input[16:23]
    print(input_ppm)

    dt = datetime.datetime.strptime(input_date, '%Y%m%d %H%M%S')
    print(dt)

Thank you so much for this, this is a great start. The problem I'm running into now is that this works great for doing one line at a time but the text file contains multiple lines. Is there a way to do this in bulk so the entire file is read into this format? thanks in advance! — Sebastian Suarez, Jan 22 '17 at 00:57
@SebastianSuarez I edited the answer to add the information you asked for. Basically you read a file to a list of strings. Then you wrap previously used code with for loop. If it was helpful, you can sign it as an accepted answer — Filip Kubicz, Jan 25 '17 at 17:30

Michael Kaesy · Answer 2 · 2017-01-21T12:09:26.400

Assuming all the date strings are going to have the same format...

from datetime import datetime

input = "20170109 001203 379.00 22824.13 1.00"

list = input.split(" ") #Split the input into parts where blank space is the delimiter
date_and_time = str([' '.join(list[:2])]) #Merge the first item with the second and convert to string
#Insert all the white spaces we need to then convert to date time object
date_and_time = date_and_time[2:6]+' '+date_and_time[6:8]+' '+date_and_time[8:10]+' '+date_and_time[11:13]+' '+date_and_time[13:15]+' '+date_and_time[15:-2]

datetime_object = datetime.strptime(date_and_time, '%Y %m %d %H %M %S')
print (datetime_object)

The reason this is tricky is because you need to somehow differentiate between the different units of time when you do your string -> date conversion. Code is very crude but it should provide some insight to your issue.

score 0 · Answer 3 · answered Jan 21 '17 at 01:07

I would start by splitting up the text file:

text = '20170109 001203 379.00 22824.13 1.00'
texts = text.split(" ")
print(texts)

Then you could extract it bit by bit with date:

date = datetime.strptime(texts[0], '%Y%m%d')
print("The day is {}".format(date.day))
time = datetime.strptime(texts[1], '%H%M%S')
print("The minute is {}".format(time.minute))
ppm = texts[2]
print("ppm is {}".format(ppm))

If you get stuck in the plotting you should open up a new question. I recommend going here http://matplotlib.org/gallery.html and clicking on a plot you like. It will provide all the code you need.

Importing a Text File with dates, times and data points and plotting into Matplotlib

3 Answers3