I am reading in some data which looks like this:
In this dataset, a number of rows have null in column 16. I need to shift the values in such rows to the right, so that the values which begin with "*" (eg. column 16 row 4, column 13 row 5 etc.) will move to the columns right of them. (Eventually I will do this in a loop so that those values will go into column 16) .
The data to the left of these values also have to move too. For example when the data in {column 7 row 16} moves to {column 8, row 16}, the data in {column 2 row 16} should move to {column 3 row 16}.
However, I do not want the data in column 1 (zero index column 0) to move as I will be using that as an index for my data.
Hence my expected output is this:
I am using the code below to achieve this:
import StringIO
import pandas
# Store the csv string in a variable and turn that into a dataframe
# This string here is the same as the data in the image above.
gps_string = """2010-01-12 18:00:00,$GPGGA,180439,7249.2150,N,11754.4238,W,2.0,10,0.9,-8.1,M,-12.4,M,,*57,,,
2010-01-12 17:30:00,$GPGGA,173439,7249.2160,N,11754.4233,W,2.0,11,0.8,-4.5,M,-12.4,M,,*5B,,,
2010-01-12 17:00:00,$GPGGA,170439,7249.2152,N,11754.4235,W,2.0,11,0.8,-3.1,M,-12.4,M,,*5C,,,
2010-01-12 16:30:00,N,11754.4210,W,2,9.0,1.1,-13.1,M,-12.4,M,,*6C,,,,,,
2010-01-12 16:00:00,N,11754.4229,W,2,10.0,0.9,-2.9,M,-12.4,M,,*53,,,,,,
2010-01-12 15:30:00,N,11754.4269,W,2,9.0,0.8,-4.3,M,-12.4,M,,*54,,,,,,
2010-01-12 15:00:00,N,11754.4267,W,2,10.0,0.8,-1.6,M,-12.4,M,,*56,,,,,,
2010-01-12 14:30:00,$GPGGA,143439,7249.2152,N,11754.4253,W,2.0,11,0.7,-4.3,M,-12.4,M,,*56,,,
2010-01-12 14:00:00,N,11754.4245,W,2,10.0,0.9,-7.0,M,-12.4,M,,*50,,,,,,
2010-01-12 13:30:00,$GPGGA,133439,7249.2134,N,11754.4243,W,2.0,11,0.7,-10.7,M,-12.4,M,,*61,,,
2010-01-12 13:00:00,N,11754.4245,W,2,10.0,0.8,-5.5,M,-12.4,M,,*56,,,,,,
2010-01-12 12:30:00,N,11754.4226,W,2,10.0,0.9,-7.1,M,-12.4,M,,*59,,,,,,
2010-01-12 12:00:00,N,11754.4238,W,2,10.0,0.8,-6.5,M,-12.4,M,,*51,,,,,,
2010-01-12 11:30:00,N,11754.4227,W,2,10.0,0.8,0.1,M,-12.4,M,,*73,,,,,,
2010-01-12 11:00:00,-7.4,M,-12.4,M,,*5F,,,,,,,,,,,,
2010-01-12 10:30:00,N,11754.4271,W,2,8.0,1.1,-8.4,M,-12.4,M,,*5A,,,,,,
"""
# Read the csv string into a dataframe, with no headers
# Make the first column with timestamp values the index column.
gps_df = pd.read_csv(StringIO.StringIO(gps_string), header=None,
index_col=0)
rows_to_shift = gps_df[gps_df[15].isnull()].index
# Shift the rows here.
gps_df.loc[rows_to_shift] = gps_df.loc[rows_to_shift].shift(periods=1, axis=1)
gps_df.to_csv("f.csv") # Creates a file after shift to see the output
I get the following output file when the code is executed.
From this I see that the shift function creates a column of null(s) at column 5 for some reason, and it also moves the data that was originally in column 10 into column 15, any idea why this might be the case?
Could this be a bug in the dataframe.shift() function? or am I doing something wrong here?
