Getting the columns messed up after pd.concat Pandas

Question

I am trying to concatenate some dataframes but I getting the wrong order of columns after done.

My code is:

def numOfDays(date1, date2): 
return (date2-date1).days

first_case_report = datetime.strptime('22-01-2020', '%d-%m-%Y')
NumOfdays_reported = numOfDays(first_case_report, datetime.today())

column_names = ['Province/State','Country/Region','Last Update','Confirmed','Deaths','Recovered']
df = pd.DataFrame(columns = column_names)
df.to_csv(index=True)

df.head()
Output:
    Province/State  Country/Region  Last Update Confirmed   Deaths  Recovered

ind = 0
while ind < NumOfdays_reported:
    date_report = (pd.Timestamp(first_case_report) + pd.DateOffset(days=ind)).strftime('%m-%d-%Y')
    url = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/{0}.csv".format(date_report)
    source = pd.read_csv(url,index_col=0,parse_dates=[0])

    df = pd.concat([df,source], sort=True)
    ind += 1   

df.head()
Output:
Confirmed   Country/Region  Deaths  Last Update Latitude    Longitude   Province/State  Recovered

The last df.head() shows that the columns are messed up, for example compare the column Province/State with the df.head() executed, why is that happening?

Any ideas would be highly appreciated.

Thanks a lot beforehand.

your sample code doesnt work for me : `NameError: name 'NumOfdays_reported' is not defined` , please provide an mcve — anky, Mar 18 '20 at 17:11
Please have a look at [How to make good pandas examples](https://stackoverflow.com/questions/20109391/how-to-make-good-reproducible-pandas-examples) and provide sample input and output. "The columns are messed up" doesn't tell us enough to understand the issue — G. Anderson, Mar 18 '20 at 17:17
I think might be clearer now.I also edited the code @anky_91 — Vssmm, Mar 18 '20 at 17:33
Just to be clear, your concern isn't that the data is in the wrong columns, but that the columns are in the wrong order? If that's the case, then you can reorder your columns however you want and this would be a duplicate of [How to change order of dataframe columns](https://stackoverflow.com/questions/13148429/how-to-change-the-order-of-dataframe-columns) — G. Anderson, Mar 18 '20 at 18:50
This is very difficult to read, the output is seemingly mixed in alongside the code, and you're not following naming conventions. As an aside, concatenating DataFrames inside a loop is almost never the right choice. It's best to create a temporary data structure and concatenate everything at once after the loop. — AMC, Mar 18 '20 at 19:20
Does this answer your question? [How to change the order of DataFrame columns?](https://stackoverflow.com/questions/13148429/how-to-change-the-order-of-dataframe-columns) — AMC, Mar 18 '20 at 19:21

score 0 · Answer 1 · answered Mar 18 '20 at 19:06

0

solved.

I have just created a empty dataframe, even without columns, and then when it concatenate at the right way.

Code:

df = pd.DataFrame()

answered Mar 18 '20 at 19:06

Vssmm

51
7

Getting the columns messed up after pd.concat Pandas

1 Answers1