43

I have some cvs data that has an empty column at the end of each row. I would like to leave it out of the import or alternatively delete it after import. My cvs data's have a varying number of columns. I've tried using df.tail(), but haven't managed to choose the last column with it.

employment=pd.read_csv('./data/spanish/employment1976-1987thousands.csv',index_col=0,header=[7,8],encoding='latin-1')

Data:

4.- Resultados provinciales
Encuesta de Población Activa. Principales Resultados

Activos por provincia y grupo de edad (4).
Unidades:miles de personas


,Álava,,,,Albacete,,,,Alicante,,,,Almería,,,,Asturias,,,,Ávila,,,,Badajoz,,,,Balears (Illes),,,,Barcelona,,,,Burgos,,,,Cáceres,,,,Cádiz,,,,Cantabria,,,,Castellón de la Plana,,,,Ciudad Real,,,,Córdoba,,,,Coruña (A),,,,Cuenca,,,,Girona,,,,Granada,,,,Guadalajara,,,,Guipúzcoa,,,,Huelva,,,,Huesca,,,,Jaén,,,,León,,,,Lleida,,,,Lugo,,,,Madrid,,,,Málaga,,,,Murcia,,,,Navarra,,,,Orense,,,,Palencia,,,,Palmas (Las),,,,Pontevedra,,,,Rioja (La),,,,Salamanca,,,,Santa Cruz de Tenerife,,,,Segovia,,,,Sevilla,,,,Soria,,,,Tarragona,,,,Teruel,,,,Toledo,,,,Valencia,,,,Valladolid,,,,Vizcaya,,,,Zamora,,,,Zaragoza,,,,Ceuta y Melilla,,,,
,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,
1976TIII,"8.9","11.6","60.4","11.8","16.4","14.4","65.2","14.9","47.9","49.9","246.0","60.1","20.5","14.3","88.9","11.2","34.5","42.5","278.0","91.3","6.6","7.2","41.5","13.3","25.3","22.8","135.3","37.5","19.8","24.4","153.0","43.0","166.8","203.7","1079.0","230.7","14.1","16.4","86.0","23.8","17.0","18.3","86.6","28.6","31.0","38.7","180.4","29.8","15.3","19.2","120.6","30.4","19.9","15.3","104.2","23.4","19.7","19.5","97.5","29.7","28.0","23.9","140.5","30.1","29.1","46.1","263.8","70.0","8.9","6.2","45.7","14.6","19.7","19.7","123.0","35.3","26.8","22.5","141.0","36.2","4.8","6.0","33.1","13.4","23.1","31.6","174.5","33.8","11.9","14.3","83.8","18.8","7.0","9.3","50.3","20.0","22.4","23.4","125.8","28.6","22.7","21.6","143.1","50.9","12.5","13.7","89.5","33.2","14.3","14.7","134.0","54.7","136.6","207.5","1067.6","218.6","34.7","41.1","196.4","38.4","37.2","35.0","200.5","46.1","15.6","23.8","111.6","30.7","14.0","16.8","120.2","74.9","5.7","6.4","39.2","8.0","24.5","25.6","135.3","27.1","36.4","39.4","246.1","74.0","10.2","11.3","63.9","13.4","10.5","11.0","74.1","19.6","19.3","23.9","140.3","31.7","5.5","6.0","35.6","11.3","55.2","55.6","262.5","68.1","3.1","3.2","24.4","5.4","21.8","18.4","116.7","37.1","4.6","3.4","37.3","12.0","20.3","16.7","102.2","23.1","73.5","85.5","454.6","101.5","19.2","23.4","90.7","20.5","41.3","54.7","272.2","57.0","6.0","7.1","56.5","28.9","29.2","32.1","192.7","49.8","0.0","0.0","0.0","0.0",
1976TIV,"8.7","11.7","60.8","11.4","14.4","13.6","63.3","14.5","49.1","50.6","244.9","54.2","19.0","16.9","86.8","11.4","33.2","42.3","271.8","86.0","5.8","7.5","40.3","13.9","25.1","24.7","132.7","38.4","18.8","23.4","151.8","43.9","172.2","201.7","1070.7","228.1","11.1","15.7","82.5","21.1","16.4","18.0","89.2","26.6","32.6","40.0","176.5","30.5","15.8","18.1","121.3","30.2","19.0","17.3","106.3","24.1","19.9","19.0","101.7","26.9","25.3","22.3","142.7","28.9","30.0","42.4","267.6","70.1","7.3","7.0","44.4","13.0","17.8","21.4","122.8","34.0","28.4","21.6","140.5","36.8","4.7","6.6","32.6","10.8","24.8","32.7","177.2","32.3","11.9","12.5","85.4","20.5","6.9","8.5","48.8","19.9","22.4","22.1","127.6","25.1","18.5","21.1","137.8","48.7","12.4","11.1","84.9","31.5","13.6","15.6","132.7","52.0","144.0","202.3","1054.0","222.5","35.6","40.1","194.1","37.5","36.7","34.7","203.8","47.1","15.6","23.6","114.3","31.3","14.0","15.9","118.3","76.7","5.5","7.3","36.9","9.3","25.5","25.1","138.7","26.8","34.8","42.9","250.3","74.9","9.9","11.8","62.8","14.0","10.0","13.2","74.5","19.2","19.5","24.2","142.7","31.0","4.0","5.9","35.5","12.0","55.0","56.7","264.7","63.3","2.8","3.5","23.9","5.1","20.0","21.6","116.4","34.9","4.5","3.7","36.5","12.1","21.1","17.6","100.6","25.7","74.6","87.5","455.5","102.1","18.9","22.9","90.0","21.6","40.2","57.1","273.9","58.5","5.6","8.3","57.6","23.9","28.3","31.4","192.2","46.4","0.0","0.0","0.0","0.0",
1977TI,"9.2","11.8","59.9","11.2","14.2","13.2","65.9","14.7","48.2","50.4","251.1","50.8","17.8","15.4","86.5","11.8","30.6","42.9","272.6","84.1","5.8","7.4","37.2","12.8","24.1","22.8","131.3","38.2","17.8","23.5","151.1","42.5","168.1","200.4","1077.2","223.3","11.6","12.8","80.9","17.6","14.4","16.4","88.2","23.9","34.5","37.5","176.3","30.8","15.2","19.7","121.3","31.6","18.4","19.4","107.4","24.7","20.0","18.1","98.3","26.6","24.9","23.6","150.7","27.5","29.5","40.3","267.4","70.5","5.6","7.5","44.2","12.8","17.1","21.1","122.8","33.6","29.6","23.3","142.1","37.9","4.6","5.5","33.7","11.2","23.5","30.4","175.2","32.8","12.0","12.7","84.8","21.3","7.3","9.3","46.6","17.8","30.2","26.0","147.1","25.2","15.9","22.7","133.2","45.1","12.8","12.1","84.3","28.0","12.4","16.5","131.2","55.6","150.9","202.9","1065.4","223.7","36.6","44.0","194.3","39.9","36.7","31.5","196.7","45.7","14.8","22.5","115.1","29.4","11.7","17.2","114.2","75.8","5.0","7.7","38.0","9.4","24.0","26.8","143.5","27.0","35.3","43.0","247.4","73.5","9.7","12.1","61.6","13.3","9.5","11.9","73.9","18.9","20.4","26.7","143.0","31.6","4.0","5.0","35.5","12.3","52.3","58.0","266.0","62.5","2.6","2.7","24.2","6.0","17.3","21.0","113.0","33.3","4.5","5.2","33.8","10.6","18.7","18.8","98.3","24.8","77.4","87.6","446.6","100.3","20.5","23.4","90.2","20.4","38.7","50.7","277.6","57.3","6.4","8.7","60.1","21.5","28.6","31.0","194.8","45.7","0.0","0.0","0.0","0.0",
Artturi Björk
  • 3,643
  • 6
  • 27
  • 35

8 Answers8

93

Here's a one-liner that does not require specifying the column name

df.drop(df.columns[len(df.columns)-1], axis=1, inplace=True)
conner.xyz
  • 6,273
  • 8
  • 39
  • 65
  • 1
    This should be marked as the answer as it permorms automation when reading different files! – M.K Jul 01 '19 at 13:32
  • to be more explicit, add that 'df' is the name of the dataframe, so here 'employment', giving employment.drop(employment.columns[len(employment.columns)-1], axis=1, inplace=True) ;) – sol Oct 05 '21 at 08:26
  • `df.columns[len(df.columns)-1]` could just be `df.columns[-1]` – Marcin Dec 02 '21 at 13:54
67

Another method to delete last column in DataFrame df:

df = df.iloc[:, :-1]

Gusev Slava
  • 2,136
  • 3
  • 21
  • 26
  • 3
    This just creates a view. It's not actually deleting/dropping anything in the source dataframe. – conner.xyz Feb 14 '17 at 16:16
  • 1
    @conner.xyz Can you elaborate, why this is not a valid answer? Thanks! – tommy.carstensen Jan 14 '18 at 07:02
  • It returns a view of a data frame, which *still* contains the last column. It's not actually updating the reference to a new underlying data structure that does not contain the last column (i.e. a copy). See: http://pandas.pydata.org/pandas-docs/version/0.20/indexing.html#returning-a-view-versus-a-copy https://stackoverflow.com/questions/23296282/what-rules-does-pandas-use-to-generate-a-view-vs-a-copy – conner.xyz Jan 18 '18 at 00:43
19

You can specify which columns to import using usecols parameter for read_csv

So either create a list of column names or integer values:

cols_to_use = ['col1', 'col2'] # or [0,1,2,3]
df = pd.read_csv('mycsv.csv', usecols= cols_to_use)

or drop the column after importing, I prefer the former method (why import data you are not interested in?).

df = df.drop(labels='column_to_delete', axis=1) # axis 1 drops columns, 0 will drop rows that match index value in labels

Note also you misunderstand what tail does, it returns the last n rows (default is 5) of a dataframe.

Additional

If the columns are varying length then you can just the header to get the columns and then read the csv again properly and drop the last column:

def df_from_csv(path):
    df = read_csv(path, nrows=1) # read just first line for columns
    columns = df.columns.tolist() # get the columns
    cols_to_use = columns[:len(columns)-1] # drop the last one
    df = read_csv(path, usecols=cols_to_use)
    return df
EdChum
  • 376,765
  • 198
  • 813
  • 562
  • I would like to drop the last column of the cvs, but the cvs's are of varying column lenghts. If I understood correctly your method would entail first checking the column lenght of the cvs. I would like the program to do this for me. – Artturi Björk Dec 12 '13 at 08:56
  • 2
    Droping by label maybe has the same problem as the label of the column will be different depending on the column lenght. Is it possible to drop based on location? As a workaround I'm currently using this `a=len(df.columns)-1 df=df.iloc[:,:a]` – Artturi Björk Dec 12 '13 at 08:58
  • Also I very much agree with your sentiment of not importing data that I'm not interested in. – Artturi Björk Dec 12 '13 at 09:02
  • @ArtturiBjörk you could read just a couple lines to get the columns, then re-read the csv again and pass the columns less the last one to `usecols` – EdChum Dec 12 '13 at 09:17
15

Improve from @conner.xyz answer above:

df.drop(df.columns[[-1,]], axis=1, inplace=True)

If you want to delete the last two columns, replace [-1,] by [-1, -2].

Nelson Dinh
  • 465
  • 6
  • 14
12

Another way to remove the last column:

df = df[df.columns[:-1]]
AidinZadeh
  • 724
  • 1
  • 8
  • 15
4

After importing the data you could drop the last column whatever it is with:

employment = employment.drop(columns=employment.columns[-1])
william_grisaitis
  • 5,170
  • 3
  • 33
  • 40
Diego BV
  • 53
  • 4
3

As with all index based operations in Python, you can use -1 to start from the end.

df.drop(df.columns[-1], axis=1, inplace=True)

Bhushan
  • 2,256
  • 3
  • 22
  • 28
0

Just to complete the accepted answer, if you have one dataframe with only two columns, and these two columns have the same name, be aware. First you need to rename one column and then drop the desirable column.

*Edited after comment below

Douglas Flores
  • 100
  • 4