How to skip header of CSV file while appending it to another CSV file using python

Question

I am receiving a CSV response from API. I want to append the received CSV response to already present CSV file in my machine. Problem i am facing is it is also appending the header from second CSV file. I want to remove that header. I have also attached screenshot of how my csv looks after appending.

screenshot

click here to see the screenshot screenshot of the response recieved I am trying this code

response = requests.get(Link)
actual_file = glob.glob(path_to_data+'\\Data\\*')
new_target_file = path_to_data+'\\Data'+'\\'+State+'_'+date+'_'+St_Id+'.csv'
# Write to .CSV
if not os.path.exists(new_target_file):
   for x in response.text.split('\n')[1:]:
       f = open(actual_file[0], "a")
       f.write(x)
   f.close()
   os.rename(actual_file[0],new_target_file)
else: 
   logging.warning("File already exist")

There isn't an obvious direct problem. Have you tried fixing the f = open() line? It does not belong in your loop. You might want to do `appenedFile = actual_file[0]; with open(appendedFile) as f: for x ....`. — Charles Merriam, Jun 19 '17 at 00:54
Please provide an extract of the first several lines of a sample response (file). Although you have used slicing [1:] to skip the first line of response, you might need to skip more than one line of response. — ChuckCottrill, Jun 19 '17 at 01:06
@CharlesMerriam I have tried your approach . It is removing the header but it only appending 1 row to the file . here is the code `response = requests.get(Link) actual_file = glob.glob(path_to_data+'\\Data\\*') new_target_file = path_to_data+'\\Data'+'\\'+State+'_'+date+'_'+St_Id+'.csv' appendedfile = actual_file[0] # Write to .CSV if not os.path.exists(new_target_file): with open(appendedfile, "a")as f: for x in response.text.split('\n')[1:]: f.write(x)` — PriyalChaudhari, Jun 19 '17 at 01:08
@ChuckCottrill I am uploading the screenshot in question as there is no space here — PriyalChaudhari, Jun 19 '17 at 01:14
Maybe there's a newline at the beginning of the response text - try `response.text.strip().split("\n")[1:]` instead. — zwer, Jun 19 '17 at 01:25
@zwer tried it. its still appending only 1 row. It is skipping the header but only appending 1 row to the file — PriyalChaudhari, Jun 19 '17 at 01:31
@PriyalChaudhari, yes, but that line should then contain all the data... use `f.write(x + "\n")` to bring it back to multiple rows (`str.split()` removed them) — zwer, Jun 19 '17 at 01:34
@zwer Wll thank you that worked. `f.write(x + "\n")` solved the problem . — PriyalChaudhari, Jun 19 '17 at 01:50
Your supplied screen capture shows header row, so it wasn't skipped by slicing. The response has something in [0] (which you skip), and header in [1] - this is not unusual, so you probably need something that recognizes all of the header possibilities. — ChuckCottrill, Jun 19 '17 at 01:57
@ChuckCottrill that screen capture was before i tried `f.write(x + "\n")`. — PriyalChaudhari, Jun 19 '17 at 02:24

score 0 · Accepted Answer · answered Jun 19 '17 at 01:48

The problem you have is recognizing the response, which lines to skip, and which lines to retain. You have implemented slicing to extract response lines after the first line, assuming that the actual content starts on the second line. Based upon your described symptoms, this is not (always) the case.

#fully describe header here,
header="STATION,STATION_ELEVATION,LATITUDE,LONGITUDE,..."

def isheader(line,header,delim=','):
    l = line.split(delim) #may want to fold case
    h = header.split(delim) #may want to fold case
    n = sum(list(set(l) & set(h)))
    return n==len(h)

response = requests.get(Link)
actual_file = glob.glob(path_to_data+'\\Data\\*')
new_target_file = path_to_data+'\\Data'+'\\'+State+'_'+date+'_'+St_Id+'.csv'
# Write to .CSV
if not os.path.exists(new_target_file):
    with open(actual_file[0], "a") as f:
        for x in response.text.split('\n')[1:]:
            if len(x) < 2: continue #skip empty lines
            if isheader(x,header,','): continue #skip header
            f.write(x+'\n')
    #with performs close automatically
    os.rename(actual_file[0],new_target_file)
else:
    logging.warning("File already exist")

Take a look at this question for how to use with open, How to open a file using the open with statement

Here is an example of how to compare two lists (such as a header list to a row list), Comparing two lists in Python

`f.write(x+'\n')` this alone solved the problem in my code. I will try your approch as well. Thank you for response. — PriyalChaudhari, Jun 19 '17 at 01:53

How to skip header of CSV file while appending it to another CSV file using python

1 Answers1