Django/Python: CSV for-in loop overriding first row each time through

Question

class CSVDownload(View):
    """ Prepares CSV file version to download """       

        #more code here

        f = StringIO.StringIO()
        writer = csv.writer(f, dialect='excel')
        for v in visit_list:
            writer.writerow([v.idfa.idfa, v.name, v.duration, v.firstSeen, v.lastSeen, v.identifier, v.closestProximity])
        f.seek(0)
        response = HttpResponse(f, content_type='text/csv')
        response['Content-Disposition'] = 'attachment; filename=AdvertiserData.csv'

        return response

For some reason, when opening the file in excel, the file will only output the last item in the list, see here

This leads me to believe that each row is overriding the first row. Although this shouldn't be the case. Look at these tests I have preformed in terminal.

>>> f.getvalue()
'FFAC6F6C-1B2E-47C2-8110-5E619B239FB1,PPTest1,00:00:24,2014-11-11 23:20:24.730000,2014-11-11 23:20:48.750000,nkfe-cnb7s,NEAR\r\nFFAC6F6C-1B2E-47C2-8110-5E619B239FB1,moo2,00:00:24,2014-11-11 23:20:24.730000,2014-11-11 23:20:48.750000,nkfe-cnb7s,NEAR\r\n'

This returns more than one value, with \r\n between the values.

Also, I tried

>>> print response
Content-Type: text/csv
Content-Disposition: attachment; filename=boo.csv

FFAC6F6C-1B2E-47C2-8110-5E619B239FB1,PPTest1,00:00:24,2014-11-11 23:20:24.730000,2014-11-11 23:20:48.750000,nkfe-cnb7s,NEAR
FFAC6F6C-1B2E-47C2-8110-5E619B239FB1,moo2,00:00:24,2014-11-11 23:20:24.730000,2014-11-11 23:20:48.750000,nkfe-cnb7s,NEAR

And that also showed more than one value.

I saw recently a SO thread here that discusses adding to a file here:How do you append to a file? But since I am creating this item in memory only (using stringIO), how can I get the same effect?

Thanks!

EDIT For extra information:

The visit_list is compromised of visit models:

class Visit(models.Model):
    idfa = models.ForeignKey(Report) 
    name = models.CharField(max_length=255)
    lastSeen = models.CharField(max_length=255)
    duration = models.CharField(max_length=255)
    firstSeen = models.CharField(max_length=255)
    identifier = models.CharField(max_length=255)
    closestProximity = models.CharField(max_length=255)

    def __unicode__(self):
        return self.name

    class Meta:
        verbose_name = "Visit"
        verbose_name_plural = "Visits"

In this particular case, visit_list returns:

>>> visit_list
[<Visit: PPTest1>, <Visit: moo2>]

What does your `visit_list` look like? – Justin O Barber Nov 26 '14 at 02:16 — Justin O Barber, Nov 26 '14 at 02:16

Justin O Barber · Accepted Answer · 2014-11-26T04:58:49.213

Bottom Line: I suspect you need to pass f.getvalue() to HttpResponse rather than f.

The Explanation:

HttpResponse prefers strings, although it can also be treated like a file object or accept iterators. Nevertheless, f is a StringIO.StringIO instance and not a string per se.

Consider the following setup:

import StringIO
import csv

>>> f = StringIO.StringIO()
>>> writer = csv.writer(f, dialect='excel')
>>> row = range(4)                           # first row is [0, 1, 2, 3]
>>> for i in range(5):
        writer.writerow([row])               # write more rows
        row = [x + 1 for x in row]

>>> f.seek(0)

Now compare f with the result of f.getvalue():

# an instance
>>> f
<StringIO.StringIO instance at 0x03656990>

# a string
>>> f.getvalue()
'"[0, 1, 2, 3]"\r\n"[1, 2, 3, 4]"\r\n"[2, 3, 4, 5]"\r\n"[3, 4, 5, 6]"\r\n"[4, 5, 6, 7]"\r\n'

Now note the difference in your response object when you pass the instance versus the string to HttpResponse:

# The instance
>>> response = HttpResponse(f, content_type='text/csv')  # f is an instance
>>> response['Content-Disposition'] = 'attachment; filename=AdvertiserData.csv'
>>> response._container
['']

# The string
>>> response = HttpResponse(f.getvalue(), content_type='text/csv')  # f.getvalue() returns a string
>>> response['Content-Disposition'] = 'attachment; filename=AdvertiserData.csv'
>>> response._container
['"[0, 1, 2, 3]"\r\n"[1, 2, 3, 4]"\r\n"[2, 3, 4, 5]"\r\n"[3, 4, 5, 6]"\r\n"[4, 5 , 6, 7]"\r\n']

Note that the response's _container attribute is empty when you pass the instance to HttpResponse but not empty when you pass the f.getvalue() string to HttpResponse.

I would try this instead, then:

>>> response = HttpResponse(f.getvalue(), content_type='text/csv')
>>> # etc.

Beautiful explanation, and answer worked. Although something interesting happened. I ended up running my old code(returning just f) after running it through a PEP8 check for indentation/tab check, and it seemed to work fine. I am still abit confused about how that happened because it is literally the same code. even github revisions couldn't tell the difference. Nevertheless, your answer works and I think is a better way to return f. Thanks Justin! — ApathyBear, Nov 26 '14 at 04:58
@ApathyBear How interesting. I worked with your code for a while and eventually found these differences. I never received any errors that suggested tabs would be an issue, but that may certainly be the case! In any case, I'm glad you got it working now! — Justin O Barber, Nov 26 '14 at 05:00

Django/Python: CSV for-in loop overriding first row each time through

1 Answers1