loading data from a csv file into a Django model

Question

I have a simple yet frustrating problem that I can't seem to figure out.

I'm trying to load data from a csv file into a Django model. To do so I've written the following script as a view:

import csv
def import_db(request):
    dataReader = csv.reader(open('/home/<name>/webapps/<name2>/employees.csv'), delimiter=',', quotechar='"')
    for row in dataReader:
        emp = Employee()
        emp.first_name = row[0]
        emp.last_name = row[1]
        emp.email = row[2]
        emp.level = row[3]
        emp.service_area = row[4]
        emp.service_line = row[5]
        emp.office = row[6]
        emp.save()
return HttpResponse("Completed", content_type="text/plain")

I've linked the view to a url as follows:

from reviews import views as emp  
url(r'^load/$',   emp.import_db, name='importdb')

The idea being that when I go to the link sitename.com/load, my data will get loaded from my employee.csv file into my Employee model.

The problem is when I run this script, I get 2 entries in my Django model for every line in my csv file. I have 1530 employee lines in the csv and the model gets populated with 3060 instances when I do this. What's even more annoying is that the order of the entries in the model is not the same as the csv file so I can't simply delete the second 'group' of 1530 model instances. Even when I try it with a subset of 20 lines of data in the csv file I get 40 model instances. Any idea why this is happening and what i can do to fix it?

Thanks a lot!

What is Employee() ? Is it your model? – Tms91 Oct 10 '19 at 14:34 — Tms91, Oct 10 '19 at 14:34

score 6 · Accepted Answer · edited Oct 10 '19 at 15:22

Performing this logic in a view function is not a good idea. I don't know exactly why in your case, but it is possible to trigger the same view function twice (e.g., browsers like Google Chrome often prefetch URLs in the background as soon as you paste them into the address bar - this can cause two hits on that view. I don't know whether that is the problem here but it may be worth ruling out).

You should consider moving this logic into a custom management command that you can call deterministically. Something like:

# myapp/management/commands/import_csv.py
from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):

    def add_arguments(self, parser):
        parser.add_argument('csv_file', nargs='+', type=str)

    def handle(self, *args, **options):
        for csv_file in options['csv_file']:
            dataReader = csv.reader(open(csv_file), delimiter=',', quotechar='"')
            for row in dataReader:
                emp = Employee()
                # etc...
                self.stdout.write(
                    'Created employee {} {}'.format(emp.first_name, emp.last_name)
                )

You would then call this with:

./manage.py import_csv --csvfile "/home/<name>/webapps/<name2>/employees.csv"

This approach gives you more control over what is going on and will make it easier to debug the issue (if there still is one).

Thanks for your answer, I tryed to implement it but it led me to "line 20, in handle dataReader = csv.reader(open(csv_file), delimiter=',', quotechar='"') FileNotFoundError: [Errno 2] No such file or directory: 'csv_file' " . what's wrong with it? See here for more details: https://stackoverflow.com/a/58326339/7658051 — Tms91, Oct 11 '19 at 14:18

score 1 · Answer 2 · answered Aug 08 '16 at 04:45

1

Where you are opening your file, check number of rows do something like this:

    reader = csv.reader(open('/home/<name>/webapps/<name2>/employees.csv'), delimiter=',', quotechar='"')
    data = list(reader)
    print len(data) # check this

answered Aug 08 '16 at 04:45

Kishan Mehta

2,598
5
39
61

score 0 · Answer 3 · answered Oct 15 '19 at 10:49

If you want only your administrator to be able to import files, but not your app users, you can use django-import-export. It's a widget that allows you to import excel files from admin section.

Read the docs!
Here you find the installation tutorial, and here you find an expample.

It's very easy to install and it's more user-friendly than implementing a custom management command. This latter can be useful if you want to "standardize"/"pre-format" the data before inserting it in the data (like uniforming the characters ' " '', or e' è é, or à a', or ò o', ecc.)

loading data from a csv file into a Django model

3 Answers3

Linked