Can't find working solution to csv import into django models

Question

This is messy, so bear with me.

As a prelude to importing a much larger file into my Django models, I wanted to start by using a csv import to create the foreign key field the models in the larger file rely on. This csv is exactly two lines, the header and the data for the one instance of the fk model.

This seems simple enough, and all the posts I read here on SO and elsewhere on the net make it seems dead simple. But not for me.

Here's what I've tried:

1. I wrote my own script, based on http://mitchfournier.com/2011/10/11/how-to-import-a-csv-or-tsv-file-into-a-django-model/ Then I found out I had to use Django.setup(). The original blog post I got this from didn't mention this. It may have been written before this requirement was added. In any event, I had to use Django.setup now because this was a 'standalone script'. I kept getting a slew of errors, like

(cannon)malikarumi@Tetuoan2:~/Projects/cannon$ python ./load_data_2.py Traceback (most recent call last): File "./load_data_2.py", line 3, in <module> import Django ImportError: No module named Django ImportError: No module named essell Traceback (most recent call last): File "./load_data_2.py", line 8, in <module> django.setup() NameError: name 'django' is not defined

2. I refactored the code to get rid of these problems, only to run into new ones.

This blog post has the fieldnames listed as rows by index number. So when I finally got it to run, I got:

File "/home/malikarumi/Projects/cannon/jamf/essell/management/commands/my_loaddata1.py", line 21, in handle jurisdiction.address = row[1] IndexError: list index out of range

Since I only had two rows, the header and data for one instance, it occurred to me that maybe I needed columns, not rows. But this is the way the original script had it. So that took me to the official docs on csv, https://docs.python.org/2/library/csv.html, where I was dismayed to see NO examples of using rows by index number.

Frustrated, I wondered about a custom management command...

3. A custom django management command (cannon)malikarumi@Tetuoan2:~/Projects/cannon/jamf$ python manage.py myloaddata3

`Traceback (most recent call last): ImportError: No module named cannon.jamf.essell.models'

I changed the import numerous times, but Python refused to recognize any of them. Apparently this is because it only imports once. I tried imp.reload but that did not work. I closed the terminal and re-opened, and that didn't work. I rebooted the computer AND THAT STILL DIDN'T WORK. btw, this script is based on Friendm1's answer in how to import csv data into django models

3. I looked at COPY in Postgres, but decided to pass on that because: a) it is outside the ORM, and b) the docs make it seem like it to refers to a database, and not just a table.

4. django-csvimport Warning to all of you in a similar position, there are a lot of similar sounding apps out there, but some have not been maintained in years. Django-csvimport has recent updates, but the documentation is pretty sparse. I tried to do exactly as it told me, and of course got errors. Then it occurred to me that maybe I should run startapp for this thing, even though the instructions said nothing about that. It might have silently assumed I would know that instead of relying on exactly what was written. But when I tried that, I got:

(cannon)malikarumi@Tetuoan2:~/Projects/cannon/jamf$ python manage.py startapp csvimport

CommandError: 'csvimport' conflicts with the name of an existing Python module and cannot be used as an app name. Please try another name. (cannon)malikarumi@Tetuoan2:~/Projects/cannon/jamf$ ls essell jamf manage.py Procfile README.md requirements.txt runtime.txt

Note I continued to get this error even after I renamed, and then deleted the csvimport folder I had created manually. I tried renaming it to csvimport2, but how will that affect other parts of the program? One reason I don't already know the answer to that question is because, now that I have this csvimport2 app, what do I put in models.py? Am I supposed to run migrations and make tables for csvimport2? Apparently, the docs do say to 'run syncdb'. But I got 'No migrations to apply.' even with csvimport2 in INSTALLED APPS.

So finally, I have turned to you for help.

Answer 1

I found the answer: csvkit. http://csvkit.readthedocs.org/en/latest/index.html

It does everything I need (convert csv to json) and more, it is well documented, and currently maintained.

Then I paired that with Django fixtures and the loaddata command https://docs.djangoproject.com/en/1.8/howto/initial-data/ and I am happily back in business with a greatly reduced stress level.

Look at the json example on the Django page carefully, and follow that example or you will get errors, but it DOES work, and what more can you ask for?