CSV not importing to django models

Question

I'm trying to create a django web app that uses data from CSV's. I've created a django model to house the data from these csv's, I've stored all the csv's in STATIC_ROOT. When I load the page, the called template (datalandingpage.html) loads with no problem. However, when I check django admin, none of the CSVs have been imported. If I were to guess, I think the shortcoming has something to do with django not being able to find the files (though I'm just guessing, I could very well be wrong since I'm a relatively new developer). Below is all the relevant code I've written as an attempt.

Edit: So it looks like my view function may be skipping straight to the render part, which may be why I don't see any errors when it runs. Any ideas as to why this may be the case?

settings.py:

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.11/howto/static-files/

STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static/data')

models.py:

from __future__ import unicode_literals
from django.db import models
from django.contrib import admin

class Data(models.Model):
     # A bunch of model objects (CharField/IntegerFields mainly)

    class Meta:
        verbose_name_plural = 'Data Sets'

    def __str__(self):
        return self.someobject

views.py:

from __future__ import unicode_literals
from django.conf import settings
from django.shortcuts import render, render_to_response
from django.template import RequestContext
from .models import Data
import csv
import os

def landing(request):

# Opens all csv files in specified directory
directory = os.path.join(settings.STATIC_ROOT)
for files in os.walk(directory):
    for file in files:
        if file.endswith(".csv"):
            f = open(file, 'r')
            reader = csv.reader(f)

            #Checks the rows/columns and tries to find the specified string that marks the start of data
            for i, row in enumerate(reader):
                target_string = "Some string in the CSV"
                if target_string in row:
                    target_row = i
                    return target_row
                    break

            #Checks the rows/columns and tries to find the specified string that marks the end of data
            for j, row in enumerate(reader):
                end_target_string = "Another string in the CSV"
                if end_target_string in row:
                    end_target_row = j
                    return end_target_row
                    break

            #Begins parsing the csv, but skips until two rows past the target beginning string
            for k, row in enumerate(reader):

                #imports csv data when the row number is between target string and target end string
                if k >= (target_row + 2):
                    row = row.split(',')
                    DataSource = Data.objects.create()
                    DataSource.someobject1 = row[0]
                    DataSource.someobject2 = row[1]
                    DataSource.save()

                # When the parse gets to the row with the end string, it closes the file.
                elif k >= end_target_row:
                    reader.close()
                    break

        # Prints something if the file isn't a csv file
        else:
            print "This isn't a csv file"

return render(request, "Data/datalandingpage.html")

Answer 1

I ended up re-writing the script and it worked. I think my strange looping structure caused the script to not work as intended (or at all, really). I also used bulk_create instead of having to call save after every row to make the process more efficient. Here's the version that works:

views.py:

from __future__ import unicode_literals
from django.conf import settings
from django.shortcuts import render, render_to_response
from django.template import RequestContext
from .models import Data
import os
import csv, sys


def import_data(request):

    # Opens all csv files in specified directory

    directory = os.path.join('Some Target Directory')
    count = 0
    for root, dirs, files in os.walk(directory):
        for filename in files:
            if filename.endswith(".csv"):
                file_path = os.path.join(root, filename)
                with open(file_path) as f:
                     reader = csv.reader(f)

                    #Checks the rows/columns and tries to find the specified string that marks the start of data

                    for i, row in enumerate(reader):
                        target_string = "Some Target String"
                        if target_string in row:

                        #Skips to the next line (where the data starts) when the target string is found

                            true_target_row = next(reader)
                            for row in reader:

                                #Counter tells you which file number is importing. 

                                print "importing file number %d" %(count)

                                #Creates a bunch of objects

                                objects = [
                                    Data(
                                        SomeObject = row [n],
                                        ...
                                        ...
                                )
                                for row in reader if row[0] != "Some Target End String"
                            ]

                            # Does a bulk_create after the file is completed

                            Data.objects.bulk_create(objects)
                            count += 1

            # Prints something if the file isn't a csv file

            else:
                print "This isn't a csv file"

    print "Imported %d files! All done!" %(count)

    return render(request, "Data/DataLandingPage.html")

Answer 2

If I understand correctly, you want to create a list of model from a CSV file. You might consider using an existing library for this: http://django-import-export.readthedocs.io/en/stable/index.html

Answer 3

You have an issue with how you're looping over the directories and opening the files. os.walk yields for each directory walked a tuple of (root_dir, directories, files) where directories is a tuple of directory names that are found in root_dir and files is a tuple of file names found in root_dir

In your code you have the following:

for files in os.walk(directory):
    for file in files:
        if file.endswith(".csv"):
            f = open(file, 'r')
            reader = csv.reader(f)

This would not do what you probably think it does and will undoubtedly result in an AttributeError because files in for files in os.walk(...) will always have two tuples in it and tuples do not have a .endswith method.

Furthermore, when you correctly loop over files provided by os.walk -- they are just file names. You have to join the path with the root directory.

To fix, you should modify this as follows....

for root, dirs, files in os.walk(directory):
    for filename in files:
        if filename.endswith(".csv"):
            file_path = os.path.join(root, filename)
            with open(file_path) as f:
                reader = csv.reader(f)

Notice that this unpacks the tuples yielded by os.walk into root, dirs, files now files is actually a list of file names. Additionally, os.path.join(root, filename) is used to create the full path to the file from the name.

Also note I modified this to use the with open(...) context manager. Code under this block executes with the file opened and once the block ends the file will be closed for you.