Import into MongoDB/GridFS

Question

I am new to NoSQL and I am working on developing a parts&supplies database within the Grails framework using MongoDB as the supporting database. I am trying to do an automated initial bulk import of projects and I am looking for some ideas as to where to start.

I basically have a hard drive of data consisting of a ready folder and within that folder is x amount of projects and within each project folder is a CSV and 1-5 PNG's that correspond to that CSV. I can easily import a large CSV file - that is not what I am asking about. I am concerned with getting all of these nested files into the database in one motion. The images will have to go through GridFS as they exceed the 16MB limit. I know Mongo doesn't do well with nesting so I'm assuming I won't just be able to import all in one file and have it properly disperse/organize the data. Here is an outline:

Ready Folder
   -Project 1 
     -CSV
     -PNG's
   -Project 2 
     -CSV
     -PNG's
   -Project 3 
     -CSV
     -PNG's
   -ETC........

http://stackoverflow.com/questions/4686500/how-to-use-mongoimport-to-import-csv — cfrick, Jan 07 '15 at 09:58

score 0 · Answer 1 · answered Jan 07 '15 at 05:20

From my very limited experience, you will have to convert the csv file into a JSON or Groovy data file. MongoDB stores "objects" and a csv file is relational more-or-less while json or groovy files are non-relational.

You can write JSON-like markup in groovy files. They look and act very similarly. Biggest reason to go to groovy for data would be the ability to put in comments.

I would have made this a comment rather than an answer as this doesn't really answer your question but rather guides you in the right direction, but I am not allowed to comment on questions because I don't have enough reputation. I figured that trying to help you out with a failed answer was better than just ignoring your question.

The CSV (flat) file stands up as JSON once it is in Mongo. I have no issue getting one CSV file with how ever many strings into the db. My problem is taking 1000+ CSV files and images and getting them all in at once instead of doing them one by one. I appreciate your input though! — jmpman, Jan 07 '15 at 20:09

score 0 · Accepted Answer · answered Jan 08 '15 at 20:22

0

Write a Java program that will import all the images into GridFS first, then import the CSV files. Once they are all imported, allow the program to "update" the CSV with the corresponding PNG ID. Run from CMD.

answered Jan 08 '15 at 20:22

jmpman

23
1
13

Import into MongoDB/GridFS

2 Answers2