4

I have been working on refining and refactoring some 57k+ records from two legacy databases into one Django-compatible entity. Now when I'm done, I dumped it as a fixture and I am trying to load it in a production environment.

My problem is that the process is being "Killed" after a short while. My process is:

./manage.py syncdb --noinput
./manage.py loaddata core/fixtures/auth.json  # just a default user
./manage.py migrate

result:

Running migrations for django_extensions:  # custom apps migrate just fine
 - Migrating forwards to 0001_empty.
 > django_extensions:0001_empty
 - Loading initial data for django_extensions.
Installed 0 object(s) from 0 fixture(s)
Running migrations for myotherapp:
 - Migrating forwards to 0001_initial.
 > myotherapp:0001_initial
 - Loading initial data for myotherapp.
Installed 4 object(s) from 1 fixture(s)  # my other app with a fixture migrates ok
Running migrations for myapp:
 - Migrating forwards to 0001_initial.
 > myapp:0001_initial
 - Loading initial data for myapp.
Killed

I must note that the process is going without problems on my development machine. Another note is that my dev machine is running postgres 9.2, in production there is 9.1 - can this be that much of a problem?

How do I approach debugging this? I don't even know what is wrong from vague "Killed" being printed. Does South store any logs? Any help appreciated.

EDIT: As Paulo Scardine pointed out, the problem was with JSON file being to heavy. First I tried XML dump and it went further but was killed off eventually too. A way to go is SQL dump. For Postgres what worked for me was:

pg_dump dbname | gzip > filename.gz # dump data on dev machine
createdb dbname # create empty db in production
gunzip -c filename.gz | psql dbname # restore the dump in production
gwaramadze
  • 4,523
  • 5
  • 30
  • 42
  • Probably you have more spare RAM in the development machine. I think you are running into a known bug related to the size of the fixture. – Paulo Scardine Oct 27 '13 at 20:08
  • The size of my fixture is 72.4 mb. I will try dumping/loading every of my 4 models separately. – gwaramadze Oct 27 '13 at 20:15
  • I was told XML fixtures behave better than JSON ones when you have a lot of data. I tend to use plain sql dump/load when I run into this error; – Paulo Scardine Oct 27 '13 at 20:21
  • Thanks, I am gonna need some time to follow this clue. You might want to post it as an answer. – gwaramadze Oct 27 '13 at 20:23

1 Answers1

5

Could not find the specific bug regarding loading fixtures. This one is for dumping, but I guess the root cause is related:

There are a couple duplicates of your question:

When I run into this bug I was told to use XML fixtures instead because the XML parser behaves better regarding the memory footprint.

My advice is to not lose much sleep over this issue, resort to plain SQL dumps if you can.

Community
  • 1
  • 1
Paulo Scardine
  • 73,447
  • 11
  • 124
  • 153