2

I'm ran into a problem using dump/restore to move a MongoDB database from my local machine to the MongoHQ.com. When I restore the dump to MongoHQ it is missing about 10% of the documents with no apparent errors.

To troubleshoot I though I would use export/import locally to validate the data and I am getting a error on import, the export did not report any errors. I used json format for the export. Here is one of the import errors; 

exception:BSON representation of supplied JSON is too large: code FailedToParse: FailedToParse: Date milliseconds overflow: offset:10357

It reports errors on 726 documents out of about 130k. Based on the error I think it is having problems with a date field but I cant figure out how to find the problem documents. I tried doing a repair with no luck.
Anyone have an idea to try to find what's wrong?

SteveO7
  • 2,430
  • 3
  • 32
  • 40
  • Can you try using --objcheck with mongorestore. Maybe also --verbose – geakie Jan 23 '13 at 11:47
  • Good suggestion, it restored with no errors using the --objcheck option but the export/import is still reporting the same errors on import. I'll see if I can find a tool to open the 450mb JSON file and see if I can find dates that don't look right. – SteveO7 Jan 23 '13 at 16:30

1 Answers1

7

Here I am answering my own question again, hopefully it helps someone else.

The problem occurs on any document with dates before Jan 1 1970, the Unix epoch. I am assuming that dump/restore handles it okay but not export/import because dump/restore is using BSON and import/export uses JSON or CSV. This is an unresolved bug report for MongoDB https://jira.mongodb.org/browse/SERVER-961.

As a footnote, I initially loaded my MongoDB using Mongoid from Ruby which was apparently able to handle those pre 1970 dates.

SteveO7
  • 2,430
  • 3
  • 32
  • 40