1

I have a JSON file with over 3 million lines in my project and want to write a function that dynamically returns values from that file for a given key. The structure of the JSON is like so:

{
  "12345": {  // objects are identified by hash keys
    "objectProperty1": 12345,
    "objectProperty2": "bla",
    // ...
  },
  "67890": {
    "objectProperty1": 67890,
    "objectProperty2": "blub",
    // ...
  },
  // ...
}

I currently just import the entire file at the top of my code which makes starting the program take several minutes. I'd like to change that somehow.

Then I'd like to know how I can access specifc objects in that file. My function currently looks like this but it always returns undefined even when I'm sure that the passed key is in the JSON:

import * as jsonFile from './data.json';

const getObjectFromJsonFile = (key: number) => {
  const parsedFile = JSON.parse(JSON.stringify(jsonFile));
  const keyAsString = key.toString();
  return parsedFile[keyAsString];
};

When I call this function with getObjectFromJsonFile(12345) I get undefined. What's wrong here?

DaDo
  • 443
  • 1
  • 6
  • 20
  • 1
    People that had similar problems invented database systems. That is what you should use ;) – str Dec 14 '20 at 17:55
  • @DaDo This is a related article that may help you: (https://stackoverflow.com/questions/11874096/parse-large-json-file-in-nodejs) – Ryan Wilson Dec 14 '20 at 17:56
  • 2
    `JSON.parse(JSON.stringify(jsonFile))` this looks seriously fishy, and is most-likely a mistake. It seems to me that in order for this to work, `jsonFile` is already a JS object. "Quick" serialize-deserialize is costing you loads of time for nothing. For this to make sense, `const parsedFile = jsonFile;` must be equivalent to what you currently have. – spender Dec 14 '20 at 17:57
  • Sorry, my original comment was thinking this was a react question, it's nodejs, just use [readFile](https://nodejs.org/api/fs.html#fs_fs_readfile_path_options_callback) instead of importing it and you should be fine. – Adam Jenkins Dec 14 '20 at 17:58
  • @Adam how would I break up the JSON into smaller pieces? – DaDo Dec 14 '20 at 18:09
  • @str I have a MongoDB database in my project. Should I try to load the JSON in there and then only query the DB for requested data? – DaDo Dec 14 '20 at 18:09
  • @spender I get a TypeScript error when I try to do jsonFile[key], but after parsing it twice the error is gone. I got that from here: https://stackoverflow.com/a/41949494/13575631 – DaDo Dec 14 '20 at 18:09
  • 1
    @DaDo You definitely need to load these data to MongoDB. – Anatoly Dec 14 '20 at 18:49

0 Answers0