Stack Overflow
Stack Overflow

Questions tagged [ndjson]

NDJSON is a newline delimited JSON format.

NDJSON is a newline delimited JSON format. It is a convenient format for storing or streaming structured data that may be processed one record at a time. It works well with unix-style text processing tools and shell pipelines. It's a great format for log files. It's also a flexible format for passing messages between cooperating processes.

Spec: http://ndjson.org

119 questions
248
votes
4 answers

How do I select multiple fields in jq?

My input file is newline delimited JSON (ndjson) looking something like this: { "login": "dmaxfield", "id": 7449977, ... } { "login": "dmaxfield", "id": 7449977, ... } I can get all the login names with this : cat members | jq…
J. Groesser
  • 2,547
  • 3
  • 11
  • 4
8
votes
2 answers

Serialize as NDJSON using Json.NET

Is it possible to serialize to NDJSON (Newline Delimited JSON) using Json.NET? The Elasticsearch API uses NDJSON for bulk operations, and I can find nothing suggesting that this format is supported by any .NET libraries. This answer provides…
Nathan Taylor
  • 24,423
  • 19
  • 99
  • 156
6
votes
3 answers

How to open .ndjson file in Python?

I have .ndjson file that has 20GB that I want to open with Python. File is to big so I found a way to split it into 50 peaces with one online tool. This is the tool: https://pinetools.com/split-files Now I get one file, that has extension…
taga
  • 3,537
  • 13
  • 53
  • 119
4
votes
1 answer

Unable to consume ndjson stream in Node, but similar code works in React

I am trying to consume data from this endpoint on lichess.org. Here is a minimal working example of a React component consuming that stream of data. I'm using a library called can-ndjson-stream. import ndjsonStream from "can-ndjson-stream" import {…
geekTechnique
  • 850
  • 1
  • 11
  • 38
4
votes
1 answer

How do I parse ndjson file using Golang?

I have a ndjson (newline delimited JSON) file, I need to parse it and get the data for some logical operation. Is there any good method for parsing ndjson files using golang. A sample ndjson is given…
ASHWIN RAJEEV
  • 2,525
  • 1
  • 18
  • 24
3
votes
1 answer

Wikidata - get labels for a large number of ids

I have a list of around 300.000 wikidata ids (e.g. Q1347065, Q731635 etc.) in an ndjson file as {"Q1347065": ""} {"Q731635": ""} {"Q191789": ""} ... etc What I would like is to get the label of each id, and form a dictionary of key values, such…
Paschalis
  • 191
  • 10
3
votes
1 answer

Reading in very, very large NDJSON

I have a 33GB NDJSON file I need to read into a data.table in R. It's gzipped into a 2GB file, ideally I would like to keep it compressed. The structure isn't so important except that (when imported via jsonlite::stream_in), the data I need are in…
Chris
  • 313
  • 1
  • 11
3
votes
1 answer

JSON to NDJSON on Google cloud platform

I am developing a prototype on Google cloud platform for which I am using cloud storage, appengine and bigquery. Now, one of the tasks is to load a file daily from google cloud storage to bigquery for which I am using Cron task on Appengine The…
Shiladitya
  • 133
  • 2
  • 11
2
votes
1 answer

How do I run jq in parallel on multiple CPUs

My script processes ~30 lines per second and uses just one CPU core. while read -r line; do echo "$line" | jq -c '{some-tansfomration-logic}'; done < input.json >> output.json The input.json is ~6GB 17M lines file. It's a new-line delimited json,…
stkvtflw
  • 12,092
  • 26
  • 78
  • 155
2
votes
1 answer

Converting json to ndjson using only input/output variables in Python?

I am writing a script that converts from json to ndjson in Python and am pretty new to the language. The cloud environment we use doesn't make use of files, but rather input and output variables. I found this code elsewhere on the site in…
Collaborator
  • 21
  • 1
2
votes
1 answer

Can't set document_id for deduplicating docs in Filebeat

What are you trying to do? I have location data of some sensors, I want to make geo-spatial queries to find which sensors are in a specific area (query by polygon, bounding-box, etc). The location data (lat-lon) for these sensors may change in the…
WISERDIVISOR
  • 154
  • 2
  • 12
2
votes
1 answer

How can I property output for application/x-ndjson

@GetMapping(produces = { MediaType.APPLICATION_JSON_VALUE, MediaType.APPLICATION_NDJSON_VALUE }) public Flux read() { } When I curl with --header 'Accept: application/x-ndjson' The outer array is gone but all new lines in each…
Jin Kwon
  • 20,295
  • 14
  • 115
  • 184
2
votes
1 answer

Reading .ndjson file into R

Can anyone direct me to how I can load large .ndjson files into R. My aim is to explore Parler social media data - which can be found here (https://zenodo.org/record/4442460#.YCOB32j7TFj) Initially, as the file is large I have downloaded the…
Sean
  • 47
  • 6
2
votes
1 answer

How to get a stream of a ndjson response

I am trying to connect to a http API. This API responses with a ndjson, that is a newline separated json strings. I need to consume these lines one by one, before I download them all (in fact even before the server knows what it will output on the…
Michal Maršálek
  • 53
  • 4
2
votes
1 answer

how-to-import-multiple-ndjson-files-into-r-as-a-dataframe?

I need to open 100 ndjson large files (with same columns) , I have prepared a script to apply to each file but I would not like to repeat this 100 times ! With ndjson::stream_in , I can only open 1 ndjson file into R as a data frame I tried the…
sandrine huart
  • 21
  • 3
1
2 3 4 5 6 7 8