For questions about Miller, an open source command line utility, that is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON.
Questions tagged [miller]
20 questions
4
votes
2 answers
Split huge CSV by columns with Miller
I need to split huge (>1 Gb) CSV files containing 50K+ columns each on a daily basis.
I've found Miller as an interesting and performant tool for such a task.
But I'm stuck on Miller's documentation.
How could I split one CSV to N smaller CSV files…
franchb
- 1,174
- 4
- 19
- 42
3
votes
2 answers
Miller: ofs="\n"?
Suppose I have:
cat file
"Field 1, Line 1",Field 2,"Field 3, Line 1"
"Field 1, Line 2",Field 2,"Field 3, Line 2"
With Miller, I want to produce:
"Field 1, Line 1"
"Field 2"
"Field 3, Line 1"
"Field 1, Line 2"
"Field 2"
"Field 3, Line 2"
I tried…
dawg
- 98,345
- 23
- 131
- 206
3
votes
1 answer
How to convert a CSV to JSON array using the Miller command line tool?
Using the Miller command line tool I want to convert a CSV file with headers into a JSON array.
Currently I am using this command: mlr --icsv --ojson cat sample.csv > sample.json
It is outputting JSON, but not in array format.
This is the sample CSV…
TinyTiger
- 1,801
- 7
- 47
- 92
2
votes
2 answers
How can I clean up empty fields when converting CSV to JSON with Miller?
I have several CSV files of item data for a game I'm messing around with that I need to convert to JSON for consumption. The data can be quite irregular with several empty fields per record, which makes for sort of ugly JSON output.
Example with…
Aldoro
- 57
- 6
2
votes
1 answer
Is there a simple way to convert a CSV with 0-indexed paths as keys to JSON with Miller?
Consider the following CSV:
email/1,email/2
abc@xyz.org,bob@pass.com
You can easily convert it to JSON (taking into account the paths defined by the keys) with Miller:
mlr --icsv --ojson --jflatsep '/' cat file.csv
[ { "email": ["abc@xyz.org",…
Fravadona
- 13,917
- 1
- 23
- 35
1
vote
2 answers
Miller: selecting columns based on their last value
I'm processing a TSV file with a hundred columns and I would like to pick some of them based on their last values (ie. the value on the last row).
For instance with the following data (and I let you imagine 96 more columns!):
Year An Ti Gi …
mikael-s
- 310
- 2
- 12
1
vote
1 answer
MLR tool to convert JSON to CSV and pad missing fields from all possible headers in json
If I have three json objects in : test.json
{
"queue": "A",
"field1": 22,
"field2": 2,
"unique_field": 0
}
{
"queue": "B",
"field1": 39,
"field2": 3
}
{
"queue": "C",
"field1": 336,
"field2": 5
}
Running MLR to…
pankajsaha
- 13
- 5
1
vote
2 answers
Convert field names to lower case using miller
I would like to use miller (mlr) to convert column names to lower case. The closest I get is using the rename verb with a regular expression. \L should change the case, but instead the the column names are getting prefixed by "\L".
I'm using macOS…
Ben Carlson
- 1,053
- 2
- 10
- 18
1
vote
2 answers
Miller - Ignore valid field names when using -N
I'm using miller to process some CSV files like so:
mlr --mmap --csv --skip-comments -N cut -f 2 my.csv
It works well, but some of the CSV files contain field names and some do not, which is why I'm using -N. In the files that have field names,…
T145
- 1,415
- 1
- 13
- 33
1
vote
1 answer
Is there a Miller function to recognize csv numbers formatted as currency as numbers rather than string?
I've recently discovered Miller (mlr) for batch process csv files and love it, however it appears that numbers in the original data formatted as currency are recognized as strings by Miller, so trying to perform mathematical functions on them…
Hurston
- 48
- 7
1
vote
1 answer
How can Miller convert a local date and time to UTC?
How can Miller 5.6.2 convert a local date and time to UTC using an expression simpler than the following?
$ printf "time1\n2019-06-13 05:54 PM\n" | mlr --csv put '
$time1=sec2gmt(
localtime2sec(
strftime(
strptime($time1,…
Derek Mahar
- 27,608
- 43
- 124
- 174
0
votes
0 answers
How to use wget to install miller on Centos
I'm trying to run this command on a centos machine :
wget -nv -O/usr/bin/mlr https://github.com/johnkerl/miller/releases/download/v5.10.2/mlr.linux.x86_64
but i get the following error :
wget: unable to resolve host address ‘github.com’
Does…
TenEM
- 127
- 1
- 8
0
votes
1 answer
1) Reordering one csv file based on another file header and 2) Merging one column of one csv file to another and remove duplicate
I have two csv file.
Both files might have same or different data. File2 has only few columns from file 1. Some column in file 2 may have different header. eg File 2 has Name in place of First Name
Username, Identifier,One-time password,Recovery…
Haru Suzuki
- 142
- 10
0
votes
1 answer
How to split a large CSV file into multiple JSON files using the Miller command line tool?
I am currently using this Miller command to convert a CSV file into a JSON array file:
mlr --icsv --ojson --jlistwrap cat sample.csv > sample.json
It works fine, but the JSON array is too large.
Can Miller split the output into many smaller JSON…
TinyTiger
- 1,801
- 7
- 47
- 92
0
votes
1 answer
What is the Miller command for separating emails into their own rows, while also copying down other column data?
I have a very large csv file (213,265 rows) with many columns.
In one of those columns I have some emails seperated by commas. A trimmed down version of the csv file looks like this:
I would like to use Miller to seperate out those emails into…
TinyTiger
- 1,801
- 7
- 47
- 92