Append new lines to a csv from json.parse

Question

more sysadmin (chef) than ruby guy, so this may be a five minute fix. I am working on a task where i write a ruby script that pulls json data from multiple files, parses it, and writes the desired fields to a single .csv file. Basically pulling metadata about aws accounts and putting it in an accountant friendly format.

Got a lot of help from another stackoverflow on how to solve the problem for a single file, json.parse help.

My issue is that I am trying to pull the same data from multiple JSON files in an array. I can get it to loop through each file with the code below.

require 'csv'
require "json"

delim_file = CSV.open("delimited_test.csv", "w")
aws_account_list = %w(example example2)
aws_account_list.each do |account|
json_file = File.read(account.to_s + "_aws.json")
parsed_json = JSON.parse(json_file)

delim_file = CSV.open("delimited_test.csv", "w")
# This next line could be a problem if you ran this code multiple times
delim_file << ["EbsOptimized", "PrivateDnsName", "KeyName",               "AvailabilityZone", "OwnerId"]
parsed_json['Reservations'].each do |inner_json|
inner_json['Instances'].each do |instance_json|
   delim_file << [[instance_json['EbsOptimized'].to_s,   instance_json['PrivateDnsName'], instance_json['KeyName'], instance_json['Placement']['AvailabilityZone'], inner_json['OwnerId']],[]]
 end
 delim_file.close
end

end

However, whenever I do it, it overwrites every time to the same single row in the .csv file. I have tried adding a \n string to the end of the array, converting the array to a string with hashes and doing a \n, but all that does is add a line to the same row that it overwrites.

How would I go about writing that it reads each json file, then appending each files metadata to a new row? This looks like a simple case of writing the right loop, but I can't figure it out.

http://stackoverflow.com/q/7085595/438992 E.g., open in append mode, not write mode. — Dave Newton, Apr 22 '15 at 14:59

Nic · Accepted Answer · 2017-03-19T01:02:14.870

1

You declared your file like this:

delim_file = CSV.open("delimited_test.csv", "w")

To fix your issue, all you have to do is change "w" to "a":

delim_file = CSV.open("delimited_test.csv", "a")

See the docs for IO#new for a description of the available file modes. In short, w creates an empty file at the filename, overwriting anyothers, and writes to that. a only creates the file if it doesn't exist, and appends otherwise. Because you have it currently at w, it'll overwrite it each time you run the script. With a, it'll append to what's already there.

edited Mar 19 '17 at 01:02

answered Apr 22 '15 at 15:00

Nic

6,211
10
46
69

nail on the head. Fixed things perfectly. – jcarapet Apr 22 '15 at 18:47

score 0 · Answer 2 · answered Apr 22 '15 at 15:33

You need to open file in append mode, use

delim_file = CSV.open("delimited_test.csv", "a")

'a' Write-only, starts at end of file if file exists, otherwise creates a new file for writing.

'a+' Read-write, starts at end of file if file exists, otherwise creates a new file for reading and writing'

Append new lines to a csv from json.parse

2 Answers2