0

I have a python script to convert json files to csv. It currently converts all files, but I want it to run only on those json files that have not been converted to csv already. All are in the same directory. How can I modify my code:

#!/bin/bash
# file: foo.sh
for f in *.json; do
  python ~/bin/convert.py "$f" "-fcsv"
done
supercontra
  • 165
  • 1
  • 9
  • *that have not been converted to csv already* - and how do you expect this condition to be verified? Noone even saw your `convert.py` content – RomanPerekhrest Feb 18 '18 at 15:44
  • I was hoping for some bash conditional looping, of the sort: for f in *.json && !*.csv; do – supercontra Feb 18 '18 at 15:59
  • Well obviously your convert.py file should rename the resulting file into .csv – Gnudiff Feb 18 '18 at 16:16
  • Perhaps the issue is that OP is running this over a lot of large json files each of which takes a long time to convert. So he is trying to optimize run time, so that the `convert.py` runs only over those files that havent been converted. Is that correct? IMO, its better to change the convert.py program, rather than write a bash wrapper on top of the `convert.py` (unless you are not allowed, or dont know how to change the `convert.py` program) – alpha_989 Feb 18 '18 at 16:40

2 Answers2

1

Assuming your script creates basename.csv for an input file named basename.json

for f in *.json; do
    test -e "${f%.json}.csv" && continue
    python ~/bin/convert.py "$f" "-fcsv"
done

The shell parameter expansion $(variable%pattern} produces the value of variable with any match on the glob pattern removed.

tripleee
  • 175,061
  • 34
  • 275
  • 318
-1

This leverages the find GNU-util and uses python for the rest. Since you are running a python script, I am assuming that python is installed on the system.

Full command below:

find . \( -type f -regextype posix-extended -regex  '.*json' \) - exec python -c "import sys; import os; filename, file_extension = os.path.splittext(sys.argv[1]); if not os.path.isfile(filename + '.csv') : os.system('python ~/bin/convert ' + filename + file_extension + ' -fcsv')" {}

searches for files with .json extension

find . \( -type f -regextype posix-extended -regex  '.*json' \)

takes the output from the find command and enters it into the python namespace

filename, file_extension = os.path.splittext(sys.argv[1]);

checks if there is a filename with extension .csv, if not runs the convert.py program

if not os.path.isfile(filename+'.csv'): os.system('python ~/bin/convert.py ' + filename+file_extension + ' -fcsv')"

You can put the entire script into a bash script and use \ to break it up into multiple lines as shown here: How can I split a shell command over multiple lines when using an IF statement?

alpha_989
  • 4,882
  • 2
  • 37
  • 48