Converting all files in a folder to md using pandoc on Mac

Question

I am trying to convert an entire directory from html into markdown. The directory tree is quite tall, so there are files nested two and three levels down.

In answering this question, John MacFarlane suggested using the following Makefile:

TXTDIR=sources
HTMLS=$(wildcard *.html)
MDS=$(patsubst %.html,$(TXTDIR)/%.markdown, $(HTMLS))

.PHONY : all

all : $(MDS)

$(TXTDIR) :
    mkdir $(TXTDIR)

$(TXTDIR)/%.markdown : %.html $(TXTDIR)
    pandoc -f html -t markdown -s $< -o $@

Now, this doesn't seem to go inside subdirectories. Is there any easy way to modify this so that it will process the entire tree?

I don't need this to be in make. All I'm looking for is a way of getting a mirror of the initial directory where each html file is replaced by the output of running pandoc on that file.

(I suspect something along these lines should help, but I'm far from confident that I won't break things if I try to go at it on my own. I'm illiterate when it comes to GNU make).)

Answer 1

Since you mentioned you don't mind not using make, you can try bash.

I modified the code from this answer, use in the parent directory:

find ./ -iname "*.md" -type f -exec sh -c 'pandoc "${0}" -o "${0%.md}.pdf"' {} \;

It worked when I tested it, so it should work for you.

As per the request Any ideas how to specify the output folder? (Using html as the original file and md as the output):

find ./ -iname "*.html" -type f -exec sh -c 'pandoc "${0}" -o "./output/$(basename ${0%.html}.md)"' {} \;

I have tested this and it works for me.

Edit: As per a comment, the {} \; when used with find and the -exec option is used as a, more or less, placeholder for where the filename should be. As in it expands the filenames found to be placed in the command. The \; ends the -exec. See here for more explanation.

Answer 2

This is how I did it!

files=($(find ${INPUT_FOLDER} -type f -name '*.md'))
for item in ${files[*]}
do
  printf "   %s\n" $item
  install -d ${DIR}/build/$item
  pandoc $item -f markdown -t html -o ${DIR}/build/$item.html;
  rm -Rf ${DIR}/build/$item
done

Answer 3

I've created a python script for converting all files under a folder tree which have a given suffix. It's called Pandoc-Folder. It might be useful, so I've put it on github: https://github.com/andrewrproper/pandoc-folder

You can create a settings folder and file (YAML format), and then run it like this:

python pandoc-folder.py ./path/to/book/.pandoc-folder/settings-file.yml

there is an example-book folder and matching .bat and .sh scripts for how to convert the markdown from the example-book folder into a single output file.

I hope this might be useful to someone.

Answer 4

John MacFarlane's answer is almost right. However, one needs to create the subfolder for pandoc, in case it doesn't exist. This is how I'd do it:

TXTDIR=sources
HTMLS=$(wildcard *.html)
MDS=$(patsubst %.html,$(TXTDIR)/%.markdown, $(HTMLS))

.PHONY : all

all : $(MDS)

$(TXTDIR)/%.markdown : %.html $(TXTDIR)
    mkdir -p $(dir $@)
    pandoc -f html -t markdown -s $< -o $@

Answer 5

This is a solution using ipython:

from pathlib import Path
files = [path for path in Path('.').rglob('*.html')]
for f in files:
    !pandoc -s {str(path)} -o {path.name.replace(".html",".md")} 

Note that you must execute the command inside the directory where you keep the HTML files, and your file will be saved in the same directory. In case just change the output path.