Read all .md files, convert them to html and send them

Question

I use fs to read the file which is in .md format and I want to transform it into html file.

This is my code so far:

fs = require('fs');
fs.readFile(__dirname + '/posts/react-v16.13.0.md', 'utf8', function (err, data) {
  if (err) {
    return console.log(err);
  }
  console.log(data);
});

the file is situated in that folder and has that name.

This function puts in console the content of the .md file.

For converting it to html I added this:

const showdown = require('showdown');
converter = new showdown.Converter();
...
fs = require('fs');
fs.readFile(__dirname + '/posts/react-v16.13.0.md', 'utf8', function (
  err,
  data
) {
  if (err) {
    return console.log(err);
  }
  text = data;
  html = converter.makeHtml(text);
  console.log(html);
});

It puts the file as html in the log which is fine.

My problem is how to do this if there are multiple files in /posts/ folder, how to read and send those files?

I would like to send them to front-end using a POST method.

Is it possible to read all the files from the folder, transform them and send them?

You can use `fs.readdir()` to obtain a list of files. https://nodejs.org/api/fs.html#fs_fs_readdir_path_options_callback — AKX, Sep 21 '20 at 19:43
@AKX I tried something like `fs.readdir(__dirname + '/posts', 'utf8', function (err, data) { ...}` but I get his error: `/.../Projects/md/server/node_modules/showdown/dist/showdown.js:2459 text = text.replace(/¨/g, '¨T'); ` — Leo Messi, Sep 21 '20 at 19:46
This may help you. https://stackoverflow.com/a/10049704/1524756 — Stephen Taylor, Sep 24 '20 at 13:32
What do you mean by "all files ... send them". Concat them all to a single html blob and send it as response/request body? — madflow, Sep 24 '20 at 13:45
where do you want to send the file to? and given there is a bunch of .md, do you want them concatenated or how? — Eric Wong, Sep 29 '20 at 13:37
@EricWong I want to send them to a React front-end. I would like to send them without concatenation in the same request if possible. Otherwise with concatenation but how will the front-end know how to de-concatenate them? — Leo Messi, Sep 29 '20 at 18:03
I am not sure what it mean to "send them to a React front-end", a frontend cannot really "receive" data, do you mean to use say query string and your React will copy it into the HTML? — Eric Wong, Sep 30 '20 at 05:59
Well then we ran into a problem, with query string, the [length is limited to ~2k](https://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers), so with this approach you cannot really "send" a compiled HTML >2k — Eric Wong, Sep 30 '20 at 07:08

Lionel Rowe · Accepted Answer · 2020-10-01T10:02:44.797

It seems from the comment thread below the question that you want something that does the following:

Converts all markdown files from a given directory to HTML
Sends them all in a single request
Is usable in a single-page app

Here's an approach that fulfils all of these requirements. The HTML of each post is inserted into a template element, the content of which can be cloned and manipulated within the SPA script.

server.js

// with `.promises`, we can use `async/await`
const fs = require("fs").promises;

// ...

const getHtmlByFilename = async filename => {
  const md = await fs.readFile(
    path.join(__dirname, "posts", filename),
    "utf-8"
  );

  return converter.makeHtml(md);
};

app.get("/", async (request, response) => {
  const filenames = await fs.readdir(path.join(__dirname, "posts"));

  // we can also use Promise.all
  // to map over the filenames in parallel
  const htmls = await Promise.all(
    filenames.map(async filename => {
      const html = await getHtmlByFilename(filename);

      return { filename, html };
    })
  );

  response.send(
    htmlBoilerplate(
      htmls
        .map(
          ({ filename, html }) =>
            `<template id="${filename}">${html}</template>`
        )
        .join("\n"),
      "<h1>SPA</h1>",
      '<script src="/public/spa.js"></script>'
    )
  );
});

public/spa.js

[...document.querySelectorAll("template")].forEach(template => {
  const clone = template.content.cloneNode(true);

  const filename = template.id;

  const details = document.createElement("details");
  const summary = document.createElement("summary");

  summary.textContent = filename;

  details.appendChild(summary);
  details.appendChild(clone);

  document.querySelector(".markdown-body").appendChild(details);
});

glitch.me demo

Source | Live

Limitations

The conversion is done on-the-fly. If you have heavy traffic, you'll want to implement some caching, or perhaps simply save the HTML versions separately, and trigger updates to them whenever the corresponding Markdown is edited.
The current code is probably not XSS safe — this assumes that either the content/filenames of posts are trusted, or that you carry out proper sanitation where required.

score 1 · Answer 2 · answered Sep 30 '20 at 17:54

const {readdir, readFile} = require('fs');
const showdown  = require('showdown');
const axios = require('axios');

let fileHtmlList = [];

const converter = new showdown.Converter();

readdir(`${__dirname}/posts`, 'utf8', (fsDirError, fileNameList) => {
    if(!fsDirError) {
        fileNameList.forEach((fileName) => {
            readFile(`${__dirname}/posts/${fileName}`, 'utf8', (fsReadError, fileContent) => {
                if(!fsReadError) {
                    fileHtmlList.push({
                        fileName: `${__dirname}/posts/${fileName}`,
                        htmlContent: converter.makeHtml(fileContent)
                    }); 
                } else {
                    return console.error(fsReadError);
                }  
            });    
        });
    } else {
        return console.error(fsDirError);
    }
});

/* I'm guessing this part from your description, if the content needs to be rendered then the code needs change */

let sendFrontEnd = async (data) => {
    try {
        const response = await axios.post(`urlToSend`, data);
        console.log(response);
    } catch (error) {
        console.error(error);
    }
};

fileHtmlList.forEach((item) => {
    sendFrontEnd(item.htmlContent);
});

score 0 · Answer 3 · answered Sep 24 '20 at 13:36

0

I recommend using the sync variants of readdir and readFile

const basePath = __dirname + '/posts/';
const htmls = [];

fs.readdirSync(basePath).forEach(file => {
  const text = fs.readFileSync(basePath + file, 'utf8');
  htmls.push({
    file,
    html: converter.makeHtml(text)
  });
});

// upload htmls with axios/fetch/ ....

answered Sep 24 '20 at 13:36

Sélim Achour

718
4
7

1

Why? This will bottleneck. – George Sep 24 '20 at 13:44
I thought he wants to send all the file with one post request which means he doesn't have that many files ... maybe I misunderstood – Sélim Achour Sep 24 '20 at 14:00

Đinh Tiến Vũ · Answer 4 · 2021-12-25T06:52:33.677

Try this js lib

<!-- Lightweight client-side loader that feature-detects and load polyfills only when necessary -->
<script src="https://cdn.jsdelivr.net/npm/@webcomponents/webcomponentsjs@2/webcomponents-loader.min.js"></script>
<!-- Load the element definition -->
<script type="module" src="https://cdn.jsdelivr.net/gh/zerodevx/zero-md@1/src/zero-md.min.js"></script>

    <div class="markdown-body">
        <zero-md src="README.md"> </zero-md>
    </div>

I strongly recommend using zero markdown in html file because

Automatic update from your readme.md file.
if you using convert readme to html, you must convert manually every time you update readme file (or code more).

Full html in my source

<!DOCTYPE html>
<html>
  <head>
    <title>API Get link Zing Mp3</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <style>
      .markdown-body {
        box-sizing: border-box;
        min-width: 200px;
        max-width: 980px;
        margin: 0 auto;
        padding: 45px;
      }
    
      @media (max-width: 767px) {
        .markdown-body {
          padding: 15px;
        }
      }
    </style>
    <!-- Lightweight client-side loader that feature-detects and load polyfills only when necessary -->
<script src="https://cdn.jsdelivr.net/npm/@webcomponents/webcomponentsjs@2/webcomponents-loader.min.js"></script>

<!-- Load the element definition -->
<script type="module" src="https://cdn.jsdelivr.net/gh/zerodevx/zero-md@1/src/zero-md.min.js"></script>

<!-- Simply set the `src` attribute to your MD file and win -->

  </head>
  <body>
    <div class="markdown-body">
      <zero-md src="README.md">
      </zero-md>
    </div>
    </body>
</html>

if you using nodejs, you can add a router to your readme.md file

app.get('/README.md', function (req, res) {
    res.sendFile(path.join(__dirname, "README.md"));
})

Read all .md files, convert them to html and send them

4 Answers4

server.js

public/spa.js

glitch.me demo

Limitations