Split on space while ignoring escaped quotes

Question

I would like to take a string representing options to a spark-submit command and format them with --conf interspersed between the options. This

concatConf :: String -> String
concatConf = foldl (\acc c -> acc ++ " --conf " ++ c) "" . words

works for most collections of options, e.g.,

λ => concatConf "spark.yarn.memoryOverhead=3g spark.default.parallelism=1000 spark.yarn.executor.memoryOverhead=2000"
" --conf spark.yarn.memoryOverhead=3g --conf spark.default.parallelism=1000 --conf spark.yarn.executor.memoryOverhead=2000"

But on occasion there can be spark.executor.extraJavaOptions, which is a space-separated, escaped-quote enclosed, list of additional options; for example,

"spark.yarn.memoryOverhead=3g spark.executor.extraJavaOptions=\"-verbose:gc -XX:+UseSerialGC -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy\" spark.default.parallelism=1000 spark.yarn.executor.memoryOverhead=2000"

and the concatConf function above obviously breaks down.

The following function, using the regex-compat library works for this example

import Data.Monoid (<>)
import Text.Regex (mkRegex, matchRegexAll)

concatConf :: String -> String
concatConf conf = let regex = mkRegex "(\\ *.*extraJavaOptions=\\\".*\\\")"
                  in case matchRegexAll regex conf of
                    Just (x, y, z, _) -> (insConf x) <> " --conf " <> y <> (insConf z)
                    Nothing           -> ""
                  where insConf = foldl (\acc c -> acc ++ " --conf " ++ c) "" . words

until you figure out that there's a similar spark.driver.extraJavaOptions that comes in a similar format. In any case, this function doesn't work for when there isn't such an option. Now I'm struggling with many cases: where there is none or one or both of these, which one appears first in the string if it's there, etc.

This sort of makes me feel like regex isn't the right tool for the job, hence my question, what is the right tool for this job?

Answer 1

A split is not the right weapon of choice here. Inspired by Jan Goyvaerts answer here I suggest substituting a match pattern instead that:

1. matches characters that aren't spaces or quotes,
2. and 1) followed by characters that begin and end with a quote, with no quotes in between.

[^\s"]+|\s[^\s"]++"(?:[^"]*)"\s

Output after substitution: --conf $0

 --conf spark.yarn.memoryOverhead=3g --conf  spark.executor.extraJavaOptions=\"-verbose:gc -XX:+UseSerialGC
-XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy\"  --conf spark.default.parallelism=1000  --conf spark.yarn.executor.memoryOverhead=2000

Demo

I hope, this is useful to you.

Note: There are some unnecessary spaces in the output since I had to add surrounding spaces to the second pattern. I haven't treated them since it would make the regex even more complicated and your CLI app won't complain, I guess.

Answer 2

I have a useless but interesting partial solution. It assembles strings that are assembled by words. It is a 'scanl1' function so outputs an partially assembled string before the final string.

strl = "spark.yarn.memoryOverhead=3g spark.executor.extraJavaOptions=\"-verbose:gc -XX:+UseSerialGC -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy\" spark.default.parallelism=1000 spark.yarn.executor.memoryOverhead=2000"

Now the function

t = scanl1 (\acc l -> if (take 5 l) == "spark" then "--conf " ++ l else  acc ++ l ) $ words strl

Now the results with three excess records

t !! 0
"spark.yarn.memoryOverhead=3g"
t !! 1
"spark.executor.extraJavaOptions=\"-verbose:gc"
t !! 2
"spark.executor.extraJavaOptions=\"-verbose:gc-XX:+UseSerialGC"
t !! 3
"spark.executor.extraJavaOptions=\"-verbose:gc-XX:+UseSerialGC-XX:+PrintGCDetails"
t !! 4
"spark.executor.extraJavaOptions=\"-verbose:gc-XX:+UseSerialGC-XX:+PrintGCDetails-XX:+PrintAdaptiveSizePolicy\""
t !! 5
"spark.default.parallelism=1000"
t !! 6
"spark.yarn.executor.memoryOverhead=2000"

The only thing going for this is it assembles correctly if excessively. I added the --conf after I ran it otherwise it would appear before each line

Answer 3

This sort of makes me feel like regex isn't the right tool for the job, hence my question, what is the right tool for this job?

The right tool for this job is monadic parsers.

{-# LANGUAGE TypeFamilies #-}

import Text.Megaparsec
import Text.Megaparsec.Char
import Replace.Megaparsec
import Data.Void
import Data.Either

-- | Invert a single-token parser “character class”.
-- | For example, match any single token except a letter or whitespace: 
-- |
-- |     anySingleExcept (letterChar <|> spaceChar)
-- |
anySingleExcept :: (MonadParsec e s m, Token s ~ Char) => m (Token s) -> m (Token s)
anySingleExcept p = notFollowedBy p *> anySingle

nonSpaceQuoted :: Parsec Void String String
nonSpaceQuoted = 
  ((chunk "\\\"") *> manyTill anySingle (chunk "\\\"")) -- match anything between escaped quotes
  <|> -- or
  (pure <$> anySingleExcept spaceChar) -- match anything that's not a space

wordsQuoted :: Parsec Void String String
wordsQuoted = fst <$> match (some nonSpaceQuoted)

input = "spark.yarn.memoryOverhead=3g spark.executor.extraJavaOptions=\\\"-verbose:gc -XX:+UseSerialGC -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy\\\" spark.default.parallelism=1000 spark.yarn.executor.memoryOverhead=2000"

putStrLn $ unlines $ fmap ("--conf " <>) $ rights $ splitCap wordsQuoted input

Here's the output, printed with unlines instead of unwords for clarity:

--conf spark.yarn.memoryOverhead=3g
--conf spark.executor.extraJavaOptions=\"-verbose:gc -XX:+UseSerialGC -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy\"
--conf spark.default.parallelism=1000
--conf spark.yarn.executor.memoryOverhead=2000