19

Problem: I have thousands of documents which contains a specific character I don't want. E.g. the character a. These documents contain a variety of characters, but the a's I want to replace are inside double quotes or single quotes.

I would like to find and replace them, and I thought using Regex would be needed. I am using VSCode, but I'm open to any suggestions.

My attempt: I was able to find the following regex to match for a specific string containing the values inside the ().

".*?(r).*?"

However, this only highlights the entire quote. I want to highlight the character only.

Any solution, perhaps outside of regex, is welcome.

Example outcomes: Given, the character is a, find replace to b

Somebody once told me "apples" are good for you => Somebody once told me "bpples" are good for you

"Aardvarks" make good kebabs => "Abrdvbrks" make good kebabs

The boy said "aaah!" when his mom told him he was eating aardvark => The boy said "bbbh!" when his mom told him he was eating aardvark

Ka Mok
  • 1,937
  • 3
  • 22
  • 47
  • 1
    I couldn't follow the question. Could you be a little more clear? You want to replace all `r` to `X`? or only this `".?(r).?"` pattern? – Gauthaman Sahadevan Feb 20 '18 at 04:04
  • 1
    By the way - At least on the search box, VS Code is based on JavaScript, so its regular expressions support lookaheads, but not lookbehinds. Also, your current pattern fails for input like `"aaa" rrr "bbb"`. – Kobi Feb 20 '18 at 05:44
  • Was is in front of and behind the " in the string? Or is the string just "Peter Griffin"? Or does the string contain several Names like "Bla 'Peter Griffin' koko 'Maria Brown' super" and you just want to replace the "r" in the names? – D. Braun Feb 20 '18 at 08:30
  • Ok, let me rewrite the premise to be more clear, and I will update the post. I have thousands of documents. These documents have strings in them. I only want to replace a specific character, that's in string, inside these documents. Eg, "What a wonderful day!", if the character is the letter a, will replace it to "WhXt X wonderful dXy!". But ONLY if they are in string. Any `a` outside double or single quotes should be left alone. – Ka Mok Feb 20 '18 at 15:32
  • Th easiest possible thing to do, if you want a editor only option is to replace `"(.*?)a(.*?)"` with `"$1X$2"`. This would need to be replace multiple time and every time it will take 1 character inside the quotes. So if you have a maximum of 5 characters inside such a quote you will need to replace all 5 times. Any other solution I assume has to be using code and not using a editor – Tarun Lalwani Feb 23 '18 at 04:19
  • @KaMok Can you edit the question to include possible imput and the desired output? Also think about the following situation: _"This is a between quotes" another a between quotes "and the last a"_ What would be considered as between quotes? – André Kool Feb 23 '18 at 09:39
  • Please update your post with sample input and desired output. This will make easier to understand problem and propose a workable solution. – Saleem Feb 24 '18 at 16:07
  • @AndréKool, I've updated. Between quotes would equate to any valid string. – Ka Mok Feb 24 '18 at 17:10

6 Answers6

16

Visual Studio Code

VS Code uses JavaScript RegEx engine for its find / replace functionality. This means you are very limited in working with regex in comparison to other flavors like .NET or PCRE.

Lucky enough that this flavor supports lookaheads and with lookaheads you are able to look for but not consume character. So one way to ensure that we are within a quoted string is to look for number of quotes down to bottom of file / subject string to be odd after matching an a:

a(?=[^"]*"[^"]*(?:"[^"]*"[^"]*)*$)

Live demo

This looks for as in a double quoted string, to have it for single quoted strings substitute all "s with '. You can't have both at a time.

There is a problem with regex above however, that it conflicts with escaped double quotes within double quoted strings. To match them too if it matters you have a long way to go:

a(?=[^"\\]*(?:\\.[^"\\]*)*"[^"\\]*(?:\\.[^"\\]*)*(?:"[^"\\]*(?:\\.[^"\\]*)*"[^"\\]*(?:\\.[^"\\]*)*)*$)

Applying these approaches on large files probably will result in an stack overflow so let's see a better approach.

I am using VSCode, but I'm open to any suggestions.

That's great. Then I'd suggest to use awk or sed or something more programmatic in order to achieve what you are after or if you are able to use Sublime Text a chance exists to work around this problem in a more elegant way.

Sublime Text

This is supposed to work on large files with hundred of thousands of lines but care that it works for a single character (here a) that with some modifications may work for a word or substring too:

Search for:

(?:"|\G(?<!")(?!\A))(?<r>[^a"\\]*+(?>\\.[^a"\\]*)*+)\K(a|"(*SKIP)(*F))(?(?=((?&r)"))\3)
                           ^              ^            ^

Replace it with: WHATEVER\3

Live demo

RegEx Breakdown:

(?: # Beginning of non-capturing group #1
    "   # Match a `"`
    |   # Or
    \G(?<!")(?!\A)  # Continue matching from last successful match
                    # It shouldn't start right after a `"`
)   # End of NCG #1
(?<r>   # Start of capturing group `r`
    [^a"\\]*+   # Match anything except `a`, `"` or a backslash (possessively)
    (?>\\.[^a"\\]*)*+   # Match an escaped character or 
                        # repeat last pattern as much as possible
)\K     # End of CG `r`, reset all consumed characters
(   # Start of CG #2 
    a   # Match literal `a`
    |   # Or
    "(*SKIP)(*F)    # Match a `"` and skip over current match
)
(?(?=   # Start a conditional cluster, assuming a positive lookahead
    ((?&r)")    # Start of CG #3, recurs CG `r` and match `"`
  )     # End of condition
  \3    # If conditional passed match CG #3
 )  # End of conditional

enter image description here

Three-step approach

Last but not least...

Matching a character inside quotation marks is tricky since delimiters are exactly the same so opening and closing marks can not be distinguished from each other without taking a look at adjacent strings. What you can do is change a delimiter to something else so that you can look for it later.

Step 1:

Search for: "[^"\\]*(?:\\.[^"\\]*)*"

Replace with: $0Я

Step 2:

Search for: a(?=[^"\\]*(?:\\.[^"\\]*)*"Я)

Replace with whatever you expect.

Step 3:

Search for:

Replace with nothing to revert every thing.


Community
  • 1
  • 1
revo
  • 47,783
  • 14
  • 74
  • 117
  • I would like to reward you the bounty, but when I paste your solution inside the Search function of VSCode, and use Regex mode, I get an error saying: "Error parsing near 'a(?=[^"\' at character offset 3: Unrecognized flag: '='. (Allowed flags: i, m ,s U, u, x.) – Ka Mok Mar 01 '18 at 21:03
  • What is your version of VS Code? – revo Mar 01 '18 at 21:06
  • No it shouldn't be the case. I'm on v1.20 and can't reproduce your issue https://i.stack.imgur.com/JshQB.jpg – revo Mar 01 '18 at 21:10
  • 1
    I was able to get your regex to work according to this thread's solution to disable ripgrep: https://github.com/Microsoft/vscode/issues/30735 – Ka Mok Mar 01 '18 at 21:11
2
/(["'])(.*?)(a)(.*?\1)/g

With the replace pattern:

$1$2$4

As far as I'm aware, VS Code uses the same regex engine as JavaScript, which is why I've written my example in JS.

The problem with this is that if you have multiple a's in 1 set of quotes, then it will struggle to pull out the right values, so there needs to be some sort of code behind it, or you, hammering the replace button until no more matches are found, to recurse the pattern and get rid of all the a's in between quotes

let regex = /(["'])(.*?)(a)(.*?\1)/g,
subst = `$1$2$4`,
str = `"a"
"helapke"
Not matched - aaaaaaa
"This is the way the world ends"
"Not with fire"
"ABBA"
"abba",
'I can haz cheezburger'
"This is not a match'
`;


// Loop to get rid of multiple a's in quotes
while(str.match(regex)){
    str = str.replace(regex, subst);
}

const result = str;
console.log(result);
KyleFairns
  • 2,947
  • 1
  • 15
  • 35
2

Firstly a few of considerations:

  1. There could be multiple a characters within a single quote.
  2. Each quote (using single or double quotation marks) consists of an opening quote character, some text and the same closing quote character. A simple approach is to assume that when the quote characters are counted sequentially, the odd ones are opening quotes and the even ones are closing quotes.
  3. Following point 2, it could be worth some further thought on whether single-quoted strings should be allowed. See the following example: It's a shame 'this quoted text' isn't quoted. Here, the simple approach would think there were two quoted strings: s a shame and isn. Another: This isn't a quote ...'this is' and 'it's unclear where this quote ends'. I've avoided attempting to tackle these complexities and gone with the simple approach below.

The bad news is that point 1 presents a bit of a problem, as a capturing group with a wildcard repeat character after it (e.g. (.*)*) will only capture the last captured "thing". But the good news is there's a way of getting around this within certain limits. Many regex engines will allow up to 99 capturing groups (*). So if we can make the assumption that there will be no more than 99 as in each quote (UPDATE ...or even if we can't - see step 3), we can do the following...

(*) Unfortunately my first port of call, Notepad++ doesn't - it only allows up to 9. Not sure about VS Code. But regex101 (used for the online demos below) does.

TL;DR - What to do?

  1. Search for: "([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*"
  2. Replace with: "\1\2\3\4\5\6\7\8\9\10\11\12\13\14\15\16\17\18\19\20\21\22\23\24\25\26\27\28\29\30\31\32\33\34\35\36\37\38\39\40\41\42\43\44\45\46\47\48\49\50\51\52\53\54\55\56\57\58\59\60\61\62\63\64\65\66\67\68\69\70\71\72\73\74\75\76\77\78\79\80\81\82\83\84\85\86\87\88\89\90\91\92\93\94\95\96\97\98\99"
  3. (Optionally keep repeating steps the previous two steps if there's a possibility of > 99 such characters in a single quote until they've all been replaced).
  4. Repeat step 1 but replacing all " with ' in the regular expression, i.e: '([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*'
  5. Repeat steps 2-3.

Online demos

Please see the following regex101 demos, which could actually be used to perform the replacements if you're able to copy the whole text into the contents of "TEST STRING":

Steve Chambers
  • 37,270
  • 24
  • 156
  • 208
  • Why not a single capturing group instead of 99 and repeat that like you are saying to do in step 3? That would make it a lot shorter and easyer to read. – André Kool Feb 28 '18 at 12:30
  • Sure, that would be possible but could potentially involve a lot more manual steps - was trying to keep the manual steps to a minimum to make life simpler for the person doing it, hence using as many capturing groups as possible. – Steve Chambers Feb 28 '18 at 15:40
1

If you can use Visual Studio (instead of Visual Studio Code), it is written in C++ and C# and uses the .NET Framework regular expressions, which means you can use variable length lookbehinds to accomplish this.

(?<="[^"\n]*)a(?=[^"\n]*")

Adding some more logic to the above regular expression, we can tell it to ignore any locations where there are an even amount of " preceding it. This prevents matches for a outside of quotes. Take, for example, the string "a" a "a". Only the first and last a in this string will be matched, but the one in the middle will be ignored.

(?<!^[^"\n]*(?:(?:"[^"\n]*){2})+)(?<="[^"\n]*)a(?=[^"\n]*")

Now the only problem is this will break if we have escaped " within two double quotes such as "a\"" a "a". We need to add more logic to prevent this behaviour. Luckily, this beautiful answer exists for properly matching escaped ". Adding this logic to the regex above, we get the following:

(?<!^[^"\n]*(?:(?:"(?:[^"\\\n]|\\.)*){2})+)(?<="[^"\n]*)a(?=[^"\n]*")

I'm not sure which method works best with your strings, but I'll explain this last regex in detail as it also explains the two previous ones.

  • (?<!^[^"\n]*(?:(?:"(?:[^"\\\n]|\\.)*){2})+) Negative lookbehind ensuring what precedes doesn't match the following
    • ^ Assert position at the start of the line
    • [^"\n]* Match anything except " or \n any number of times
    • (?:(?:"(?:[^"\\\n]|\\.)*){2})+ Match the following one or more times. This ensures if there are any " preceding the match that they are balanced in the sense that there is an opening and closing double quote.
      • (?:"(?:[^"\\\n]|\\.)*){2} Match the following exactly twice
      • " Match this literally
      • (?:[^"\\\n]|\\.)* Match either of the following any number of times
        • [^"\\\n] Match anything except ", \ and \n
        • \\. Matches \ followed by any character
  • (?<="[^"\n]*) Positive lookbehind ensuring what precedes matches the following
    • " Match this literally
    • [^"\n]* Match anything except " or \n any number of times
  • a Match this literally
  • (?=[^"\n]*") Positive lookahead ensuring what follows matches the following
    • [^"\n]* Match anything except " or \n any number of times
    • " Match this literally

You can drop the \n from the above pattern as the following suggests. I added it just in case there's some sort of special cases I'm not considering (i.e. comments) that could break this regex within your text. The \A also forces the regex to match from the start of the string (or file) instead of the start of the line.

(?<!\A[^"]*(?:(?:"(?:[^"\\]|\\.)*){2})+)(?<="[^"]*)a(?=[^"]*")

You can test this regex here

This is what it looks like in Visual Studio:

Visual Studio example

ctwheels
  • 21,901
  • 9
  • 42
  • 77
  • @revo my goodness, how did I miss the code part... Thanks, edited. I'll leave my answer here in case the OP is open to using VS instead of VS Code – ctwheels Feb 23 '18 at 18:49
0

I am using VSCode, but I'm open to any suggestions.

If you want to stay in an Editor environment, you could use
Visual Studio (>= 2012) or even notepad++ for quick fixup.
This avoids having to use a spurious script environment.

Both of these engines (Dot-Net and boost, respectively) use the \G construct.
Which is start the next match at the position where the last one left off.

Again, this is just a suggestion.

This regex doesn't check the validity of balanced quotes within the entire
string ahead of time (but it could with the addition of a single line).

It is all about knowing where the inside and outside of quotes are.

I've commented the regex, but if you need more info let me know.
Again this is just a suggestion (I know your editor uses ECMAScript).

Find (?s)(?:^([^"]*(?:"[^"a]*(?=")"[^"]*(?="))*"[^"a]*)|(?!^)\G)a([^"a]*(?:(?=a.*?")|(?:"[^"]*$|"[^"]*(?=")(?:"[^"a]*(?=")"[^"]*(?="))*"[^"a]*)))
Replace $1b$2

That's all there is to it.

https://regex101.com/r/loLFYH/1

Comments

(?s)                          # Dot-all inine modifier
 (?:
      ^                             # BOS 
      (                             # (1 start), Find first quote from BOS (written back)
           [^"]* 
           (?:                           # --- Cluster
                " [^"a]*                      # Inside quotes with no 'a'
                (?= " )
                " [^"]*                       # Between quotes, get up to next quote
                (?= " )
           )*                            # --- End cluster, 0 to many times

           " [^"a]*                      # Inside quotes, will be an 'a' ahead of here
                                         # to be sucked up by this match           
      )                             # (1 end)

   |                              # OR,

      (?! ^ )                       # Not-BOS 
      \G                            # Continue where left off from last match.
                                    # Must be an 'a' at this point
 )
 a                             # The 'a' to be replaced

 (                             # (2 start), Up to the next 'a' (to be written back)
      [^"a]* 
      (?:                           # --------------------
           (?= a .*? " )                 # If stopped before 'a', must be a quote ahead
        |                              # or,
           (?:                           # --------------------
                " [^"]* $                     # If stopped at a quote, check for EOS
             |                              # or, 
                " [^"]*                       # Between quotes, get up to next quote
                (?= " )

                (?:                           # --- Cluster
                     " [^"a]*                      # Inside quotes with no 'a'
                     (?= " )
                     " [^"]*                       # Between quotes 
                     (?= " )
                )*                            # --- End cluster, 0 to many times

                " [^"a]*                      # Inside quotes, will be an 'a' ahead of here
                                              # to be sucked up on the next match                    
           )                             # --------------------
      )                             # --------------------
 )                             # (2 end)
0

"Inside double quotes" is rather tricky, because there are may complicating scenarios to consider to fully automate this.

What are your precise rules for "enclosed by quotes"? Do you need to consider multi-line quotes? Do you have quoted strings containing escaped quotes or quotes used other than starting/ending string quotation?

However there may be a fairly simple expression to do much of what you want.

Search expression: ("[^a"]*)a

Replacement expression: $1b

This doesn't consider inside or outside of quotes - you have do that visually. But it highlights text from the quote to the matching character, so you can quickly decide if this is inside or not.

If you can live with the visual inspection, then we can build up this pattern to include different quote types and upper and lower case.

James
  • 1
  • 1