I want to match keywords of c# source code with regular expression. Let's say I have "new" keyword. I want to match all "new" keywords that are not inside " ", // (comment) and /* */ (second comment)
I've written so far:
\b[^\w@]new\b
However it's not working for:
new[]
var a = new[] { "bla" };
var string = "new"
foo(); // new
/* new */
How can I improve that regex?
It would be easier to capture all the undesirable matches and all the good things. Then later in programming logic test to see if a capture group was populated, if so then it is a match you want.
This expression will:
"new" or 'new'/* new */// newnew, var, and foo(\/\*(?:(?!\*\/)|.)*\*\/|\/{2}[^\r\n]*[\r\n]+)|("[^"]*"|'[^']*')|(new|var|foo)|(\w+)
I don't know c#, so I offer a powershell example to demonstrate how I would accomplish this. I made the expression case insensitve and turned on "dot matches new lines" by using (?is) and had to escape all the single quotes in the expression as ''.
Code
$String = 'NEW[]
var a = NEw[] { "bla" };
var string = "new"
foo(); // new
/*
new
*/
'
clear
[regex]$Regex = '(?is)(\/\*(?:(?!\*\/)|.)*\*\/|\/{2}[^\r\n]*[\r\n]+)|("[^"]*"|''[^'']*'')|(new|var|foo)|(\w+)'
# cycle through all matches
$Regex.matches($String) | foreach {
# Capture group 1 collects the comments, if populated then this match is a comment
if ($_.Groups[1].Value) {
Write-Host "comment at " $_.Groups[1].index " with a value => " $_.Groups[1].Value
} # end if
# capture group 2 collects the quoted strings, if populated then this match is a quoted string
if ($_.Groups[2].Value) {
Write-Host "quoted string at " $_.Groups[2].index " with a value => " $_.Groups[2].Value
} # end if
# capture group 3 collects keywords like new, var, and foo, if populated then this match is a keyword
if ($_.Groups[3].Value) {
Write-Host "keyword at " $_.Groups[3].index " with a value => " $_.Groups[3].Value
} # end if
# capture group 4 collects all the other word character chunks, so these might be variable names
if ($_.Groups[4].Value) {
Write-Host "possible variable name at " $_.Groups[4].index " with a value => " $_.Groups[4].Value
} # end if
} # next match
Output
keyword at 0 with a value => NEW
keyword at 7 with a value => var
possible variable name at 11 with a value => a
keyword at 15 with a value => NEw
quoted string at 23 with a value => "bla"
keyword at 33 with a value => var
possible variable name at 37 with a value => string
quoted string at 46 with a value => "new"
keyword at 53 with a value => foo
comment at 60 with a value => // new
comment at 68 with a value => /*
new
*/
Simple, use a lexer. A lexer finds groups of text in a string and generates tokens from those groups. The tokens are then provided with a "type". (Something to define what it is)
A C# keyword is one of the defined C# keywords.
A simple Regular expression for this would define borders followed by one of the possible C# keywords. ("\b(new|var|string|...)\b")
Your lexer will find all of the matches in a given string for keywords, create a token for each match, and say that the token "type" is "keyword".
However, like you say, you do not want to find keywords inside quotes, or comments. This is where the lexer really earns its points.
To resolve this case a (regex-based)lexer would use two methods:
A lexer works in the following steps:
Spoiler Alert Below is a fully functional lexer. It will demonstrate how a lexer works, because it is a fully functional lexer.
For Example:
Given regexes for strings, comments, and keywords, show how a lexer resolves conflicts between them.
//Simple Regex for strings
string StringRegex = "\"(?:[^\"\\\\]|\\\\.)*\"";
//Simple Regex for comments
string CommentRegex = @"//.*|/\*[\s\S]*\*/";
//Simple Regex for keywords
string KeywordRegex = @"\b(?:new|var|string)\b";
//Create a dictionary relating token types to regexes
Dictionary<string, string> Regexes = new Dictionary<string, string>()
{
{"String", StringRegex},
{"Comment", CommentRegex},
{"Keyword", KeywordRegex}
};
//Define a string to tokenize
string input = "string myString = \"Hi! this is my new string!\"//Defines a new string.";
//Lexer steps:
//1). Find all of the matches from the regexes
//2). Convert them to tokens
//3). Order the tokens by index then priority
//4). Loop through each of the tokens comparing
// the current match with the next match,
// if the next match is partially contained by this match
// (or if they both occupy the same space) remove it.
//** Sorry for the complex LINQ expression (not really) **
//Match each regex to the input string(Step 1)
var matches = Regexes.SelectMany(a => Regex.Matches(input, a.Value)
//Cast each match because MatchCollection does not implement IEnumerable<T>
.Cast<Match>()
//Select a new token for each match(Step 2)
.Select(b =>
new
{
Index = b.Index,
Value = b.Value,
Type = a.Key //Type is based on the current regex.
}))
//Order each token by the index (Step 3)
.OrderBy(a => a.Index).ToList();
//Loop through the tokens(Step 4)
for (int i = 0; i < matches.Count; i++)
{
//Compare the current token with the next token to see if it is contained
if (i + 1 < matches.Count)
{
int firstEndPos = (matches[i].Index + matches[i].Value.Length);
if (firstEndPos > matches[(i + 1)].Index)
{
//Remove the next token from the list and stay at
//the current match
matches.RemoveAt(i + 1);
i--;
}
}
}
//Now matches contains all of the right matches
//Filter the matches by the Type to single out keywords from comments and
//string literals.
foreach(var match in matches)
{
Console.WriteLine(match);
}
Console.ReadLine();
That is a valid(I tested it) almost-complete lexer.(feel free to use it or write your own) It will find all of the keywords that you define in the regex and not confuse them with string literals or comments.
