Optimizing the replacement of parts of text documents

Question

So I'm trying to read a bunch of .mcfunction files (basically .txt) and replace things that are contained inside of || with the corresponding number. But the way I'm doing it now causes huge memory usage (like 360mb --> 6gb), so I need help fixing the memory leak. This is where the problem is:

public void Parse() {
    staticIDs sIDs = GameObject.Find("Main Camera").GetComponent<staticIDs>();
    functions = Directory.GetFiles(@"downloads\", "*.mcfunction", SearchOption.AllDirectories);
    totalFunctions = functions.Length;
    curFunction = 0;
    foreach (string function in functions)
    {
        System.Threading.Thread.Sleep(100);
        string text = File.ReadAllText(function);
        if (text.Contains("|"))
        {
            Debug.Log("Things to parse");
            goto contParse;
        }
        else {
            Debug.Log("No things to parse");
            goto finishParse;
        }
        contParse:
        lines = File.ReadAllLines(function);
        int lineNumber = 0;
        foreach (string line in lines)
        {
            System.Threading.Thread.Sleep(100);
            parts = line.Split('|');
            temp = "";
            int curLine = lineNumber + 1;
            foreach (string part in parts)
            {
                System.Threading.Thread.Sleep(100);
                int IDchecked = 0;
                done = false;

                tryAgain:
                if (IDchecked < sIDs.IDnames.Capacity && sIDs.IDnames[IDchecked] == part)
                {
                    temp = temp + sIDs.IDnumbers[IDchecked].ToString();
                    goto replaced;
                }
                else if (IDchecked <= sIDs.IDnames.Capacity)
                {
                    IDchecked++;
                    goto tryAgain;
                }
                else if (IDchecked > sIDs.IDnames.Capacity)
                {
                    temp = temp + part;
                    goto replaced;
                }
                else
                {
                    goto replaced;
                }
                replaced:
                Debug.Log(temp);
            }
            done = false;
            newLines.Add(temp);
        }
        File.Delete(function);
        File.Create(function).Dispose();
        foreach (string line in newLines)
        {
            File.AppendAllText(function, line);
        }
        finishParse:
        curFunction++;
        if (curFunction > totalFunctions)
            goto move;
    }
    move:
    Debug.Log("got to moving");
    Invoke("Move", 10f);
}

I would like to get the memory usage down to as little as possible.

What's the total size of all the files you're trying to modify? Unrelated, you may want to refactor away from using `goto`. — Jonathon Chase, Jan 03 '19 at 01:48
What are all the sleeps for? I also agree with getting rid of the gotos. You might also consider if you've already read the whole file it's probably better to split that text into lines rather than read the whole file again. Same with writing a line at a time. Joint the lines and write it all at once. — Retired Ninja, Jan 03 '19 at 01:50
The sleeps where me trying to slow it down. The file size is relativly small but there are 100+ files. And I do split the text into lines but how do I write them all at once whilst maintaining the proper fortmatting? — The Nuclear Nexus, Jan 03 '19 at 02:00
Reopening file for every line should make your code slow enough already without sleep... Also that insane `goto` look like some sort of `for` loop … decompiled. Consider rewriting it by hand so it looks real. After that someone likely can give you advice on actual code. (In mean time check out https://stackoverflow.com/questions/7351031/regex-replace-multiple-groups which likely will be better than … you are doing) — Alexei Levenkov, Jan 03 '19 at 02:32
getting rid of those gotos and swapping to a for loop really helped ty — The Nuclear Nexus, Jan 03 '19 at 03:32
If you solved your problem I'd recommend deleting the post. There is no way it will be upvoted without significant editing effort and eventually it may collect some downvotes for strange looking code... (deleted 0-voted posts without answers don't impact question ban unlike negatively scored once... just saying). If you still have problem with performance you may want to [edit] post with version of the code that looks sane and provide clear goal you want to achieve (consider reading https://ericlippert.com/2012/12/17/performance-rant/ about asking "performance" questions before that so) — Alexei Levenkov, Jan 03 '19 at 05:37
Can you create a public git with the source code and the example of the file, to look into it? — Herman, Jan 04 '19 at 08:45

score 0 · Answer 1 · edited Jan 03 '19 at 21:59

0

The memory leak is likely caused by string concatenation. string + string creates a brand new (immutable) string. Avoid those, and use a StringBuilder instead

instead of doing temp = temp + something, make temp StringBuilderer and call temp.Append(something)

edited Jan 03 '19 at 21:59

Glurth

290
1
17

answered Jan 03 '19 at 12:32

zambari

4,797
1
12
22

I can't seem to find string.Append? Maybe its called something else? – The Nuclear Nexus Jan 03 '19 at 15:20
@TheNuclearNexus "StringBuilder" is a separate class from "string". Its functions, take and return "string"s. – Glurth Jan 03 '19 at 17:57

Optimizing the replacement of parts of text documents

1 Answers1