File rewriting: One line is greater than variable's max size. Workaround?

Question

I need to replace a single line in a file. Generally, this code works fine:

(The actual specifics on what this block is doing is not necessary for this question).

for /F "tokens=1* delims=:" %%a in ('findstr /N "^" %DATA%') do (
    if %%a equ %TargetLine% (
        echo !insert!>>%filepath%cc.tmp
    ) else (
        if [%%b]==[] (echo.>>%filepath%cc.tmp) else (echo %%b>>%filepath%cc.tmp)
    )
)

Unfortunately, each line is assigned to %%a, which like any other variable can only store a maximum length of 8,192 characters (thanks dbenham for that tidbid, comes in use now).

So what options do I have when the line is greater than 8,192 characters (23,708 in this case)?

Before you ask: No it cannot be separated to a new line, it is an 10k JSON array encoded in Base64 which is then written into a ByteArray.

I assume that the way to go is using regular expressions, is this the correct assumption, or is there another workaround?

Thanks.

Are there fixed characters at some points in the line, like a comma each 1000 characters or so? — jeb, Aug 06 '13 at 21:53

jeb · Accepted Answer · 2013-08-11T09:54:21.377

2

You could solve this with pure batch!

:readLongLine
< longline.tmp (
    for /L %%n in (1 1 20) do set /p part[%%n]=
)

After this your line is splitted into the variables part[1] .. part[20]

Writing this to a new file you could use

:writeLongLine
<nul (
    for /L %%n in (1 1 19) do set /p ".=!part[%%n]!"
    (echo !part[20]!)
) > longLine2.tmp

edited Aug 11 '13 at 09:54

answered Aug 06 '13 at 22:05

jeb

78,592
17
171
225

If the line must be manipulated, and the interesting bit spans multiple variables, then that could get very complicated. Also, writing out the value can be a problem if one of the values begins with space because of SET /P stripping leading spaces from prompt. I remember you had a solution for that on DosTips. Also, I was under the impression that the long line was embedded within a multi-line document. Knowing when to use the split line technique could be tricky. – dbenham Aug 06 '13 at 22:34
@dbenham You are right. There are about 10 lines, each with 10-25k characters in each. – TurdPile Aug 06 '13 at 22:41
@dbenham Also, thanks for the insight on the stripping. Good thing this batch script is simply modifying bytecode, so leading and trailing spacings are unimportant :) – TurdPile Aug 06 '13 at 22:48
@dbenham I never said that it's easy, but possible – jeb Aug 06 '13 at 22:56
I got it working - but just a note: the echo doesn't do anything. – TurdPile Aug 07 '13 at 00:01
The numbers don't match jeb, do they? 1 to 20 at the top, 1 to 9 at the bottom, plus 10. – foxidrive Aug 07 '13 at 00:42
Just noticed also that it doesn't stop at the end of the line. So really its just a read long file. When the line runs out, it'll go to the next. No bueno. – TurdPile Aug 07 '13 at 01:17
@TurdPile The echo adds a CR/LF at the line end, the counting was wrong, as my test line has only 10000 charcters – jeb Aug 11 '13 at 09:53

score 1 · Answer 2 · edited May 23 '17 at 12:05

1

You could use some other scripting language like VBScript, JScript, or PowerShell.

If you want to remain in the batch world, you can use a handy hybrid JScript/batch utility called REPL.BAT that performs regex search and replace on stdin and writes result to stdout. It is quite efficient, and works on any Windows machine from XP onward. It is pure script, so no exe download required. You can get REPL.BAT here. Full documentation is embedded within the script.

edited May 23 '17 at 12:05

Community

1
1

answered Aug 06 '13 at 22:02

dbenham

127,446
28
251
390

Or you could use a pure batch solution :-) – jeb Aug 06 '13 at 22:05
Ugh, guess I'm going to go ahead and try your answer out - even though I'd have much rather stayed within one file. – TurdPile Aug 07 '13 at 01:18
@dbenham I don't see how this will help my situation. Even if i read in a text file, the find and replace options would both need to be potentially 20k characters. Command line doesn't even support that many. – TurdPile Aug 07 '13 at 01:35
@TurdPile - yep - that would make the utility useless as written. Sounds to me like you should work directly with one of the other scripting languages. Or use a utility like sed, awk, or Perl as Endoro suggested. I prefer a scripting language because we are not allowed to install 3rd party exe files at work. – dbenham Aug 07 '13 at 01:43

score -2 · Answer 3 · answered Aug 06 '13 at 22:41

-2

Simply use sed, awk or Perl for the job.

answered Aug 06 '13 at 22:41

Endoro

37,015
8
50
63

1

Question is tagged batch for a reason. – TurdPile Aug 06 '13 at 22:42
@TurdPile All three apps can be used in a Windows command script (aka batch) context :) – Endoro Aug 06 '13 at 22:44
@TurdPile btw. your q. is also tagged `script` and : **I assume that the way to go is using regular expressions, is this the correct assumption, or is there another workaround?** I say _Yes there is! Take sed, awk or Perl to get it working._ – Endoro Aug 06 '13 at 22:48
question is also tagged `batch-file` :D – TurdPile Aug 06 '13 at 22:50
@TurdPile OK, so Good luck when dealing with this fine cmd variables xxD – Endoro Aug 06 '13 at 22:53

File rewriting: One line is greater than variable's max size. Workaround?

3 Answers3

Linked