Convert Windows file format to Linux format

Question

I am using Linux machine for development and then deploying my script files to Azure function app which is a Windows machine.

Then I am copying shell script files to wasb from Azure function app.

I am getting following error when running shell script in Edge node on Azure HDI cluster

$'\r': command not found

My script files are not executed properly.

What is the best way to convert dos2unix option in C# Azure function?

Do you use Notepad++? When you create a file on Windows, you could select Unix type. — Shui shengbao, Oct 25 '17 at 07:33
@Walter-MSFT I am not using Windows machine. I am copying files from github to Azure function and then to WASB. Files are stored in Azure function app which is a windows machine. I am pushing files from Linux machine to github. I need a automated approach not manual. — Galet, Oct 25 '17 at 07:41
There's no such thing as `Linux File Format` and `Windows File Format`. The newline characters for *text files* are different on the two OS but C# works with both. The newline in Linux is `\n`. In Windows it's `\r\n`. Neither uses `\r` — Panagiotis Kanavos, Oct 25 '17 at 08:42

score 2 · Answer 1 · answered Oct 25 '17 at 08:51

You don't have to do anything special to read a file that contains Unix-style newlines (\n) instead of \r\n. .NET IO methods treat both as newlines.

You could write

var lines-File.ReadAllLines("myUnixText.txt");

or

using(var reader=File.OpenText("myUnixText.txt"))
{
    string line;
    while( (line=reader.ReadLine()) !=null)
    {
        // Do something
    }
}

to read lines whether the line ending is \r or \n

To prove it :

var numbers = new[]{1,2,3,4,5,6};
var lines=String.Join("\n",numbers);
File.WriteAllText("myUnixText.txt",lines);

var newLines=File.ReadAllLines("myUnixText.txt");

Debug.Assert(newLines.Length==6);

Even though only a single string was written, File.ReadAllLines read 6 lines from the file

score 1 · Answer 2 · answered Oct 25 '17 at 07:41

1

You could upload your file to Linux and change file format.

dos2unix <filename>

Another easy solution is that if you install Notepad++, it supports automatic conversion format to Unix(LF).

answered Oct 25 '17 at 07:41

Shui shengbao

18,746
3
27
45

Or no. Just use `File.OpenText(filePath)`. The TextReader handles `\n` just fine – Panagiotis Kanavos Oct 25 '17 at 08:44
1

Visual Studio and Visual Studio Code also work with `\n`. In fact, you'll even get a warning from VS if the file contains inconsistent newlines – Panagiotis Kanavos Oct 25 '17 at 08:57
@PanagiotisKanavos Thanks, I will try this, I more like use Notepad++ in previous script development. – Shui shengbao Oct 25 '17 at 08:59

Galet · Answer 3 · 2017-10-25T13:00:01.940

1

I am calling Dos2Unix(filePath) method before copying file from Azure function to WASB.

Call method:-

Dos2Unix(D:\home\site\repository\sample.sh);

Following method actually works for me in C# Azure function.

private void Dos2Unix(string fileName)
{
    const byte CR = 0x0D;
    const byte LF = 0x0A;
    byte[] data = File.ReadAllBytes(fileName);
    using (FileStream fileStream = File.OpenWrite(fileName))
    {
        BinaryWriter bw = new BinaryWriter(fileStream);
        int position = 0;
        int index = 0;
        do
        {
            index = Array.IndexOf<byte>(data, CR, position);
            if ((index >= 0) && (data[index + 1] == LF))
            {
                // Write before the CR
                bw.Write(data, position, index - position);
                // from LF
                position = index + 1;
            }
        }
        while (index >= 0);
        bw.Write(data, position, data.Length - position);
        fileStream.SetLength(fileStream.Position);
    }
}

Update 1:-

Here is my code to upload files(*.sh, *.jar, *.img, etc) to blob storage.

public bool UploadBlobFile(string containerName, string blobName, string filePath)
{

    try{
        CloudBlobContainer container = blobClient.GetContainerReference(containerName);
        CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
        // convert dos2unix format
        Dos2Unix(filePath);
        using (var fileStream = System.IO.File.OpenRead(filePath))
        {
            blob.UploadFromStream(fileStream);
        }
        return true;
    } catch (Exception e) {
        log.Info("Exception: " + e);
        return false;
    }
}

edited Oct 25 '17 at 13:00

answered Oct 25 '17 at 07:48

Galet

5,853
21
82
148

Instead of doing this, try a simple `File.OpenText(fileName)`. You'll see that the TextReader can handle `\n` just fine – Panagiotis Kanavos Oct 25 '17 at 08:44
Or write `File.ReadAllLines(fileName)` to get all lines as an array. You don't need any of this code – Panagiotis Kanavos Oct 25 '17 at 08:55
@PanagiotisKanavos What should I do after I use File.OpenText(fileName) or File.ReadAllLines(fileName)?. Because currently I am using this code to upload blob file. Dos2Unix(filePath); using (var fileStream = System.IO.File.OpenRead(filePath)) { blob.UploadFromStream(fileStream); } – Galet Oct 25 '17 at 09:41
you don't need the `Dos2Unix` method at all. You don't need to do anything. .NET can handle `\n` just fine, as I show in my answer. You *did* use the `c#` tag so I assume your script is a C# file. In fact, you should post your *actual* code, not the error message – Panagiotis Kanavos Oct 25 '17 at 12:45
@PanagiotisKanavos I have updated the code. Can you tell me what I should I modify the above code? – Galet Oct 25 '17 at 13:00
is *this* the code that throws the error?? Where? Update the *question* with the actual script that throws the error. As for this code, simply remove the call to `dos2Unix`. If the Azure Function script is written in C# it can understand files with `\`n just fine – Panagiotis Kanavos Oct 25 '17 at 13:15

score 0 · Answer 4 · answered Oct 25 '17 at 07:35

0

It seems that you have Windows style line endings (\r\n) - you need to change them to unix style (\n).

Refer: '\r': command not found and Convert DOS line endings to Linux line endings in vim

answered Oct 25 '17 at 07:35

CHEEKATLAPRADEEP

12,191
1
19
42

score 0 · Answer 5 · answered Oct 25 '17 at 07:45

0

There is a tool in sourceforge that can do it. Is has been actualized recently a runs very well.

answered Oct 25 '17 at 07:45

Josem

378
7
15

Convert Windows file format to Linux format

5 Answers5