How to avoid conversion of characters while converting file to Unix format in Windows

Question

I wrote a PowerShell script to convert files from Windows to Unix with below code.

while ($line = $tempfile.ReadLine()) {
    $outstream.Write($line + "`n")
}

It works fine if there are no special characters in data file. Now, if there are some special characters in data file they get converted as below:

test‡test → test¿½test
test,test → test¿½test

Any suggestion how it can be achieved without converting any characters?

NOTE: I don't want to use dos2unix as this exe is not present on server. So I need solution either in batch script or PowerShell.

Take a look at this for pointers: https://stackoverflow.com/questions/22789415/read-utf-8-files-correctly-with-powershell — Paxz, Feb 28 '19 at 07:03
a lot of duplicates: [Replace CRLF using powershell](https://stackoverflow.com/q/19127741/995714), [Convert file from Windows to UNIX through Powershell or Batch](https://stackoverflow.com/q/8852682/995714), [How to convert DOS line endings to UNIX on a Windows machine](https://superuser.com/q/1284467/241386), [Unix newlines to windows newlines (on Windows)](https://stackoverflow.com/q/724083/995714) — phuclv, Feb 28 '19 at 07:49

JPBlanc · Answer 1 · 2019-02-28T07:24:28.940

0

Not sure it's your problem here but, I think it's just encodeing.

in spite of :

$tempfile = [System.IO.StreamReader]("d:\temp\test.txt")
$tempfile.ReadLine()

You can try :

Get-Content 'D:\temp\test.txt' -encoding UTF8

Then use :

$tempfile = New-Object  System.IO.StreamReader "d:\temp\test.txt", $([System.Text.Encoding]::UTF8)
$tempfile.ReadLine()

Available encodings :

ASCII ; Unicode ; BigEndianUnicode ; UTF32 ; UTF7 ; UTF8 ; Default (for OEM)

edited Feb 28 '19 at 07:24

answered Feb 28 '19 at 07:06

JPBlanc

Hi, thanks for your response. I tried this solution, it worked for one of my test file, however when I am trying same solution to actual data file (*.dat) is failed. Any idea why it should be failing for .dat file. I tried to convert this *.dat file to *.txt and then tried to convert to unix, however it is still converting this characters as mentioned in above problem. – TAdhav Feb 28 '19 at 08:50
If you put a link on your file (accros the cloud), I can have a look. Just do not edit it before sharing it. – JPBlanc Feb 28 '19 at 10:13

score 0 · Answer 2 · answered Feb 28 '19 at 07:37

0

Have you tried [io.file] instead? Doesn't seem to have these issues.

$txt = [io.file]::ReadAllText('E:\temp\dos.txt') -replace "`r`n","`n"
[io.file]::WriteAllText('e:\temp\unix.txt', $txt)

answered Feb 28 '19 at 07:37

LeeM

2 Answers2