3

I used Tee-Object over the weekend to generate some output of a log file I was tailing, and I tried greping the output file and could not return any results. But the original log file I was able to grep.

It seems that Tee-Object has changed the encoding of the file.

https://adamtheautomator.com/tee-object-powershell/

Is there a setting I can change to just spit out the same encoding it read in to begin with, and also keep the line endings the same?

leeand00
  • 25,510
  • 39
  • 140
  • 297
  • 2
    The short answer is unfortunately no - `Tee-Object` doesn't have an `-Encoding` parameter, it makes no effort to determine the encoding of a file before appending, and outputs as unicode only. – Bill_Stewart Nov 18 '19 at 16:37

4 Answers4

3

Short answer no, there is no -Encoding parameter.

From PowerShell Tee-Object documentation:

Tee-Object uses Unicode encoding when it writes to files. As a result, the output might not be formatted properly in files with a different encoding. To specify the encoding, use the Out-File cmdlet.

HAL9256
  • 12,384
  • 1
  • 34
  • 46
  • 3
    Indeed. Note hat in PowerShell _Core_ the encoding is (BOM-less) UTF-8, not "Unicode" (UTF-16LE), which only applies to Windows PowerShell. Also, [let's try to fix this limitation](https://github.com/PowerShell/PowerShell/issues/11104) – mklement0 Nov 18 '19 at 16:59
3

As a workaround, tee to a variable, then use set-content to save it to a file. The default encoding is "ansi".

echo hi | tee -Variable a
set-content file $a

Here's an example, if you want the extra formatting that something like out-file normally provides. I'm guessing, because the original question has no example:

ps cmd | tee -var a
$a | out-string | set-content file

Actually, it looks like tee-object is invoking out-file, so this will set the encoding to ascii for tee-object:

$PSDefaultParameterValues = @{'Out-File:Encoding' = 'Ascii'}
js2010
  • 23,033
  • 6
  • 64
  • 66
  • The `-Variable` workaround is helpful in principle, but only with `Out-File -Encoding`, not with `Set-Content`, because it will not format the objects the same way that `Tee-Object` and `Out-File` do - try `$PSVersionTable | Set-Content t.txt`, for instance. Nice find on the default-parameter workaround via `Out-File`, but to make this work in a _command-scoped_ way (without having effects linger in the remaining session) would be too cumbersome in practice. However, it is good to know that the presence of such a default parameter value in the session does affect `Tee-Object`'s behavior. – mklement0 Nov 18 '19 at 19:52
  • @mklement0 if you want the extra formatting, pipe to out-string before tee -variable then – js2010 Nov 18 '19 at 19:55
  • If your intent is to offer a workaround for `Tee-Object`, which invariably uses PowerShell's output formatting system, `Set-Content` falls short, because it uses `.ToString()` formatting. Therefore, the correct workaround is to use `$a | Out-File file -Encoding $yourEncoding`. Yes, you can do `$a | Out-String | Set-Content -Encoding $yourEncoding`, but that's obviously less efficient and more cumbersome. – mklement0 Nov 18 '19 at 20:00
  • As an aside, for PowerShell _Core_ users: `tee` doesn't work as an alias on Unix-like platforms, where `tee` refers to the native `/usr/bin/tee` utility). ANSI encoding is `Set-Content`'s default only in Windows PowerShell; in PowerShell Core, it is (BOM-less) UTF-8. – mklement0 Nov 18 '19 at 20:12
  • Since tee-object provides -Append the problem become more complex in that tee may be only one of many processes appending to the file. If this is the case then the file (my testing shows) will be a combination of UTF8 and UTF16. And if these occurs then using out-file or anything else to correct the problem will not work. Only filtering out the NULLs will fix the trick after this problem arises. It would be better if tee-object provided the -Encoding option. In short if tee -Append writes UTF16 to the end of a file that is already UTF8 then you are screwed! – Walter ZAMBOTTI Dec 23 '20 at 01:37
  • @WalterZAMBOTTI I would use add-content instead, which checks the encoding of the file. – js2010 Dec 24 '20 at 21:17
2

  • HAL's helpful answer shows that, in Windows PowerShell and as of PowerShell (Core) 7.2.x, Tee-Object does not support specifying an output encoding explicitly when outputting to a file, and instead invariably uses "Unicode" (UTF-16LE) encoding in Windows PowerShell, and (BOM-less) UTF-8 in PowerShell (Core).

    • GitHub issue #11104 suggests removing this limitation by adding an -Encoding parameter to Tee-Object that allows specifying the desired output encoding.

  • js2010's answer shows that there actually is an indirect way to control the encoding, via an entry in the default-parameter-value table $PSDefaultParameterValues aimed at Out-File (e.g., $PSDefaultParameterValues['Out-File:Encoding'] = 'utf8').

    • However, this coupling between Tee-Object and Out-File is an implementation detail, so it is best not to rely on it. (Besides, it's nontrivial to scope it to an individual invocation of Tee-Object).

  • js2010's answer also is on the right track for a good workaround, by teeing to a variable first, but Set-Content is not the right cmdlet to use to then save the captured objects, because it performs simple .ToString() stringification on its input, whereas Tee-Object - like Out-File - applies PowerShell's rich default formatting.

Therefore, consider the following workaround:

# Tee to a *variable* first ($out)...
$PSVersionTable | Tee-Object -Variable out # | ...

# ... then use Out-File -Encoding to save to a file with the desired encoding
# ; e.g., with UTF-8
Out-File -InputObject $out out.txt -Encoding utf8

As for:

Is there a setting I can change to just spit out the same encoding

No - PowerShell doesn't support that in general: It reads file content into .NET strings in memory, and applies default (or specified) character encoding when saving back to a file.

The only workaround is to determine the input file's encoding manually, and then pass that encoding's name to a write-to-file cmdlet that has an -Encoding parameter, such as Out-File or Set-Content.

mklement0
  • 382,024
  • 64
  • 607
  • 775
0

As already noted, there is no encoding option for the tee command. To get around this, I was able to use the following for the conversion:

<powershell command> | tee -Variable content
$content | Set-Content -Encoding uft8 test_output.txt

This worked better than the testing I did to try and use Out-File.

Justin
  • 707
  • 7
  • 13