Powershell - Retrieving content from a website that contains non-English characters and writing to a file results in incorrect characters

Question

There is a news website I frequent that has a series of headlines on their main page. Clicking the headline takes you to the individual story. I am trying to write a Powershell script that will loop through all the headlines on the main page and write each story to a text file.

The problem I am having is the stories are in Spanish and the Spanish characters with accent marks do not show up properly in my text file (actually the weird thing is, sometimes they do, but the majority of the time they don't). I've checked the headers of each story and the charset is set to UTF8 so I think the web pages themselves are formatted correctly. I've tried every way I know of to set the output file as UTF8 as well, but I can't seem to get it fixed.

Anyone have any ideas? Here is the code:

$ie = New-Object -ComObject 'InternetExplorer.Application'
$url = "https://www3.nhk.or.jp/nhkworld/es/news/"

#$ie.Visible = $true
$ie.Navigate($url)
while($ie.busy) {Start-Sleep 1}

$file = "C:\temp\nhk.txt"
if(Test-Path $file) { Remove-Item $file }

$lastLink = $null
foreach($link in $ie.Document.getElementsByTagName("a")) {
    if($link.href -match "\d{6}") { #the links to the stories we want are numbered with 6 digits
        if(-not($link.href -eq $lastLink)) {
            $uri = $link.href
            $w = Invoke-WebRequest -Uri $uri

            ForEach($element in $w.AllElements | where tagname -eq "p") {
                $text = $element | select -expand innerText
                $text = $text + "`r`n"
       
                Add-Content -Path $file -Value $text
                }
           
            $lastLink = $link.href
            }
        }
       
    }

Answer 1

I think it's the same basic problem as this question:

PowerShell Invoke-RestMethod Umlauts issues with UTF-8 and Windows-1252

The issue is the server is sending a response which is encoded using UTF8, but it's not correctly setting the Content-Type header to tell the client it's doing that, so the client is assuming it's encoded with the default ISO-8859-1 encoding.

This means, for example, the character ó is being sent by the server as the UTF8 byte sequence C3 B3 but the client is decoding that as an ISO-8859-1 byte sequence which becomes ó.

Since you can't presumably control the server's behaviour you might need to do some processing on the mangled text to recover the original version. I posted one way of doing that in an answer to the question above (see https://stackoverflow.com/a/58542493/3156906), but here it is again...

PS> $text = "ó"
PS> $bytes = [System.Text.Encoding]::GetEncoding("ISO-8859-1").GetBytes($text)
PS> $text = [System.Text.Encoding]::UTF8.GetString($bytes)
PS> write-host $text
ó

Answer 2

Did some more experimenting, and apparently, if you send the result of the Invoke-WebRequest call straight to a file using the -OutFile parameter, this file gets written in UTF8.

This should then (hopefully) do it:

# create a temporary file
$tempFile = (New-TemporaryFile).FullName

if(-not($link.href -eq $lastLink)) {
    $uri = $link.href
    $w = Invoke-WebRequest -Uri $uri

    Invoke-WebRequest -Uri $uri -OutFile $tempFile
    # read the file with encoding UTF8
    $content = Get-Content -Path $tempFile -Raw -Encoding UTF8
    # parse the html
    $html = New-Object -Com "HTMLFile"
    $html.IHTMLDocument2_write($content)
    # and append the innerText to your file "C:\temp\nhk.txt"
    Add-Content -Path $file -Value $html.body.innerText
    Add-Content -Path $file # add extra empty line

    # clean up
    [System.Runtime.Interopservices.Marshal]::ReleaseComObject($html) | Out-Null
    [System.GC]::Collect()
    [System.GC]::WaitForPendingFinalizers()
    $html = $null

       
    $lastLink = $link.href
}

# remove the temp file
Remove-Item -Path $tempFile -Force