Invoke-WebRequest freezes / hangs

Question

Why does the cmdlet "Invoke-WebRequest" freezes / hangs on certains URLs? Any workaround possible? I want to access the "span" objects for a given webpage and this cmdlet would be very useful if it didn't hang like that.

For example, this hangs:

Invoke-WebRequest -Uri "https://cloud.google.com/chrome-enterprise/browser/download/"

This doesn't :

Invoke-WebRequest -Uri "https://www.microsoft.com/fr-ca/"

-UseBasicParsing makes it run but I want to use the functionality of what Invoke-WebRequest returns without basic parsing because with basic parsing, the span field i'm trying to extract is not populated.

ArcSet · Accepted Answer · 2020-10-19T01:44:36.107

10

This seems to still be a bug in powershell

https://github.com/PowerShell/PowerShell/issues/2867

A possible work around is to convert the item into parsed HTML manually

Function ConvertTo-NormalHTML {
    param([Parameter(Mandatory = $true, ValueFromPipeline = $true)]$HTML)

    $NormalHTML = New-Object -Com "HTMLFile"
    $NormalHTML.IHTMLDocument2_write($HTML.RawContent)
    return $NormalHTML
}

$Content = (Invoke-WebRequest -Uri "https://cloud.google.com/chrome-enterprise/browser/download" -UseBasicParsing ).Content

$ParsedHTML = ConvertTo-NormalHTML -HTML $Content

$ParsedHTML

edited Oct 19 '20 at 01:44

answered May 17 '19 at 15:56

ArcSet

6,518
1
20
34

Good one there, much appreciated. – Rakha May 17 '19 at 18:36
1

It should be "$Content = (Invoke-WebRequest -Uri "https://cloud.google.com/chrome-enterprise/browser/download" -UseBasicParsing )" though. Without the ".content" in the end. – JustAGuy Nov 16 '19 at 02:18
1

@JustAGuy, adding -UseBasicParsing worked just great and prevents the hangups. – Tolga Jun 09 '20 at 16:57
-UseBasicParsing has not solved this issue for me – James Moore Feb 04 '21 at 22:27
No error it just hangs and never returns. I identified the error - Akamai was tarpitting the requests. Is there a way to trap this error? – James Moore Feb 07 '21 at 14:45
@JamesMoore Not that I know of....how did you figure out that was the issue? – ArcSet Feb 09 '21 at 22:14

Invoke-WebRequest freezes / hangs

1 Answers1