Some web resources require authentication to access. You can use various types of authentication with the Invoke-WebRequest cmdlet (Basic, NTLM, Kerberos, OAuth, or certificate authentication).

To perform Basic Authentication (authentication by name and password encoded in base64), you first need to get the username and password:

$cred = Get-Credential
wget -Uri 'https://somesite.com' -Credential $cred

In order to use the current Windows user credentials to perform NTLM or Kerberos authentication, add the -UseDefaultCredentials option:

Invoke-WebRequest 'https://somesite.com' -UseDefaultCredentials

To authenticate with a certificate, you need to specify its thumbprint:

Invoke-WebRequest 'https://somesite.com' -CertificateThumbprint xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

You can use modern Bearer/OAuth token authentication in your PowerShell scripts.

1. First, you need to get an OAuth token from your REST API provider (out of the scope of this post);
2. Convert token with ConvertTo-SecureString cmdlet: $Token = "12345678912345678901234567890" | ConvertTo-SecureString -AsPlainText –Force
3. Now you can perform OAuth authentication:
   $Params = @{
Uri = "https://somesite.com"
Authentication = "Bearer"
Token = $Token }
Invoke-RestMethod @Params

Parse and Scrape HTML a Web Page with PowerShell

The Invoke-WebRequest cmdlet allows you to quickly and conveniently parse the content of any web page. When processing an HTML page, collections of links, web forms, images, scripts, etc., are created.

Let’s look at how to access specific objects on a web page. For example, I want to get a list of all outgoing links (A HREF objects) on the target HTML web page:

$SiteAdress = "https://woshub.com"
$HttpContent = Invoke-WebRequest -URI $SiteAdress
$HttpContent.Links | Foreach {$_.href }

To get the link text itself (contained in the InnerText element), you can use the following command:

$HttpContent.Links | fl innerText, href

You can only select links with a specific CSS class:

$HttpContent.Links | Where-Object {$_.class -eq "page-numbers"} | fl innerText, href

Or specific text in the URL address:

$HttpContent.Links | Where-Object {$_.href -like "*powershell*"} | fl innerText,href

Then display a list of all images on this page:

$Img.Images

Create a collection of full URL paths to these images:

$images = $Img.Images | select src

Initialize a new instance of WebClient class:

$wc = New-Object System.Net.WebClient

And download all the image files from the page (with their original filenames) to the c:\too1s\ folder:

$images | foreach { $wc.DownloadFile( $_.src, ("c:\tools\"+[io.path]::GetFileName($_.src) ) ) }

How to Download File over HTTP/FTP with PowerShell Wget (Invoke-WebRequest)?

Invoke-WebRequest allows you to download files from a web page or FTP site (works like Wget or cURL on Windows). Suppose, you want to download a file from an HTTP using PowerShell. Run the following PowerShell command:

wget "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe" -outfile “c:\tools\firefox_setup.exe”

This command will download the file from the HTTP site and save it to the specified directory.

You can get the size of a file in MB before downloading it with wget:

$url = "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe"
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'/1Mb

Below is an example of a PowerShell script that will find all links to *.pdf files on a target web page and bulk download all found files from the website to your computer (each pdf file is saved under a random name):

$OutDir="C:\docs\download\PDF"
$SiteAdress = "https://sometechdocs.com/pdf"
$HttpContent = Invoke-WebRequest -URI $SiteAdress
$HttpContent.Links | Where-Object {$_.href -like "*.pdf"} | %{Invoke-WebRequest -Uri $_.href -OutFile ($OutDir + $(Get-Random 200000)+".pdf")}

As a result of the script in the target directory, all pdf files from the page will be downloaded. Each file is saved under a random name.

How to Fill and Submit HTML Form with PowerShell?

Many web services require?filling various data into HTML forms. With Invoke-WebRequest, you can access any HTML form, fill in the necessary fields, and submit the completed form back to the server. In this example, we’ll show how to sign in to Facebook through its standard web form using PowerShell.

With the following command, we will save the information about connection cookies in a separate session variable:

$fbauth = Invoke-WebRequest https://www.facebook.com/login.php -SessionVariable session

Using the next command, display the list of the fields to fill in the login HTML form (login_form):

$fbauth.Forms["login_form"].Fields

Assign the desired values to all fields:

$fbauth.Forms["login_form"].Fields["email"] = "[email protected]"

$fbauth.Forms["login_form"].Fields["pass"] = "Coo1P$wd"

Etc.

To submit (sent) the filled form to the server, call the action attribute of the HTML form:

$Log = Invoke-WebRequest -method POST -URI ("https://www.facebook.com/login.php" + $fbauth.Forms["login_form"].Action) -Body $fbauth.Forms["login_form"].Fields -WebSession $session

You can also use the JSON format to send data to a web page with the POST method:

$headers = @{
'Content-Type'='application/json'
'apikey'='0987654321'
}
$jsonbody = @{
"siteUrl" ="https://somesite.com"
"email" = "[email protected]"
}
Invoke-WebRequest -Method 'Post' -Uri $url -Body ($jsonbody |ConvertTo-Json) -Headers $headers -ContentType "application/json"

Invoke-WebRequest: Ignore SSL/TLS Certificate Checks

Another issue is that the Invoke-WebRequest cmdlet is closely related to Internet Explorer. For example, in Windows Server Core editions in which IE is not installed (or removed), the Invoke-WebRequest cmdlet cannot be used.

Invoke-WebRequest: The response content cannot be parsed because the Internet?Explorer engine is not available, or Internet Explorer’s first-launch?configuration is not complete. Specify the UseBasicParsing parameter and try?again.

In this case, the WebClient class can be used instead of Invoke-WebRequest. For example, to download a file from a specified URL, use the command.

(New-Object -TypeName 'System.Net.WebClient').DownloadFile($Url, $FileName)

If an invalid SSL certificate is used on an HTTPS site, or PowerShell doesn’t support this type of SSL/TLS protocol, then the Invoke-WebRequest cmdlet drops the connection.

Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.

By default, Windows PowerShell (in early builds of Windows 10, Windows Server 2016, and older versions of Windows) uses the legacy and insecure TLS 1.0 protocol for connections (check the blog post describing the PowerShell module installation error: Install-Module: Unable to download from URI).

If the legacy TLS 1.0 and TLS 1.1 protocols are not disabled in Windows, you must run the following command in order to use TLS 1.2 in PowerShell connections:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

If a self-signed certificate is used on the HTTPS site, the Invoke-WebRequest cmdlet refuses to receive data from it. To ignore (skip) an invalid SSL certificate, use the following PowerShell code:

add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
$result = Invoke-WebRequest -Uri "https://somesite.com"

Another significant drawback of the Invoke-WebRequest cmdlet is its rather low performance. When downloading a file, the HTTP stream is entirely buffered into memory, and only after the full complete download is completed is saved to disk. Thus, when downloading large files using Invoke-WebReques, you may run out of RAM.