Azure Arc - Installation failed: [1603]: Fatal error during installation.

While attempting to install the Azure Arc agent on a virtual machine (VM) hosted in the Vultr Cloud, I encountered the following error:

 ----

 Installation failed: [1603]: Fatal error during installation. : See C:\Users\ADMINI~1\AppData\Local\Temp\2\installationlog.txt for additional details.
At line:13 char:5
+     & "$env:TEMP\install_windows_azcmagent.ps1";
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,install_windows_azcmagent.ps1
 
Details: MSI (s) (28:F0) [23:57:11:300]: Product: Azure Connected Machine Agent -- Error 1920. Service 'Guest Configuration Arc Service' (GCArcService) failed to start.  Verify that you have sufficient privileges to start system services.
MSI (s) (28:F0) [23:57:13:215]: Product: Azure Connected Machine Agent -- Installation failed.
MSI (s) (28:F0) [23:57:13:215]: Windows Installer installed the product. Product Name: Azure Connected Machine Agent. Product Version: 1.33.02405.1188. Product Language: 1033. Manufacturer: Microsoft Corporation. Installation success or error status: 1603

-----

After spending a significant amount of time searching for solutions, I reviewed the hybrid agent's log file. You can locate the log file at the following path: C:\ProgramData\AzureConnectedMachineAgent\Log\himds.log

I found this interesting:

level=debug msg="Machine is not on AzSHCI: unable to communicate with IMDS endpoint: unable to communicate with IMDS endpoint: Get \"http://169.254.169.253:80/metadata/attested/document?api-version=2018-10-01\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
level=debug msg="Machine is not on AWS: received empty auth token"
level=debug msg="Machine is not on GCP: unable to communicate with IMDS endpoint: unable to communicate with IMDS endpoint: Get \"http://metadata.google.internal/computeMetadata/v1/?recursive=true\": dial tcp: lookup metadata.google.internal: no such host"
level=debug msg="Machine is not on OCI: received empty IMDS response"
level=debug msg="Machine is not on Azure: received empty IMDS response"
 level=info msg="Received service control message: 1"
level=warning msg="Stop service request received"
level=info msg="Service stopped"

At that point, it occurred to me that the issue might be connected to the metadata – Vultr also utilizes metadata. Let's quickly recap what metadata is, drawing from Microsoft's documentation:

"The Azure Instance Metadata Service (IMDS) provides information about currently running virtual machine instances."

--

To resolve the issue, I created an outbound firewall rule that blocked traffic to 169.254.169.0/24. After implementing this rule and attempting to reinstall the agent, everything started working as expected, and I successfully onboarded the VM to Azure Arc.