Fix: Can’t Install VMware 17.5.2 or VirtualBox on Windows 11

You’ve just downloaded the latest version of VMware Workstation 17.5.2 or Oracle VirtualBox, ready to spin up a Linux VM, test software in a sandbox, or run that legacy Windows app. You double-click the installer — and within seconds, you’re hit with an error:

“Setup failed to install the product. Error 1920. Service ‘VMware Authorization Service’ (VMAuthdService) failed to start.”

Or perhaps:

“VirtualBox installation failed. Error: Cannot register the hard disk because a hard disk with UUID already exists.”

Maybe you get no error at all — just a silent rollback after “Preparing to install…”

You’re not alone.

Thousands of Windows 11 users — from developers and IT professionals to students and hobbyists — are encountering installation failures with VMware Workstation 17.5.2 and VirtualBox 7.x. The causes? A perfect storm of Windows 11 security features, Hyper-V conflicts, driver signing policies, TPM 2.0 enforcement, and Microsoft’s push toward its own virtualization stack (WSL2, Hyper-V, Sandbox).

But don’t panic.

This comprehensive 5000-word guide will walk you through every known cause and solution — step by step — to get VMware or VirtualBox installed and running smoothly on your Windows 11 machine.

Whether you’re facing driver signature errors, service startup failures, UAC blocks, Secure Boot restrictions, or Hyper-V interference — we’ve got you covered.

By the end of this guide, you’ll not only fix the installation — you’ll understand why it failed, how to prevent it in the future, and how to configure your system for optimal virtualization performance.

Let’s dive in.

Chapter 1: Understanding Why VMware and VirtualBox Fail on Windows 11

Before troubleshooting, it’s essential to understand why these once-reliable virtualization tools now stumble on Windows 11.

Windows 11’s Aggressive Security Model

Microsoft has hardened Windows 11 to meet enterprise and government security standards. Key changes affecting virtualization:

  • Mandatory Secure Boot and TPM 2.0 — Blocks unsigned or incompatible drivers.
  • Memory Integrity (HVCI) — Prevents unsigned kernel drivers from loading.
  • Core Isolation — Isolates critical system processes, blocking low-level hooks.
  • Driver Signature Enforcement (DSE) — Requires all drivers to be Microsoft-signed.
  • Windows Defender Application Control (WDAC) — Can block installers or services.

VMware and VirtualBox rely on kernel-mode drivers to intercept CPU instructions, manage memory, and virtualize hardware. If Windows 11 blocks those drivers — installation fails.

Hyper-V: The Silent Competitor

Microsoft’s own hypervisor, Hyper-V, is now deeply integrated into Windows 11 — even if you never enabled it. Features that depend on Hyper-V:

  • Windows Subsystem for Linux 2 (WSL2)
  • Windows Sandbox
  • Microsoft Defender Application Guard
  • Docker Desktop (uses WSL2 backend)
  • Android Subsystem for Windows (WSA)

When Hyper-V is active, it takes exclusive control of the CPU’s virtualization extensions (Intel VT-x / AMD-V). This creates a hypervisor conflict: only one Type-1 hypervisor can control the hardware at a time.

Result? VMware and VirtualBox — both Type-2 hypervisors — fail to install or start VMs.

💡 Note: VMware Workstation 17+ and VirtualBox 7+ can coexist with Hyper-V using “User Mode” or “Hyper-V backend” — but only if configured correctly.

Driver Signing and WHQL Requirements

VMware and VirtualBox install kernel drivers (.sys files) that must be digitally signed. Windows 11 enforces stricter signature validation:

  • SHA-1 signatures are deprecated.
  • Cross-signed drivers (common in older versions) are blocked.
  • Drivers must pass WHQL (Windows Hardware Quality Labs) testing.

VMware 17.5.2 drivers are signed — but if your system clock is wrong, certificate chain is broken, or Secure Boot rejects the signature — installation aborts.

UAC, Admin Rights, and Installer Confusion

Windows 11’s User Account Control (UAC) and SmartScreen can silently block installers — especially if downloaded from third-party sites or flagged as “unrecognized.”

Running the installer “as Administrator” doesn’t always bypass these blocks.

BIOS/UEFI Settings: The Forgotten Layer

Even if Windows is configured correctly, your firmware might block virtualization:

  • Intel VT-x or AMD-V disabled
  • TPM 2.0 not initialized
  • Secure Boot in “Custom” or “Setup” mode
  • CSM (Compatibility Support Module) enabled — breaks UEFI boot

Without proper firmware settings, no hypervisor will work.

Chapter 2: Preliminary Checks — Don’t Skip These!

Before attempting complex fixes, perform these quick checks. Many installation failures are resolved here.

1. Verify Hardware Virtualization Is Enabled in BIOS/UEFI

This is the #1 cause of hypervisor failures.

How to Check:

  1. Reboot your PC.
  2. Press F2, Del, F10, or Esc (varies by manufacturer) to enter BIOS/UEFI.
  3. Navigate to Advanced > CPU Configuration or Security > Virtualization.
  4. Enable:
    • Intel VT-x (Intel) or AMD-V (AMD)
    • Intel VT-d or AMD IOMMU (optional, for device passthrough)
  5. Also check:
    • Secure Boot: Should be Enabled
    • TPM Device: Should be Enabled (fTPM for AMD, PTT for Intel)
    • CSM (Compatibility Support Module): Should be Disabled
  6. Save and Exit (F10).

💡 On some laptops (Dell, HP, Lenovo), virtualization settings are under “Security” or “System Configuration.”

Reboot and proceed.

2. Check Windows 11 Edition and Build

VMware Workstation 17.5.2 requires:

  • Windows 10/11 Pro, Enterprise, or Education (Home Edition has limitations)
  • 64-bit OS
  • Minimum build: Windows 11 21H2 (OS Build 22000) — but 22H2 or 23H2 recommended

To check:

  1. Press Win + R, type winver, press Enter.
  2. Confirm you’re on Version 22H2 or later.

If you’re on Windows 11 Home:

→ VMware can install — but features like “Unity Mode” or certain networking may be limited.
→ VirtualBox installs fine on Home — but may conflict with WSL2.

3. Disable Conflicting Virtualization Features

Check if Hyper-V or related features are enabled.

Method 1: PowerShell (Admin)

powershell
Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All

If “State” is Enabled, Hyper-V is active.

Also check:

powershell
Get-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform
Get-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform

→ These enable WSL2 and Windows Sandbox.

Method 2: System Information

  1. Press Win + R, type msinfo32, press Enter.
  2. Look for:
    • Hyper-V Requirements: Should show “A hypervisor has been detected…” if Hyper-V is running.
    • Virtualization Enabled In Firmware: Must be Yes

4. Run Installer as Administrator

Right-click the VMware or VirtualBox installer → Run as administrator.

💡 Even if your account is admin, UAC may block silent install steps.

5. Temporarily Disable Antivirus and Windows Security

Third-party AV (McAfee, Norton, Kaspersky) and even Windows Defender can block driver installation.

  1. Open Windows Security → Virus & threat protection → Manage settings.
  2. Turn off Real-time protection temporarily.
  3. Disable third-party AV via its system tray icon.
  4. Run the installer again.

Remember to re-enable after installation.

Chapter 3: Fixing VMware Workstation 17.5.2 Installation Failures

Now let’s tackle VMware-specific issues.

Fix 1: Error 1920 — “VMware Authorization Service Failed to Start”

This is the most common VMware installer error.

Causes:

  • Service conflict (port 912 already in use)
  • Corrupted previous installation
  • Insufficient privileges
  • Antivirus blocking service registration

Solutions:

Step 1: Clean Previous VMware Installations

Use VMware’s official cleanup tool:

  1. Download VMware Install Cleaner from:
    https://kb.vmware.com/s/article/1011354

  2. Extract and run as Administrator.

  3. Select all VMware products → Clean.

  4. Reboot.

Step 2: Manually Delete Leftover Files

After cleaning, delete these folders if they exist:

  • C:\Program Files (x86)\VMware\
  • C:\ProgramData\VMware\
  • C:\Users\%USERNAME%\AppData\Local\VMware\
  • C:\Windows\System32\drivers\vm*.sys

💡 Press Win + R, type %appdata% → Go up one level to AppData → Delete Local\VMware and Roaming\VMware.

Step 3: Free Port 912

VMware Auth Service uses TCP port 912. If another app uses it, installation fails.

In Command Prompt (Admin):

cmd
netstat -ano | findstr :912

If a PID appears:

cmd
tasklist | findstr <PID>

→ End the task in Task Manager.

Or change VMware’s port manually (advanced):

  1. After installation fails, navigate to: C:\ProgramData\VMware\VMware Workstation\

  2. Edit config.ini (create if missing)

  3. Add:

    authd.port = "915"
  1. Re-run installer.

Fix 2: “Driver Installation Failed” or “Cannot Load Driver”

Caused by Driver Signature Enforcement or Memory Integrity.

Solution A: Disable Memory Integrity (HVCI)

  1. Open Windows Security → Device Security → Core Isolation Details.
  2. Toggle “Memory Integrity” to Off.
  3. Reboot.

⚠️ Warning: This reduces security. Re-enable after VMware installs if desired.

Solution B: Temporarily Disable Driver Signature Enforcement

  1. Press Win + I → System → Recovery → Advanced startup → Restart now.
  2. After reboot: Troubleshoot → Advanced options → Startup Settings → Restart.
  3. Press F7 to “Disable driver signature enforcement”.
  4. Windows boots with signing disabled.
  5. Run VMware installer immediately.

💡 This is temporary — signing re-enables after reboot.

Fix 3: VMware Installer Hangs or Rolls Back Silently

Often caused by Windows Installer corruption or pending reboots.

Step 1: Clear Windows Installer Cache

In Command Prompt (Admin):

cmd
net stop msiserver
del %windir%\Installer\*.msi /q
net start msiserver

Step 2: Run Microsoft Program Install and Uninstall Troubleshooter

Download from:
https://support.microsoft.com/en-us/topic/fix-problems-that-block-programs-from-being-installed-or-removed-cca7d1b6-65a9-3d98-426b-e9f927e1eb4d

Run it → Select VMware → Apply fixes.

Step 3: Install in Clean Boot State

  1. Press Win + R, type msconfig, press Enter.
  2. Go to Services → Check “Hide all Microsoft services” → Disable all.
  3. Go to Startup → Open Task Manager → Disable all startup items.
  4. Reboot.
  5. Run VMware installer.

If successful, re-enable services one by one to find the culprit.

Fix 4: VMware Fails on Windows 11 Home

Windows 11 Home lacks Group Policy and some Hyper-V controls — but VMware can still work.

Workaround: Force VMware to Use User Mode

VMware Workstation 17+ supports “User Mode” when Hyper-V is active.

  1. Install VMware normally (if possible).

  2. If VMs won’t start, edit VM .vmx file:

    Add these lines:

    hypervisor.cpuid.v0 = "FALSE"
    mce.enable = "TRUE"

  3. Or, globally: Edit C:\ProgramData\VMware\VMware Workstation\config.ini

    Add:

    vmx.allowHv = "TRUE"

  4. Reboot.

💡 This allows VMware to run atop Hyper-V — with minor performance loss.

Chapter 4: Fixing VirtualBox 7.x Installation Failures

VirtualBox has its own set of quirks on Windows 11.

Fix 1: “Installation Failed — Error Creating Service”

Usually caused by leftover VirtualBox services or driver conflicts.

Step 1: Use Oracle’s Cleanup Tool

  1. Download VirtualBox Uninstaller from:
    https://www.virtualbox.org/svn/vbox/trunk/src/VBox/Installer/win/VBoxDrvInst/
    → Look for VBoxDrvInst.exe

  2. Run Command Prompt as Admin:

    cmd
    VBoxDrvInst.exe cleanup

  3. Reboot.

Step 2: Manually Remove VirtualBox Devices

  1. Press Win + X → Device Manager.
  2. Go to View → Show hidden devices.
  3. Expand System devices.
  4. Right-click any “VirtualBox” entries → Uninstall device.
  5. Check “Delete the driver software” if shown.
  6. Reboot.

Fix 2: “Cannot Register Hard Disk — UUID Already Exists”

Caused by duplicate VM disk registrations.

Solution:

  1. Open VirtualBox (if installed).
  2. Go to File → Virtual Media Manager.
  3. Select duplicate disks → Remove (not “Release”).
  4. Re-add disks to VMs.

If VirtualBox won’t start:

  1. Navigate to %HOMEPATH%\.VirtualBox\
  2. Edit VirtualBox.xml
  3. Delete duplicate <HardDisk> entries.
  4. Save and restart VirtualBox.

Fix 3: VirtualBox VMs Won’t Start — “VT-x Not Available”

Even if BIOS VT-x is enabled, Hyper-V may be stealing it.

Solution A: Disable Hyper-V Completely

In PowerShell (Admin):

powershell
bcdedit /set hypervisorlaunchtype off

→ Reboot.

To re-enable later:

powershell
bcdedit /set hypervisorlaunchtype auto

Solution B: Use VirtualBox with Hyper-V Backend (Windows 11 22H2+)

VirtualBox 7+ supports running atop Hyper-V.

  1. Ensure Windows Features:

    • Hyper-V Platform → Disabled
    • Windows Hypervisor Platform → Enabled
    • Virtual Machine Platform → Enabled

  2. In VirtualBox, go to File → Preferences → System → Acceleration.

  3. Set Paravirtualization Interface to Hyper-V.

  4. Reboot.

💡 Performance is slightly lower, but coexistence is possible.

Fix 4: “Driver Not Signed” or “Publisher Could Not Be Verified”

Caused by Secure Boot or DSE blocking Oracle’s drivers.

Solution: Disable Secure Boot Temporarily

  1. Reboot → Enter BIOS/UEFI.
  2. Find Secure Boot → Set to Disabled.
  3. Save and Exit.
  4. Install VirtualBox.
  5. Re-enable Secure Boot after installation.

⚠️ Only do this if you trust the source. Re-enable Secure Boot immediately after.

Chapter 5: Advanced Fixes — Registry, Group Policy, and Scripts

When GUI fixes fail, it’s time for advanced methods.

Fix 1: Registry Fix for VMware Service Startup

If VMAuthdService fails to start:

  1. Press Win + R, type regedit, press Enter.

  2. Navigate to:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VMAuthdService

  3. Double-click Start → Change value to 2 (Automatic).

  4. Reboot.

Fix 2: Group Policy — Allow Unsigned Drivers (Enterprise Only)

For Windows 11 Pro/Enterprise:

  1. Press Win + R, type gpedit.msc, press Enter.

  2. Navigate to:

    Computer Configuration → Administrative Templates → System → Driver Installation

  3. Enable “Code signing for device drivers” → Set to “Ignore”.

  4. Reboot.

💡 Revert to “Warn” or “Block” after installation.

Fix 3: PowerShell Script to Reset Virtualization Stack

Run this script in PowerShell (Admin) to reset Hyper-V, WSL2, and drivers:

powershell
# Reset Virtualization Stack
Write-Host "Disabling Hyper-V features..." -ForegroundColor Yellow
Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All -NoRestart
Disable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform -NoRestart
Disable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform -NoRestart

Write-Host "Restarting system..." -ForegroundColor Cyan
Restart-Computer -Force

After reboot, try installing VMware or VirtualBox.

To re-enable later:

powershell
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All -NoRestart
Enable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform -NoRestart
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform -NoRestart
Restart-Computer -Force

Chapter 6: BIOS/UEFI Deep Dive — Firmware Fixes

Sometimes, the problem is deeper — in your firmware.

Fix 1: Update BIOS/UEFI

Outdated BIOS can misreport VT-x status or conflict with Windows 11.

  1. Visit your motherboard or laptop manufacturer’s website.
  2. Download latest BIOS for your exact model.
  3. Follow their flashing instructions.

💡 Use built-in utilities (ASUS EZ Flash, Dell Update) — not third-party tools.

Fix 2: Reset BIOS to Optimized Defaults

Overclocking or custom settings can break virtualization.

  1. Enter BIOS.
  2. Find “Load Optimized Defaults” or “Load Setup Defaults”.
  3. Save and Exit.
  4. Re-enable VT-x/AMD-V manually.
  5. Reboot.

Fix 3: Disable “Fast Boot” in BIOS

Fast Boot skips hardware initialization — sometimes disabling VT-x.

  1. Enter BIOS.
  2. Find Boot → Fast Boot → Set to Disabled.
  3. Save and Exit.

Chapter 7: Coexistence Strategies — Running VMware, VirtualBox, WSL2, and Docker Together

You don’t have to choose — with proper configuration, they can coexist.

Strategy 1: Use VMware with Hyper-V Backend

As mentioned earlier, VMware Workstation 17+ supports running atop Hyper-V.

Requirements:

  • Windows 11 22H2 or later
  • Hyper-V disabled
  • Windows Hypervisor Platform enabled

In VMware:

  1. Edit config.ini:

    hypervisor.cpuid.v0 = "FALSE"
    vmx.allowHv = "TRUE"

  2. Reboot.

→ VMware VMs will run using Microsoft’s hypervisor — slightly slower, but stable.

Strategy 2: Use VirtualBox with Hyper-V Backend

VirtualBox 7+:

  1. Enable Windows Hypervisor Platform and Virtual Machine Platform.
  2. Disable Hyper-V.
  3. In VirtualBox VM settings → System → Acceleration → Paravirtualization Interface → Hyper-V.

Strategy 3: Use WSL2 with GUI Apps — Replace Lightweight VMs

Instead of a full Linux VM, use WSL2:

  1. Install WSL2:

    powershell
    wsl --install

  2. Install a GUI app server:

    powershell
    wsl --install -d Ubuntu
    wsl
    sudo apt update && sudo apt install x11-apps

  3. Use Windows X Server (like VcXsrv) to display GUI apps.

→ Faster, lighter, and coexists with everything.

Chapter 8: Prevention — Avoid Future Installation Failures

An ounce of prevention > pound of cure.

1. Keep Windows Updated

Enable automatic updates. Many virtualization bugs are fixed in cumulative updates.

2. Update VMware/VirtualBox Promptly

New versions fix compatibility issues with Windows updates.

3. Backup VMs Before Major Windows Updates

Windows feature updates can break hypervisor configurations.

4. Use System Restore Points

Create one before installing/updating virtualization software.

5. Monitor Event Viewer

Check Applications and Services Logs > VMware or VirtualBox for early warnings.

6. Avoid Mixing Hypervisors Unless Necessary

If you only need one — stick with it. Coexistence adds complexity.

Chapter 9: Real-World Case Studies

Let’s look at actual scenarios and how they were resolved.

Case Study 1: Developer Can’t Install VMware After Windows 11 23H2 Update

Symptoms: Installer fails at “Starting Services.” Event Viewer shows “VMAuthdService timeout.”

Diagnosis: Memory Integrity (HVCI) enabled by Windows Update.

Solution:

  1. Disabled Memory Integrity in Windows Security.
  2. Ran VMware Install Cleaner.
  3. Rebooted → Installed VMware successfully.
  4. Re-enabled Memory Integrity (VMware still worked).

Case Study 2: Student Can’t Run VirtualBox for Class — “VT-x Not Available”

Symptoms: BIOS showed VT-x enabled, but VirtualBox claimed it wasn’t.

Diagnosis: Hyper-V enabled for WSL2.

Solution:

  1. Ran bcdedit /set hypervisorlaunchtype off
  2. Rebooted → VirtualBox worked.
  3. Later, switched to VirtualBox with Hyper-V backend to keep WSL2.

Case Study 3: IT Admin Needs VMware and Docker on Same Machine

Symptoms: Docker (WSL2) breaks VMware VMs.

Solution:

  1. Enabled “Windows Hypervisor Platform.”
  2. Disabled “Hyper-V” feature.
  3. Configured VMware to use vmx.allowHv = "TRUE".
  4. Both Docker and VMware VMs ran simultaneously.

Chapter 10: When to Seek Professional Help

Despite your best efforts, some issues require expert intervention.

Signs You Need Help:

  • BIOS doesn’t show virtualization options
  • “Driver signature” errors persist after disabling DSE
  • System crashes (BSOD) during installation
  • Physical hardware failure (CPU doesn’t support VT-x)
  • Enterprise environment with restrictive GPOs

Where to Go:

  • Manufacturer Support (Dell, HP, Lenovo, ASUS)
  • VMware Support (if licensed)
  • Oracle VirtualBox Forums
  • Microsoft Support (for Windows-level issues)
  • Local PC repair shop (for BIOS/hardware issues)

💡 Always export Event Viewer logs before seeking help.

Conclusion: Take Back Control of Your Virtualization Stack

Installation failures with VMware Workstation 17.5.2 or VirtualBox on Windows 11 are frustrating — but solvable. Whether it’s a Hyper-V conflict, driver signature block, service startup failure, or BIOS misconfiguration — this guide has equipped you with the tools and knowledge to diagnose and fix the root cause.

Remember:

✅ Always check BIOS first — is VT-x/AMD-V enabled?
✅ Disable Memory Integrity and Hyper-V if needed
✅ Clean previous installations with official tools
✅ Run installers as Administrator, with AV temporarily disabled
✅ Use “User Mode” or “Hyper-V backend” for coexistence
✅ Update BIOS and Windows regularly
✅ Create restore points before major changes

Virtualization is a cornerstone of modern computing — for development, testing, security, and legacy support. Don’t let Windows 11’s security model lock you out. With methodical troubleshooting, you can run the hypervisor of your choice — smoothly, securely, and side-by-side with Microsoft’s own tools.

Your VMs are waiting. Go bring them to life.

Final Checklist: Quick Reference Guide

☐ Enter BIOS — Enable VT-x/AMD-V, Disable CSM, Enable TPM
☐ Update Windows 11 to latest build
☐ Disable Memory Integrity (Core Isolation) temporarily
☐ Disable Hyper-V via bcdedit /set hypervisorlaunchtype off
☐ Run VMware Install Cleaner or VBoxDrvInst cleanup
☐ Delete leftover folders: Program Files, ProgramData, AppData\Local
☐ Run installer as Administrator with AV disabled
☐ For VMware: Edit config.ini → vmx.allowHv = "TRUE"
☐ For VirtualBox: Set Paravirtualization to “Hyper-V”
☐ Re-enable security features after successful install
☐ Use WSL2 for lightweight Linux needs to reduce hypervisor load

Additional Resources

Comments

Post a Comment

Below Post Ad