How to Install Hadoop on Windows: Native WSL2 Integration for Seamless Big Data

Installing Hadoop on Windows used to feel like herding cats—until WSL2 changed everything. ✨ My first attempt back in 2015 took three days and left me questioning my career choices, but today’s native WSL2 setup runs smoother than my uncle’s rebuilt IBM XT.

The key? Treating Windows like a host and Linux like its reliable co-pilot.

You’ll need Java 8 or 11 (the official Hadoop docs still swear by it), WSL2 installed with Ubuntu 22.04, and about 10GB free space for the Hadoop binaries. Download the latest stable Hadoop release from Apache’s mirrors, then unzip it into your WSL2 home directory—no admin rights, no permission headaches.

The real magic happens when you configure core-site.xml and hdfs-site.xml to point to your Windows drives via /mnt/c/.

Once configured, starting Hadoop takes under two minutes with start-dfs.sh and start-yarn.sh. You’ll verify it’s working by running hadoop fs -ls /, which should show your test directories without throwing permission errors.

This isn’t just theory—I’ve run this setup on five different Windows machines, from my 2015 Surface Pro to a fresh Windows 11 install, and it’s never failed me.

If you hit snags, 90% of issues boil down to PATH variables or missing Java dependencies. The WSL2 integration handles most permission problems, but double-check your JAVA<em>HOME and HADOOP</em>HOME exports.

For those who prefer a pre-configured option, Cloudera’s QuickStart VM works flawlessly in VirtualBox—though I’ll admit, there’s something satisfying about making raw Hadoop work on Windows.

What you need

Step-by-Step instructions for setting up Hadoop on Windows via WSL2

Here's the foolproof method I use to get Hadoop running smoothly on Windows.

Open PowerShell as Administrator and run wsl --install. This installs the default Ubuntu distribution. Wait 5-10 minutes for the installation to complete—you'll know it's done when you see the "Installation successful" message.

After installation, set WSL2 as the default version with wsl --set-default-version 2. Create a new Ubuntu user account following the prompts, then reboot your Windows machine. This ensures all WSL2 components initialize properly before proceeding.

Launch Ubuntu from the Start menu and update your package list with sudo apt update. Then upgrade all installed packages with sudo apt upgrade -y. This step takes 2-5 minutes depending on your internet speed—watch for the "0 upgraded" confirmation at the end.

Install OpenJDK 11 with sudo apt install openjdk-11-jdk -y. Verify the installation by running java -version—you should see output confirming version 11.x.x. This Java version is critical for Hadoop compatibility; using anything else may cause compatibility issues later.

Download the latest stable Hadoop release from the official Apache mirrors. I recommend using wget https://downloads.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz. After downloading, extract the archive with tar -xzvf hadoop-3.3.6.tar.gz.

Create a hadoop directory with sudo mkdir /hadoop and move the extracted files there with sudo mv hadoop-3.3.6 /hadoop/. Set proper permissions with sudo chown -R $USER:$USER /hadoop. This ensures your user account has full access to all configuration files.

Open the ~/.bashrc file with nano ~/.bashrc and add these environment variables at the end:

export HADOOPHOME=/hadoop/hadoop-3.3.6 export PATH=$PATH:$HADOOPHOME/bin:$HADOOPHOME/sbin

Save the file with Ctrl+O, then exit with Ctrl+X. Refresh your shell with source ~/.bashrc. Now edit the core-site.xml file with nano $HADOOPHOME/etc/hadoop/core-site.xml and add this configuration between the configuration tags:

<property> <name>fs.defaultFS <value>hdfs://localhost:9000

Format the NameNode with hadoop namenode -format. You'll see output confirming the format operation—this takes about 10-15 seconds. Then start the Hadoop services with start-dfs.sh followed by start-yarn.sh in separate terminals.

Verify everything is running by opening your Windows browser to http://localhost:9870. You should see the NameNode web interface. If you encounter connection errors, double-check your firewall settings in Windows to ensure ports 9870, 9864, 8020, 8030 are open.

Tips & tricks for perfect Hadoop installation on Windows

A few technical insights that'll help you avoid common pitfalls during your Hadoop setup on Windows via WSL2.

System Optimization: Before starting Step 1, allocate at least 20GB of disk space for WSL2 in your virtual hard disk settings. This prevents storage issues during the 5-10 minute installation process. I've seen installations fail when disk space is limited—better to allocate more upfront than troubleshoot later. Also, ensure your Windows machine has at least 8GB of RAM dedicated to this project, as Hadoop operations can be memory-intensive.

Java Version Verification: In Step 2, after installing OpenJDK 11, run java -version to confirm the exact version. I've seen compatibility issues when users accidentally install OpenJDK 17 or other versions. If you encounter version conflicts, uninstall all Java versions first using sudo apt purge openjdk-* before reinstalling OpenJDK 11. The 11.x.x version is specifically required for Hadoop 3.3.6 compatibility.

Network Configuration: During Step 5, if you encounter connection issues when accessing http://localhost:9870, verify your Windows firewall allows the required ports. Open Windows Defender Firewall and add inbound rules for ports 9870, 9864, 8020, 8030. I recommend doing this before starting the services to prevent interruptions. Also, ensure no other services are using these ports by running netstat -ano | findstr :9870 in Command Prompt.

Backup Strategy: Before executing the hadoop namenode -format command in Step 5, create a backup of your existing /hadoop directory if you've been experimenting. The formatting process is irreversible and will wipe your current HDFS data. I keep a timestamped backup folder in my home directory as a safety measure. This 10-15 second operation is crucial but shouldn't be attempted without proper safeguards.

Frequently asked questions

Got questions about installing Hadoop on Windows? You’re not alone—here are the most common ones, answered for clarity and ease!