Skip to main content

WSL: Filesystem fast and furious?

Introduction

By now, everyone that is working long enough with WSL (a.k.a WSLinux ?) complaint at least once about a specific pain point: Filesystem performance.

Microsoft acknowledged it several times already, and while they're aware of the problem they also know this is complicated to address as WSL is not virtualized.
I do strongly recommend to watch (again) the Channel9 video on the Filesystem implementation (here's the blog post too).
Please note that some limitations described in the video and/or the blog post are since then been addressed. But the "logic" remains.

So while a real solution will be implemented in the future by Microsoft, here is a quick "hack" that could help on certain projects.

Our workspace is $HOME

as usual, I will target the most used directory: the Desktop folder ... hmm ... I mean the $HOME directory.
I blogged about it, on different ways you can have $HOME mounted/linked to a directory in Windows space (latest one even using Azure cloud shell), other persons blogged about it (Brian looking at you), so it's only normal I continue on this trend.

The idea

Ok, I will try to make some sense of this crazy mind I have and the "neuronal junction" that lead me to this workaround:
WSL FS (VolFS) is "slow" > WSL Interop (DrvFS) is also "slow".
Both of these FS write on the disk directly, so the next idea that was triggered is: what if I tried to write to the RAM instead? Would that be faster?
I mean, there's a lot of "in Memory" technologies now, so it might be possible to emulate it at a very small scale.

Therefore I tried some TMPFS filesystem, but while a bit faster, I really thought it could be better somehow.

Prerequisites: Windows side

After some research and with the goal in mind to, somehow, create a "Disk in RAM" (well, it has a name, Ramdisk, duh), I found a small, but powerful, freeware that was doing exactly what I had in mind. I present you: ImDisk
Once you installed the ImDisk toolkit, you can configure it by launching the Ramdisk configuration application.

I will not share all the results with the different settings that I did, however I do encourage you to perform a fine tuning in order to find the settings that would be the best for your system.
So the following configuration is what works best for me for an "overall" usage:
  1. I created a 4Go "static disk" (the dynamic option will affect the performance) and decided to not add the Windows TEMP directory in this drive.
    For a "persistent" setup, I choose also to have the drive mounted at Startup.
  2. In the advanced tab, I picked 2KB cluster (block) size (but the difference with 4 and 8 is really small).
    The important setting that had more an impact on performance is the AWE that needs to be enabled.
  3. Finally, and this one is really important to be explained, I choose to copy the data of a specific directory.
    Please note that, a RamDisk is volatile and will be erased upon restart.
    So ensure that you don't keep important files and, just as a small safety net, enable the auto-sync upon reboot (however sometimes it doesn't work, so back to not save any imporant data)
Everything is now prepared on Windows side, so let's mount all this in WSL and see the behavior.

My $HOME in RAM

As I will mount on top of my original $HOME directory, I just copied over the basic files:

Finally, here are very basic tests of creation of small files and a big file.
Without further due, here are the tests results: 
  1. Creation of 1 Go file:
  2. Creation of and 1000 files of 1Mo:

Conclusion

while the multiple small files does not bring a real advantage (actually the DrvFS is even a bit faster when I tested it, so props to Microsoft here), the "big" advantage is shown for the bigger files.

I do encourage, if you managed to not get bored by now, to have maybe more complete tests which are (surely) out of my league.

Once again, I hope this will at least help or even better give an idea to someone and I'm really looking forward to read anyone's blog on this topic. So please share it and tag me @nunixtech


>>> Nunix Out <<<

Comments

Popular posts from this blog

Scripting with GO: what about Windows?

!!! Warning !!!Once again, let me repeat, I'm by no means a developer nor a sysadmin (at least no more since 5 years).
This means that, while the solution below worked for me and I could reproduce it, it's not 100% safe neither!
Also, my solution still feels incomplete (read further), so if someone know the tweak to make it even more compete, please share it and reach out in Twitter, I'll be waiting.
IntroductionBy now, everyone following @jessfraz read her take about the very neat "little hack" from @cloudflareUsing Go as a scripting language in Linux

As a good WSL Corsair, I tried it and it just worked!
But then, based on my latest playground of having a seamless coding experience between WSL and Windows (thanks @bketelsen for the love and notification spam), I really thought it would be unfair to "my Windows side" to not be able enjoying the fun of GO scripting (yes, split personality is bad).

So as the good Corsair I am, it was time to see if some shinn…

Docker + WSL: Get 2 daemon for the price of 1

Introductionalmost two years ago, Docker announced the capability of switching between the Linux and Windows containers "mode" (far from the right click that we have today).
At that time, I wrote a blog post on how to run both daemons at the same time (http://darthnunix.blogspot.ch/2016/10/docker-for-windows-2-daemons-enter-in.html)

Fast forward to 2018, and while we were blogging on how to get the TLS connection from WSL docker client with Rory McCune (https://raesene.github.io/blog/2018/03/29/WSL-And-Docker/), another blog post, by Stefan Stranger drew my attention (read: blew my mind) as I was trying to reproduce the same: how could I "bind" the docker socket in WSL with the Docker for Windows Linux mode socket (https://blogs.technet.microsoft.com/stefan_stranger/2018/04/02/access-my-docker-for-windows-kubernetes-cluster-from-debian-wsl/)
From 1 to 2 daemon: DemonHunter mode achievedNow that we have all the required setup resources, let's bind them together. An…

WSL: One Home to host them all

IntroductionThis blog post will explain how a single home mount can be shared accross all the WSL instances. It was quite fun to find the idea and then make it happen. It will use all the tools currently available from WSL, so please don't expect a "Linux only" as Interopability will be heavily used. And before I start, here is my point of view about WSL: It's not only Linux!
It's different, just like GNU is different from Unix (yes I said it).We have finally the strength of both worlds combined. And while I do understand the "dev" aspect is meant to be reproducible in a Linux prod environment, the "ops" side is meant to take advantages of everything that can make the full environment feeling Home. Setup requirementsIn order to really enjoy this solution, I do recommend having 2 or more WSL distros installed or, if you fell like a real WSLCorsair, try @bketelsencrazy setup (I love it)!
While the distros are downloading, let's prepare the Windo…