Skip to main content

WSL: Filesystem fast and furious?

Introduction

By now, everyone that is working long enough with WSL (a.k.a WSLinux ?) complaint at least once about a specific pain point: Filesystem performance.

 Microsoft acknowledged it several times already, and while they're aware of the problem they also know this is complicated to address as WSL is not virtualized.
I do strongly recommend to watch (again) the Channel9 video on the Filesystem implementation (here's the blog post too).
Please note that some limitations described in the video and/or the blog post are since then been addressed. But the "logic" remains.

 So while a real solution will be implemented in the future by Microsoft, here is a quick "hack" that could help on certain projects.

Our workspace is $HOME

as usual, I will target the most used directory: the Desktop folder ... hmm ... I mean the $HOME directory.
I blogged about it, on different ways you can have $HOME mounted/linked to a directory in Windows space (latest one even using Azure cloud shell), other persons blogged about it (Brian looking at you), so it's only normal I continue on this trend.

The idea

Ok, I will try to make some sense of this crazy mind I have and the "neuronal junction" that lead me to this workaround:
WSL FS (VolFS) is "slow" > WSL Interop (DrvFS) is also "slow".
Both of these FS write on the disk directly, so the next idea that was triggered is: what if I tried to write to the RAM instead? Would that be faster?
I mean, there's a lot of "in Memory" technologies now, so it might be possible to emulate it at a very small scale.

 Therefore I tried some TMPFS filesystem, but while a bit faster, I really thought it could be better somehow.

Prerequisites: Windows side

After some research and with the goal in mind to, somehow, create a "Disk in RAM" (well, it has a name, Ramdisk, duh), I found a small, but powerful, freeware that was doing exactly what I had in mind. I present you: ImDisk
Once you installed the ImDisk toolkit, you can configure it by launching the Ramdisk configuration application.

I will not share all the results with the different settings that I did, however I do encourage you to perform a fine tuning in order to find the settings that would be the best for your system.
So the following configuration is what works best for me for an "overall" usage:
  1. I created a 4Go "static disk" (the dynamic option will affect the performance) and decided to not add the Windows TEMP directory in this drive.
    For a "persistent" setup, I choose also to have the drive mounted at Startup.
  2. In the advanced tab, I picked 2KB cluster (block) size (but the difference with 4 and 8 is really small).
    The important setting that had more an impact on performance is the AWE that needs to be enabled.
  3. Finally, and this one is really important to be explained, I choose to copy the data of a specific directory.
    Please note that, a RamDisk is volatile and will be erased upon restart.
    So ensure that you don't keep important files and, just as a small safety net, enable the auto-sync upon reboot (however sometimes it doesn't work, so back to not save any imporant data)
Everything is now prepared on Windows side, so let's mount all this in WSL and see the behavior.

My $HOME in RAM

As I will mount on top of my original $HOME directory, I just copied over the basic files:

Finally, here are very basic tests of creation of small files and a big file.
Without further due, here are the tests results: 
  1. Creation of 1 Go file:
  2. Creation of and 1000 files of 1Mo:

Conclusion

while the multiple small files does not bring a real advantage (actually the DrvFS is even a bit faster when I tested it, so props to Microsoft here), the "big" advantage is shown for the bigger files.

I do encourage, if you managed to not get bored by now, to have maybe more complete tests which are (surely) out of my league.

Once again, I hope this will at least help or even better give an idea to someone and I'm really looking forward to read anyone's blog on this topic. So please share it and tag me @nunixtech


>>> Nunix Out <<<

Comments

Post a Comment

Popular posts from this blog

Docker + WSL: Get 2 daemon for the price of 1

Introduction almost two years ago, Docker announced the capability of switching between the Linux and Windows containers "mode" (far from the right click that we have today). At that time, I wrote a blog post on how to run both daemons at the same time ( http://darthnunix.blogspot.ch/2016/10/docker-for-windows-2-daemons-enter-in.html ) Fast forward to 2018, and while we were blogging on how to get the TLS connection from WSL docker client with  Rory McCune  ( https://raesene.github.io/blog/2018/03/29/WSL-And-Docker/ ), another blog post, by Stefan Stranger drew my attention (read: blew my mind) as I was trying to reproduce the same: how could I "bind" the docker socket in WSL with the Docker for Windows Linux mode socket ( https://blogs.technet.microsoft.com/stefan_stranger/2018/04/02/access-my-docker-for-windows-kubernetes-cluster-from-debian-wsl/ ) From 1 to 2 daemon: DemonHunter mode achieved Now that we have all the required setup resources, let's b...
4 comments
Read more

WSL: One Home to host them all

Introduction This blog post will explain how a single home mount can be shared accross all  the WSL instances. It was quite fun to find the idea and then make it happen. It will use all the tools currently available from WSL, so please don't expect a "Linux only" as Interopability will be heavily used. And before I start, here is my point of view about WSL: It's not only  Linux! It's different, just like GNU is different from Unix (yes I said it). We have finally the strength of both worlds combined. And while I do understand the "dev" aspect is meant to be reproducible in a Linux prod environment, the "ops" side is meant to take advantages of everything that can make the full environment feeling Home. Setup requirements In order to really enjoy this solution, I do recommend having 2 or more WSL distros installed or, if you fell like a real WSLCorsair, try @bketelsen crazy setup (I love it)! While the distros are downlo...
3 comments
Read more

VSCode + WSL: the SSH way

The initial idea early 2017, Rich Turner made a demo where Visual Studio was used to develop C++ code, however the debugging environment targeted was WSL! With some magic, he showed something that he had forbidden from the initial release of WSL: writting directly to the WSL filesystem (still a big no-no actually). Of course, this triggered a lot of curiosity and the WSL Corsair had to loot from it too! SSH to save the day the biggest challenge was to find a way that not only could be easy to setup on both ends, and also would be approved  by the WSL team.  TLDR: if you use the Windows Path, You Die! #ThereBeDragons Another way, more Linux-oriented, needed to be used and hopefully a solution already existed and was simply waiting on being used: SSH🔐 The setup the setup is done in 2 distinct parts: Setup the SSH server on WSL Setup the SSH client on VSCode  While the SSH server is a standard procedure in Linux, finding the right  plugin for VSCode w...
Post a Comment
Read more