VersionMismatchWarning: Mismatched versions found - blosc - dask

Question

I cannot do a 'pip install blosc' on windows. I devop on windows and have my workers and schedule running on vm's with dask-docker. Anyone have any ideas? Seem like dask really wants all linux all the time.
blosc
+-----------------------+---------+
| | version |
+-----------------------+---------+
| client | None |
| scheduler | 1.9.1 |
| tcp://127.0.0.1:38323 | 1.9.1 |
+-----------------------+---------+
(venv) D:\dev\code\datacrunch>pip install -U blosc
Collecting blosc
Using cached blosc-1.9.1.tar.gz (809 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Building wheels for collected packages: blosc
Building wheel for blosc (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: 'd:\dev\code\netsense.support\datacrunch\venv\scripts\python.exe' 'd:\dev\code\netsense.support\datacrunch\venv\lib\site-packages\pip_vendor\pep517_in_process.py' build_wheel 'C:\Users\H166631\AppData\Local\Temp\tmpwgt4t634'
cwd: C:\Users\H166631\AppData\Local\Temp\pip-install-r1476vwy\blosc
Complete output (162 lines):
Not searching for unused variables given on the command line.
-- The C compiler identification is unknown
CMake Error at CMakeLists.txt:3 (ENABLE_LANGUAGE):
No CMAKE_C_COMPILER could be found.

Answer 1

The compression has to match throughout the dask cluster and because you don't have blosc installed you run into some issues. As a side note, there is an effort to improve messaging of the error in PR #3742 . I can think of two solutions:
Switch to conda instead of pip (though this is perhaps a non-starter for you)
Use a different compression (one that you have installed or can easily install on your machine)
For 2. you can either set the compression programmatically like the following:
In [1]: import dask
In [2]: import distributed
In [3]: dask.config.set({'distributed.comm.compression': 'lz4'})
Or on the CLI:
DASK_DISTRIBUTED__COMM__COMPRESSION=zlib dask-worker
Or with with the dask config file. For more info, I would recommend reading through: https://docs.dask.org/en/latest/configuration.html and https://docs.dask.org/en/latest/configuration-reference.html#distributed.comm.compression

Answer 2

You can always just not install blosc on your linux machines. Dask is happy to run on Windows. It's even happy (to a certain extent) to mix between Windows and Linux. But it's not happy if you have libraries on some of your machines that you don't have on others. Library uniformity is key.