With respect specifically to CatBoost:
Under what scenarios might one want to use fewer than the max number of threads of one's CPU? I cannot find an answer to this.
Is there a fixed cost/overhead associated with each core utilized? I.e., is more always better for all data set types/sizes?
Do the answers to the questions above generalize to all machine learning algorithms?
I think that most of the reasons for changing the thread_count are not catboost specific. Other libraries like sklearn offer the same feature. Reasons for not running with all CPUs are:
Debugging: If there is a problem it might be handy to only have one thread thus making the process more simple.
You want other processes on your machine to have CPU power. Especially if you have a server for in-memory data analysis shared by a team of data scientists. Your colleagues won't be happy if you take all resources.
Your job is so small that it simply does not need all the resources.
Your parallelize in another way: For example you try different hyper parameters using cross validation. Then it would make sense to dedicate one CPU to training one model rather than training a model with with all CPUs and then move on to train the next model with all CPUs
I hope this answers question 1. This generalizes to other in-memory ml libraries like sklearn.
Regarding question 2 I'm not sure. CatBoost does the parallelisation somewhere in its C++ Code and uses it via Cython in the Python package. I assume it introduces some overhead (since distributed computing always introduces overhead) but it's probably not too much. You could find out by timing some experiments.
I love the MPC module in Drake, providing numerous out-of-the-box cost and constraint functions. The three system building methods I know are importing from a urdf file, LinearSystem, and SymbolicVectorSystem. However, it's hard to include a neural network into the dynamics through them. Is there any suggestion here?
You could use other framework ,like pytorch, to add a neural network to your control system written by drake. They don't conflict.
I have started using Julia.I read that it is faster than C.
So far I have seen some libraries like KNET and Flux, but both are for Deep Learning.
also there is a command "Pycall" tu use Python inside Julia.
But I am interested in Machine Learning too. So I would like to use SVM, Random Forest, KNN, XGBoost, etc but in Julia.
Is there a native library written in Julia for Machine Learning?
Thank you
A lot of algorithms are just plain available using dedicated packages. Like BayesNets.jl
For "classical machine learning" MLJ.jl which is a pure Julia Machine Learning framework, it's written by the Alan Turing Institute with very active development.
For Neural Networks Flux.jl is the way to go in Julia. Also very active, GPU-ready and allow all the exotics combinations that exist in the Julia ecosystem like DiffEqFlux.jl a package that combines Flux.jl and DifferentialEquations.jl.
Just wait for Zygote.jl a source-to-source automatic differentiation package that will be some sort of backend for Flux.jl
Of course, if you're more confident with Python ML tools you still have TensorFlow.jl and ScikitLearn.jl, but OP asked for pure Julia packages and those are just Julia wrappers of Python packages.
Have a look at this kNN implementation and this for XGboost.
There are SVM implementations, but outdated an unmaintained (search for SVM .jl). But, really, think about other algorithms for much better prediction qualities and model construction performance. Have a look at the OLS (orthogonal least squares) and OFR (orthogonal forward regression) algorithm family. You will easily find detailed algorithm descriptions, easy to code in any suitable language. However, there is currently no Julia implementation I am aware of. I found only Matlab implementations and made my own java implementation, some years ago. I have plans to port it to julia, but that has currently no priority and may last some years. Meanwhile - why not coding by yourself? You won't find any other language making it easier to code a prototype and turn it into a highly efficient production algorithm running heavy load on a CUDA enabled GPGPU.
I recommend this quite new publication, to start with: Nonlinear identification using orthogonal forward regression with nested optimal regularization
Is there software out there that optimises the best combination of learning rate, weight ranges, hidden layer structure, for a certain task? After presumably trying and failing different combinations? What is this called? As far as I can tell, we just do it manually at the moment...
I know this is not differently code related but am sure it will help many others too. Cheers.
The above comes under multi variate optimization problem, use an optimization algorithm and check the results. Particle Swarm Optimization would do it ( there are however considerations to use this algorithm) as long as you have a cost function to optimize for example the error rate of the network output
I'm not looking for a Neural Networks library, since I'm creating new kinds of networks. For that I need a good "dataflow" language.
Of course you can do this in C, C++, Java and co. but dealing from scratch with the multithreading etc. would be a nightmare.
At the other extremity, languages like Oz or Erlang seem more adapted, but they don't have many libraries, and they are harder to master (it's easy to play with them, but is it OK to create complete software ?).
What would you suggest ?
I watched an interesting conference presentation about using Erlang for Neural Networks. You might want to check it out:
From Telecom Networks to Neural Networks; Erlang, as the unintentional Neural Network Programming Language
I also know that the presented system is going to be open-sourced any day now according the authors tweet.
Erlang is very well suited for NN.
Neurons can be modeled by processes (no problem with having millions of them)
Connections/synapses can be represented by PIDs of target neuron. It is very easy to initialize such a network as part of standard init procedure in OTP. Communication would be realized by message passing.
Maybe it would be good to have global address space in ETS/mnesia (build in datastores) in order to do dynamic reconfiguration of network structure.
Pattern matching in receive block can determine what kind of signal neuron receives and modify it on the fly.
It would be very easy to monitor such a network.
Also consider that Erlang NN would be 'live' all the time. You would be able to query neurons, layers, routers etc any time.
In C/C++ you just read current state of arrays/data structure.
Regarding performance, we all know that C/C++ is orders of magnitude faster than Erlang,
however NN topic is tricky.
If the network would hold very few neurons, in very wide address space, in regular array,
iterating over it again and again could be costly (in C). Equivalent situation in Erlang would be solved by single query to root/roots (input layer) neurons, which would propagate query directly to well addressed neighborhs.
DXNN1, and DXNN2 which was built and introduced in the textbook: Handbook of Neuroevolution Through Erlang: http://www.amazon.com/Handbook-Neuroevolution-Through-Erlang-Gene/dp/1461444624/ref=zg_bs_760204_22
Are open source and available at: https://github.com/CorticalComputer
If you are interested in data flow programming and multi-threading then I would suggest National Instruments LabVIEW. In this case you don't need to bother about multi-threading since its already there and you can also use OOP since now OOP is also native with LabVIEW. LabVIEW OOP is also purely based on data flow programming paradigm.
If you have any Java experience, then use Scala which is a JVM language that is based on the same concept of "actors" as Erlang. But it is less strict than Erlang and can easily use any existing Java libraries.
Then, when you find a computationally expensive task that would work better in Erlang, you can use Erlang's jinterface library to communicate between your Scala code and your distributed Erlang nodes.
Using Java does not mean dealing from scratch with multithreading - just use one of numerous Java Actor Libraries.
It's not a language in and of itself, but Emergent is very powerful and can be highly customized (it has a full scripting language).
It's open source, too, which could be helpful as a guide if you need to make your own version for your novel architectures.
Why reinvent the wheel? Try PyBrain. It's free and very comprehensive:
Quickstart
Another big plus for Erlang is full integration with Drakon
http://drakon-editor.sourceforge.net/drakon-erlang/intro.html
It all depends on your application. C++, Python are some good programming languages for machine learning