I am seeing the different types of glusterfs volume types from replicated to distributed to distributed replicated and many more, there is no where to find out about the ones that offer the best read/write performance
https://docs.gluster.org/en/v3/Administrator%20Guide/Setting%20Up%20Volumes/
Also searched online for performance tests of these different volume types but there are barely information on this
My belief is that since glusterfs is distributed shared storage, one should be able to spread read/write across the nodes but i am not finding information about the performance aspects
So maybe people with some experience can share that.
Related
Is there a suggested way to serve hundreds of machine learning models in Kubernetes?
Solutions like Kfserving seem to be more suitable for cases where there is a single trained model, or a few versions of it, and this model serves all requests. For instance a typeahead model that is universal across all users.
But is there a suggested way to serve hundreds or thousands of such models? For example, a typeahead model trained specifically on each user's data.
The most naive way to achieve something like that, would be that each typeahead serving container maintains a local cache of models in memory. But then scaling to multiple pods would be a problem because each cache is local to the pod. So each request would need to get routed to the correct pod that has loaded the model.
Also having to maintain such a registry where we know which pod has loaded which model and perform updates on model eviction seems like a lot of work.
You can use Catwalk mixed with Grab.
Grab has a tremendous amount of data that we can leverage to solve
complex problems such as fraudulent user activity, and to provide our
customers personalized experiences on our products. One of the tools
we are using to make sense of this data is machine learning (ML).
That is how Catwalk is created: an easy-to-use, self-serve, machine
learning model serving platform for everyone at Grab.
More infromation about Catwalk you can find here: Catwalk.
You can serve multiple Machine Learning models using TensorFlow and Google Cloud.
The reason the field of machine learning is experiencing such an epic
boom is because of its real potential to revolutionize industries and
change lives for the better. Once machine learning models have been
trained, the next step is to deploy these models into usage, making
them accessible to those who need them — be they hospitals,
self-driving car manufacturers, high-tech farms, banks, airlines, or
everyday smartphone users. In production, the stakes are high and one
cannot afford to have a server crash, connection slow down, etc. As
our customers increase their demand for our machine learning services,
we want to seamlessly meet that demand, be it at 3AM or 3PM.
Similarly, if there is a decrease in demand we want to scale down the
committed resources so as to save cost, because as we all know, cloud
resources are very expensive.
More information you cna find here: machine-learning-serving.
Also you can use Seldon.
Seldon Core is an open source platform for deploying machine learning models on a Kubernetes cluster.
Features:
deploying machine learning models in the cloud or on-premise.
gaining metrics ensuring proper governance and compliance for your
running machine learning models.
creating inference graphs made up of multiple components.
providing a consistent serving layer for models built using
heterogeneous ML toolkits.
Useful documentation: Kubernetes-Machine-Learning.
I have a deep learning workstation where there are 4 GPUs with 6 GB of memory each. Would it be possible to make a docker container see the 4 GPUs as one but with 24 GB?
Thank you.
I haven't work with docker before but work a lot with CUDA with Multiple GPU. Since multiple GPUs is physically are separated, hence working with multiple GPUs required a lot of memory synchronization in code level.
I don't think that docker can virtually merge all the GPU memory as that will make the computation very complicated on the GPU side. working with Multiple GPU required custom kernel to synchronize to each other.
The best analogy I relate is, "Can you get two bare-metal computers to merge the RAM and run Microsoft Word as if it were a single machine?".
Short answer: No.
Alternate answer: Yes, but requires additional hardware, expensive, and probably incompatible with your existing hardware.
Explanation:
It is possible if your GPUs are connected using NVIDIA NVLink (take a look at the details here https://www.nvidia.com/en-us/design-visualization/nvlink-bridges/).
Usually NVLink used for pairs of GPUs, like GPU0 connected with GPU1 and GPU2 connected with GPU3, in this case best option you can obtain is 2 GPUs with doubled memory each.
Another option is special InfiniBand module, installed to modern GPU servers by some vendors.
Is the a way to physically separate between neo4j partitions?
Meaning the following query will go to node1:
Match (a:User:Facebook)
While this query will go to another node (maybe hosted on docker)
Match (b:User:Google)
this is the case:
i want to store data of several clients under neo4j, hopefully lots of them.
now, i'm not sure about whats is the best design for that but it has to fulfill few conditions:
no mixed data should be returned from a cypher query ( its really hard to make sure, that no developer will forget the ":Partition1" (for example) in a cypher query)
performance of 1 client shouldn't affect another client, for example, if 1 client has lots of data, and another client has small amount of data, or if a "heavy" query of 1 client is currently running, i dont want other "lite" queries of another client to suffer from slow slow performance
in other words, storing everything under 1 node, at some point in the future, i think, will have scalability problem, when i'll have more clients.
btw, is it common to have few clusters?
also whats the advantage of partitioning over creating different Label for each client? for example: Users_client_1 , Users_client_2 etc
Short answer: no, there isn't.
Neo4j has high availability (HA) clusters where you can make a copy of your entire graph on many machines, and then serve many requests against that copy quickly, but they don't partition a really huge graph so some of it is stored here, some other parts there, and then connected by one query mechanism.
More detailed answer: graph partitioning is a hard problem, subject to ongoing research. You can read more about it over at wikipedia, but the gist is that when you create partitions, you're splitting your graph up into multiple different locations, and then needing to deal with the complication of relationships that cross partitions. Crossing partitions is an expensive operation, so the real question when partitioning is, how do you partition such that the need to cross partitions in a query comes up as infrequently as possible?
That's a really hard question, since it depends not only on the data model but on the access patterns, which may change.
Here's how bad the situation is (quote stolen):
Typically, graph partition problems fall under the category of NP-hard
problems. Solutions to these problems are generally derived using
heuristics and approximation algorithms.[3] However, uniform graph
partitioning or a balanced graph partition problem can be shown to be
NP-complete to approximate within any finite factor.[1] Even for
special graph classes such as trees and grids, no reasonable
approximation algorithms exist,[4] unless P=NP. Grids are a
particularly interesting case since they model the graphs resulting
from Finite Element Model (FEM) simulations. When not only the number
of edges between the components is approximated, but also the sizes of
the components, it can be shown that no reasonable fully polynomial
algorithms exist for these graphs.
Not to leave you with too much doom and gloom, plenty of people have partitioned big graphs. Facebook and twitter do it every day, so you can read about FlockDB on the twitter side or avail yourself of relevant facebook research. But to summarize and cut to the chase, it depends on your data and most people who partition design a custom partitioning strategy, it's not something software does for them.
Finally, other architectures (such as Apache Giraph) can auto-partition in some senses; if you store a graph on top of hadoop, and hadoop already automagically scales across a cluster, then technically this is partitioning your graph for you, automagically. Cool, right? Well...cool until you realize that you still have to execute graph traversal operations all over the place, which may perform very poorly owing to the fact that all of those partitions have to be traversed, the performance situation you're usually trying to avoid by partitioning wisely in the first place.
I am working on building a recommendation system.
And I found that neo4j and mahout are the best available solution for building recommendation system.
But I am unable to clearly distinguish between the advantage and issues related to these two.
Could anyone provide me a comparison between neo4j and mahout ?
Which is best for building a recommendation system ?
This is a deep question that can't be specifically answered without a lot more information about what kind of recommendation system you want, what kind of scalability you need, what your hardware/software/platform configuration looks like, and so on.
But let me give you this to consider: the issue isn't a comparison of neo4j and mahout, it's a comparison between neo4j and hadoop. Mahout would generally run on top of a hadoop infrastructure, so all of the assumptions and implementation constraints that come with hadoop are for the most part more important to consider than just the raw graph layer implementation differences.
You should read up more on hadoop - the approach it takes to data storage and management is completely different than what neo4j does. Both have their pros and cons, and entire books can and have been written about the pros and cons of the approach hadoop takes. The brief version is that if you need massive scalability and you have huge datasets, hadoop is set up to work well in those environments; neo4j doesn't promise scalability beyond billions or tens of billions of nodes, and the kinds of scalability that it does promise are quite different than hadoop.
You should do some more reading on this topic, or further refine your question to prevent people from getting into opinionated flamewars about what is better. Maybe there is a better choice for you in consideration of all of your details, but we don't have those details.
It looks like that RADOS is best suited to be used as the storage backend for Ceph Block Storage and File System. But if i want to use the Object Storage itself:
Is there an optimum object size which gives the best performance?
Is there a problem with a large number of small objects?
How big objects can get without making troubles?
It would be great if you can share your experience.
There is no optimal size for objects in the object store, in fact this flexibility is one of the big benefits over fixed-size block stores. Typically an application will use this flexibility to decompose its data models along convenient boundaries. That said, if you are storing very small or very large objects, you should take into account some considerations.
Is there a problem with a large number of small objects?
There has never been a functional problem with small objects, though in the past it has been inefficient due to the way that objects are stored. However, in the next release of Ceph (Firefly) there is a way to use LevelDB as a backend, making small objects much more efficient.
How big objects can get without making troubles?
Assuming that you are using replication in RADOS (in contrast to the proposed object striping feature and the erasure coding backend) an object is replicated in its entirety to a set of physical storage nodes. Thus, the size of an object has an inherent limitation in size based on the storage capacity of the physical nodes to which the object is replicated.
This mode of operation also alludes to the practical limitation that per-object I/O performance will correspond to the performance of the physical devices (data and journal drives). This means that it is often useful to think of an object as a unit of I/O parallelism, although in practice many objects will map to the same set of devices.
This question will likely have a different answer for the erasure coded backend, and applications can always stripe large datasets across smaller objects.