I am attempting to utilise TensorFlow Federated for an image classification task with 7 classes and 3-5 clients. Each client has a different class distribution of labels. I have successfully implemented this tutorial for my use-case and am now looking for improvements. I have a few questions:
Can individual clients have different class weights in their loss function based on the class distribution that is unique to that client?
If so, how would one implement this?
If not, is it because federated averaging process requires that the clients and the global model share the same loss function?
If i understand your question, I can say yes, individual clients can have different class weight, in this case we talk about non iid data . Suppose that we have 7 labels, each client have data from 1 or 2 labels.
Related
I built a machine learning (ML) model to classify real-time network traffic as an attack or normal traffic using a dataset consisting of approximately 3 million records.
Then, I built a second ML model to classify the real-time network traffic according to their application, i.e., Google, Facebook, YouTube, etc. using another dataset consisting of approximately 1.5 million records.
Now I want to cascade these two models so that if the traffic is normal, then the traffic should be classified by the second ML model. Otherwise, it should be discarded since there is no need to pass through the second model.
Can I cascade these two models even though they are built using different datasets? And if so, how can I do that?
I do the cascading logic simply in a programming language code C++ or Python, not using ML-tool features. If the data from the second model, doesn't contribute to the decision of the first model - just keep the models separated.
I just started studying federated learning and want to apply it to a certain dataset, and there are some questions that have risen up.
My data is containing records of 3 categories, each of which is having 3 departments. I am planning to have 3 different federated learning models for each category and treat the three department of this category as the distributed clients.
Is this possible? or building federated learning models requires having thousands of clients?
Thanks
Difficult to say by what you have provided in your question. Usually, when building a federated learning system, you are extending your centralized approach to one with data split/partitioned between segregated clients. Again, depending on the type of data you have, the type of task you are trying to solve and also the amount of data required to solve the task in a centralized approach, these factors along with other ones will depend how many clients you can use and how much data is required at each client. Additionally, the aggregation method that you wish to use combine the parameters from different clients will affect this. I suggest experimenting with different client numbers and partitioning methods and seeing what suits your needs.
I practice on this tutorial, I would like that each client train a different architecture and different model, Is this possible?
TFF does support different clients having different model architectures.
However, the Federated Learning for Image Classification tutorial uses tff.learning.build_federated_averaging_process which implements the Federated Averaging (McMahan et. al 2017) algorithm, defined as each client receiving the same architecture. This is accomplished in TFF by "mapping" (in the functional programming sense) the model to each client dataset to produce a new model, and then aggregating the result.
To achieve different clients having different architectures, a different federated learning algorithm would need to be implemented. There are couple (non-exhaustive) ways this could be expressed:
Implement an alternative to ClientFedAvg. This method applies a fixed model to the clients dataset. An alternate implementation could potentially create a different architecture per client.
Create a replacement for tff.learning.build_federated_averaging_process
that uses a different function signature, splitting out groups of clients
that would receive different architectures. For example, currently FedAvg
looks like:
(<state#SERVER, data#CLIENTS> → <state#SERVER, metrics#SERVER>
this could be replaced with a method with signature:
(<state#SERVER, data1#CLIENTS, data2#CLIENTS, ...> → <state#SERVER, metrics#SERVER>
This would allow the function to internally tff.federated_map() different model architectures to different client datasets. This would likely only be useful in FL simulations or experimentation and research.
However, in federated learning there will be difficult questions around how to aggregate the models back on the server into a single global model. This probably needs to be designed out first.
I noticed that the Gradient Quantization compression method is already implemented in TFF framework. How about non-traditional compression methods where we select a sub-model by dropping some parts of the global model? I come across the "Federated Dropout" compression method in the paper "Expanding the Reach of Federated Learning by Reducing Client Resource Requirements" (https://arxiv.org/abs/1812.07210). Any idea if Federated Dropout method is already supported in Tensorflow Federated. If not, any insights how to implement it (the main idea of the method is dropping a fixed percentage of the activations and filters in the global model to exchange and train a smaller sub-model)?
Currently, there is no implementation of this idea available in the TFF code base.
But here is an outline of how you could do it, I recommend to start from examples/simple_fedavg
Modify top-level build_federated_averaging_process to accept two model_fns -- one server_model_fn for the global model, one client_model_fn for the smaller sub-model structure actually trained on clients.
Modify build_server_broadcast_message to extract only the relevant sub-model from the server_state.model_weights. This would be the mapping from server model to client model.
The client_update may actually not need to be changed (I am not 100% sure), as long as only the client_model_fn is provided from client_update_fn.
Modify server_update - the weights_delta will be the update to the client sub-model, so you will need to map it back to the larger global model.
In general, the steps 2. and 4. are tricky, as they depend not only what layers are in a model, but also the how they are connected. So it will be hard to create a easy to use general solution, but it should be ok to write these for a specific model structure you know in advance.
We have several compression schemas implemented in our simulator:
"FL_PyTorch: Optimization Research Simulator for Federated Learning."
https://burlachenkok.github.io/FL_PyTorch-Available-As-Open-Source/
https://github.com/burlachenkok/flpytorch
FL_PyTorch is a suite of open-source software written in python that builds on top of one of the most popular research Deep Learning (DL) frameworks PyTorch. We built FL_PyTorch as a research simulator for FL to enable fast development, prototyping, and experimenting with new and existing FL optimization algorithms. Our system supports abstractions that provide researchers with sufficient flexibility to experiment with existing and novel approaches to advance the state-of-the-art. The work is in proceedings of the 2nd International Workshop on Distributed Machine Learning DistributedML 2021. The paper, presentation, and appendix are available in DistributedML’21 Proceedings (https://dl.acm.org/doi/abs/10.1145/3488659.3493775).
To resolve the problem of non-iid data in federated learning, I read a paper which add a new node with a different data domain and transfer knowledge from decentralized nodes. My question is what is the information transfered, is that updates or data ?
In layman terms, non-idd means that not all class labels are distributed evenly between clients for training. For obvious reasons, in federated environment it is not feasible for every client to hold and train on idd data. With regards your specific query of how it works in the paper mentioned in your question, you may please share the link of the paper.