Chatbots that use traditional machine learning models like ChatGPT and Claude AI are trained by data mined from millions of users. But as calls for data privacy and AI safety continue to mount, AI is shifting away from data scraping in favor of decentralized techniques. Newer AI models are being trained collaboratively — using raw data that never leaves its source device, be it a smartphone, laptop or IoT device.
This decentralized approach is called federated learning, which allows edge devices — devices at the periphery of a network and close to data sources — to collaboratively train models without sharing their data. The method enables organizations to protect data while leveraging the power of machine learning.
Federated Learning Definition
Federated learning is a distributed machine learning technique that allows devices to train an AI model collaboratively while avoiding sharing data with a central server. Each device trains a local model on its data and shares only model updates, ensuring privacy and data security while advancing the machine learning process.
What Is Federated Learning?
Federated learning is a decentralized way to train machine learning models by involving edge devices that collaboratively train models without sharing their data. The technique goes hand-in-hand with edge computing, which is the practice of bringing data storage and applications closer to data sources. Edge computing removes the need to constantly transfer data, reducing latency and increasing the efficiency of federated learning. In return, federated learning gives edge devices the ability to process larger volumes of data.
Google introduced federated learning as a term in 2017. The development came in the aftermath of major data security breaches like the Cambridge Analytica-Facebook scandal, after which the public took a heightened interest in protecting their privacy.
“One of the challenges in modern machine learning is the need for large amounts of data to build modern-scale learning models,” said Suhas Diggavi, a computer engineering professor at the University of California, Los Angeles. “There [are] a lot of legitimate concerns about users' data being used for this purpose.”
Since then, other major tech players have jumped into the federated learning space. In particular, Apple has strived to keep pace with Google. After using federated learning to train Siri, Apple has expanded its use of the technique in areas like neural networks, tokenizer training and automatic speech recognition.
Today, common examples of federated learning in practice include predictive tools like Gboard’s next-word prediction, emoji suggestion and autocorrect. Google uses federated learning to improve Gboard’s performance without sending personal data — like text conversations — to its central servers. This approach protects user privacy while optimizing model accuracy.
“You probably don’t want every word you type on your keyboard stored in the cloud — in federated learning, [your phone] performs small computations with local data, [which it sends as a training update],” said Justin Kang, a computer science PhD candidate at the University of California, Berkeley. “This local update is nearly impossible for a human to interpret.”
How Does Federated Learning Work?
Federated learning decouples machine learning from private data storage, relying on decentralized devices like smartphones or IoT sensors. After local data updates hone the global model, the improvements are distributed back to the devices for further local training.
“This process is repeated multiple times until the model reaches a desired level of accuracy or performance,” said Salman Avestimehr, Tensor Opera AI CEO and University of Southern California professor.
Federated Learning Steps
The federated learning process involves the following steps:
- Initialization: A global model is initialized on a central server.
- Local training: The global model is copied and distributed among client devices, which train the global model only on their local data to create an updated model.
- Update sharing: Client devices share model updates with the central server.
- Model aggregation: The server aggregates the updates from various devices to improve the global model.
- Global model update: The improved global model is returned to the devices, and the process repeats.
Federated Learning Frameworks
Multiple frameworks have been developed to support federated learning across various devices and networks. Frameworks provide the necessary APIs to develop real-world applications. Examples include:
- TensorFlow Federated: Developed by Google, TFF is an open-source framework for building federated learning applications.
- PySyft: PySyft is a federated learning framework built on PyTorch that supports privacy-preserving technologies like secure multi-party computation and differential privacy.
- Flower: Flower is an open-source federated learning framework designed to integrate with machine learning libraries like TensorFlow and PyTorch.
Types of Federated Learning
There are several different approaches to federal learning, but the following are some of the main methods for performing this process.
Centralized Federated Learning
Sometimes referred to as server-based federal learning, centralized federal learning uses a central server to orchestrate the entire process. The central server copies and shares the global model among client devices, receives updates after the local training phase and uses these updates to improve the global model. While coordinating communications with a central server is efficient, the whole process is affected if anything happens to the central server.
Decentralized Federated Learning
Decentralized federated learning forgoes the use of a central server. Instead, the client devices communicate with each other, both training a global model and sending and receiving updates. In this setup, devices often use blockchain technology to form a peer-to-peer network, with each device coming to store a refined global model through their various interactions with other client devices. This approach removes bottlenecks and single points of failure in an attempt to be more secure, although a network of devices presents more openings for disruptions.
Heterogeneous Federated Learning
Heterogeneous federated learning uses client devices with varying features and capabilities for creating updated models, including computers, mobile phones and IoT devices. Adaptive algorithms are employed to process the different types of data from these devices, resulting in a global model that can adjust to various contexts. This method is ideal when applying federated learning to real-world use cases.
Cross-Silo Federated Learning
Cross-silo federated learning involves organizations instead of client devices. Compared to individual devices, organizations have much larger amounts of data and more reliable connections, enhancing the learning process. A group of institutions may then establish formal agreements to train and update a global model using their data. This allows all parties to access a more powerful global model while maintaining privacy and control over their data.
Applications for Federated Learning
Federated learning use cases fall into two categories: cross-device learning and cross-silo learning:
Cross-Device Applications
“In cross-device learning, the goal is to train AI models using data collected from various edge devices,” Avestimehr told Built In. Examples include:
- Smartphones: One of the earliest applications of federated learning is smartphones, particularly in personalized services like keyboard predictions, voice assistants and personalized photo stories. For example, Apple copies and sends its speech-recognition models to individual iPhones or iPads to train Siri on local device data.
- Transportation: To improve autonomous driving systems, data from multiple vehicles are used to train models for object detection, route planning, traffic prediction and EV battery range prediction without sharing sensitive information. Waymo uses this approach, having each vehicle gather data from its environment and share updates rather than the raw data.
- Manufacturing: Federated learning can optimize manufacturing processes by leveraging data from IoT devices to train predictive maintenance models.
Cross-Silo Applications
“In cross-silo learning, federated learning is used to train models on data held by different organizations or institutions, often referred to as silos,” Avestimehr said. In this scenario, the data remains local while institutions collaborate to strengthen the robust model. Examples include:
- Healthcare: Sensitive medical data, like patient records and diagnostic imaging, cannot be shared across institutions due to privacy regulations like HIPAA. Federated learning can enable hospitals and research institutions to collaborate on building more accurate predictive models for disease detection, personalized treatment plans and drug discovery. For instance, researchers are using an NVIDIA tool that uses federated learning to enhance AI imaging, improving cancer detection.
- Financial institutions: Banks and financial services companies utilize federated learning to detect fraud across multiple locations without exposing customer data. Lucinity offers federal learning technology designed to deliver anti-money laundering insights to financial institutions.
Challenges of Federated Learning
Common pitfalls of federated learning include:
Communication Efficiency
Since model updates must be exchanged between devices and the central server — or between devices in decentralized architectures — bandwidth and latency can significantly affect machine learning performance. Techniques like model compression, quantization and asynchronous updates can reduce the communication load, but these solutions can introduce trade-offs in model accuracy.
Data Protection
Although federated learning enhances data privacy by keeping data local, research has shown that AI models are not immune to cyberattacks. “It’s possible [to attack] the AI model and learn information about users,” Kang told Built In.
Techniques like model inversion attacks can be used to infer sensitive information from model updates — and bad actors may even be able to intentionally corrupt federated learning using malicious devices. To mitigate these risks, federated learning often incorporates techniques like differential privacy and homomorphic encryption, which further protect user data during training.
Data Heterogeneity
Federated learning systems must handle the inherent heterogeneity of data and devices, including differences in processing power, memory and network conditions. This can lead to imbalanced updates and varying contributions to the global model. Even the unique way users interact with their devices can skew model accuracy.
Lack of Guaranteed Privacy
“A major misconception about federated learning is that it guarantees privacy for the users’ data,” Kang said. Companies like Google strive to guarantee privacy using algorithms like Federated Stochastic Gradient Descent (SGD) — which amplifies privacy by sampling data from random devices — and Federated Learning with Dynamic Regularization (FedDyn) — which allows systems to maintain accuracy in diverse environments by better handling data variability.
User Participation
“Another significant challenge is to incentivize users to participate and therefore help each other,” Diggavi told Built In. To encourage users to share their data, companies must drive user motivation through free services or ad-free content.
Benefits of Federated Learning
Despite its challenges, federated learning offers significant benefits:
Data Security
By decentralizing data storage and processing, federated learning reduces the risks associated with central data breaches. If individual devices are compromised, the global dataset remains protected since individual data points are not stored in one location.
Privacy
The bottom line: Users don’t want personal data like voice recordings or interactions with smart home devices to be sent to enterprise companies’ central servers.
“If [you] combine federated learning with provable privacy methods, [you] could give a privacy guarantee while enabling collaborative learning models,” Diggavi said. This privacy-preserving approach is particularly valuable in fields like healthcare and finance, where data confidentiality is paramount.
Environmental Impact
According to a 2021 study conducted by the University of Cambridge, federated learning can reduce the carbon footprint of the AI model training process when components are arranged in a certain way. However, the environmental effects of federated learning depend on a number of factors, and more research is needed in this area.
Frequently Asked Questions
Frequently Asked Questions
What is federated learning?
Federated learning is a distributed technique where devices collaboratively train a model by sharing only updates, not data, ensuring privacy and security while enabling decentralized machine learning.
What are examples of federated learning?
Examples of federated learning include Google’s Gboard improving typing predictions without accessing user data, Apple’s Siri being trained solely on local device data and Waymo’s vehicles gathering data from their surroundings and sharing updates instead of the raw data.
What are the benefits of federated learning?
Federated learning reduces communication costs, enhances data privacy, enables collaboration across organizations and improves machine learning models without centralizing sensitive data.
What is the difference between federated learning and machine learning?
Federated learning (FL) differs from machine learning (ML) in a few crucial ways:
- ML performs the entire model training process on a single server; FL involves various client devices training models on their local data. The distributed nature of FL is much more secure than traditional ML’s centralized approach.
- ML assumes all data involved in training is identically distributed; FL assumes the data is non-identically distributed. As a result, FL can produce models that are more able to adapt to different user habits, devices and contexts.
- ML relies on a central server at all times, resulting in lag times when users need fast answers. FL uses edge devices located closer to their data sources, resulting in quicker communication.
- Traditional ML often involves compiling raw data in a single location for training, presenting security risks and potential violations of data privacy regulations. FL doesn’t require this kind of centralization since client devices can train and store updated global models.
