Skip to content

Commit 0b45905

Browse files
authored
Merge branch 'main' into main
2 parents 002c6d7 + 3934851 commit 0b45905

File tree

12 files changed

+2074
-1487
lines changed

12 files changed

+2074
-1487
lines changed

.circleci/config.yml

Lines changed: 0 additions & 517 deletions
This file was deleted.

.circleci/flake8_config.ini

Lines changed: 0 additions & 119 deletions
This file was deleted.

README.md

Lines changed: 52 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -2,39 +2,54 @@
22

33
<hr/>
44

5-
[![CircleCI](https://dl.circleci.com/status-badge/img/gh/pytorch/opacus/tree/main.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/pytorch/opacus/tree/main)
5+
[![GitHub Actions](https://github.com/pytorch/opacus/actions/workflows/ci_cpu.yml/badge.svg)](https://github.com/pytorch/opacus/actions/workflows/ci_cpu.yml)
66
[![Coverage Status](https://coveralls.io/repos/github/pytorch/opacus/badge.svg?branch=main)](https://coveralls.io/github/pytorch/opacus?branch=main)
77
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
88
[![License](https://img.shields.io/badge/license-apache2-green.svg)](LICENSE)
99

10-
[Opacus](https://opacus.ai) is a library that enables training PyTorch models with differential privacy.
11-
It supports training with minimal code changes required on the client, has little impact on training performance, and allows the client to online track the privacy budget expended at any given moment.
10+
[Opacus](https://opacus.ai) is a library that enables training PyTorch models
11+
with differential privacy. It supports training with minimal code changes
12+
required on the client, has little impact on training performance, and allows
13+
the client to online track the privacy budget expended at any given moment.
14+
1215

1316
## Target audience
17+
1418
This code release is aimed at two target audiences:
15-
1. ML practitioners will find this to be a gentle introduction to training a model with differential privacy as it requires minimal code changes.
16-
2. Differential Privacy researchers will find this easy to experiment and tinker with, allowing them to focus on what matters.
1719

20+
1. ML practitioners will find this to be a gentle introduction to training a
21+
model with differential privacy as it requires minimal code changes.
22+
2. Differential Privacy researchers will find this easy to experiment and tinker
23+
with, allowing them to focus on what matters.
1824

1925
## Installation
26+
2027
The latest release of Opacus can be installed via `pip`:
28+
2129
```bash
2230
pip install opacus
2331
```
32+
2433
OR, alternatively, via `conda`:
34+
2535
```bash
2636
conda install -c conda-forge opacus
2737
```
2838

29-
You can also install directly from the source for the latest features (along with its quirks and potentially occasional bugs):
39+
You can also install directly from the source for the latest features (along
40+
with its quirks and potentially occasional bugs):
41+
3042
```bash
3143
git clone https://github.com/pytorch/opacus.git
3244
cd opacus
3345
pip install -e .
3446
```
3547

3648
## Getting started
37-
To train your model with differential privacy, all you need to do is to instantiate a `PrivacyEngine` and pass your model, data_loader, and optimizer to the engine's `make_private()` method to obtain their private counterparts.
49+
50+
To train your model with differential privacy, all you need to do is to
51+
instantiate a `PrivacyEngine` and pass your model, data_loader, and optimizer to
52+
the engine's `make_private()` method to obtain their private counterparts.
3853

3954
```python
4055
# define your components as usual
@@ -54,21 +69,25 @@ model, optimizer, data_loader = privacy_engine.make_private(
5469
# Now it's business as usual
5570
```
5671

57-
The [MNIST example](https://github.com/pytorch/opacus/tree/main/examples/mnist.py) shows an end-to-end run using Opacus. The [examples](https://github.com/pytorch/opacus/tree/main/examples/) folder contains more such examples.
72+
The
73+
[MNIST example](https://github.com/pytorch/opacus/tree/main/examples/mnist.py)
74+
shows an end-to-end run using Opacus. The
75+
[examples](https://github.com/pytorch/opacus/tree/main/examples/) folder
76+
contains more such examples.
5877

5978
### Migrating to 1.0
6079

61-
Opacus 1.0 introduced many improvements to the library, but also some breaking changes.
62-
If you've been using Opacus 0.x and want to update to the latest release,
63-
please use this [Migration Guide](https://github.com/pytorch/opacus/blob/main/Migration_Guide.md)
64-
80+
Opacus 1.0 introduced many improvements to the library, but also some breaking
81+
changes. If you've been using Opacus 0.x and want to update to the latest
82+
release, please use this
83+
[Migration Guide](https://github.com/pytorch/opacus/blob/main/Migration_Guide.md)
6584

6685
## Learn more
6786

6887
### Interactive tutorials
6988

70-
We've built a series of IPython-based tutorials as a gentle introduction to training models
71-
with privacy and using various Opacus features.
89+
We've built a series of IPython-based tutorials as a gentle introduction to
90+
training models with privacy and using various Opacus features.
7291

7392
- [Building an Image Classifier with Differential Privacy](https://github.com/pytorch/opacus/blob/main/tutorials/building_image_classifier.ipynb)
7493
- [Training a differentially private LSTM model for name classification](https://github.com/pytorch/opacus/blob/main/tutorials/building_lstm_name_classifier.ipynb)
@@ -78,9 +97,13 @@ with privacy and using various Opacus features.
7897
- [Opacus Guide: Module Validator and Fixer](https://github.com/pytorch/opacus/blob/main/tutorials/guide_to_module_validator.ipynb)
7998

8099
## Technical report and citation
81-
The technical report introducing Opacus, presenting its design principles, mathematical foundations, and benchmarks can be found [here](https://arxiv.org/abs/2109.12298).
100+
101+
The technical report introducing Opacus, presenting its design principles,
102+
mathematical foundations, and benchmarks can be found
103+
[here](https://arxiv.org/abs/2109.12298).
82104

83105
Consider citing the report if you use Opacus in your papers, as follows:
106+
84107
```
85108
@article{opacus,
86109
title={Opacus: {U}ser-Friendly Differential Privacy Library in {PyTorch}},
@@ -92,21 +115,29 @@ Consider citing the report if you use Opacus in your papers, as follows:
92115

93116
### Blogposts and talks
94117

95-
If you want to learn more about DP-SGD and related topics, check out our series of blogposts and talks:
118+
If you want to learn more about DP-SGD and related topics, check out our series
119+
of blogposts and talks:
96120

97121
- [Differential Privacy Series Part 1 | DP-SGD Algorithm Explained](https://medium.com/pytorch/differential-privacy-series-part-1-dp-sgd-algorithm-explained-12512c3959a3)
98122
- [Differential Privacy Series Part 2 | Efficient Per-Sample Gradient Computation in Opacus](https://medium.com/pytorch/differential-privacy-series-part-2-efficient-per-sample-gradient-computation-in-opacus-5bf4031d9e22)
99123
- [PriCon 2020 Tutorial: Differentially Private Model Training with Opacus](https://www.youtube.com/watch?v=MWPwofiQMdE&list=PLUNOsx6Az_ZGKQd_p4StdZRFQkCBwnaY6&index=52)
100124
- [Differential Privacy on PyTorch | PyTorch Developer Day 2020](https://www.youtube.com/watch?v=l6fbl2CBnq0)
101125
- [Opacus v1.0 Highlights | PyTorch Developer Day 2021](https://www.youtube.com/watch?v=U1mszp8lzUI)
102-
126+
- [Enabling Fast Gradient Clipping and Ghost Clipping in Opacus](https://pytorch.org/blog/clipping-in-opacus/)
103127

104128
## FAQ
105-
Check out the [FAQ](https://opacus.ai/docs/faq) page for answers to some of the most frequently asked questions about differential privacy and Opacus.
129+
130+
Check out the [FAQ](https://opacus.ai/docs/faq) page for answers to some of the
131+
most frequently asked questions about differential privacy and Opacus.
106132

107133
## Contributing
108-
See the [CONTRIBUTING](https://github.com/pytorch/opacus/tree/main/CONTRIBUTING.md) file for how to help out.
109-
Do also check out the README files inside the repo to learn how the code is organized.
134+
135+
See the
136+
[CONTRIBUTING](https://github.com/pytorch/opacus/tree/main/CONTRIBUTING.md) file
137+
for how to help out. Do also check out the README files inside the repo to learn
138+
how the code is organized.
110139

111140
## License
112-
This code is released under Apache 2.0, as found in the [LICENSE](https://github.com/pytorch/opacus/tree/main/LICENSE) file.
141+
142+
This code is released under Apache 2.0, as found in the
143+
[LICENSE](https://github.com/pytorch/opacus/tree/main/LICENSE) file.

docs/faq.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,8 @@ Opacus computes and stores *per-sample* gradients under the hood. What this mean
108108

109109
Although we report expended privacy budget using the (epsilon, delta) language, internally, we track it using Rényi Differential Privacy (RDP) [[Mironov 2017](https://arxiv.org/abs/1702.07476), [Mironov et al. 2019](https://arxiv.org/abs/1908.10530)]. In short, (alpha, epsilon)-RDP bounds the [Rényi divergence](https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy#R%C3%A9nyi_divergence) of order alpha between the distribution of the mechanism’s outputs on any two datasets that differ in a single element. An (alpha, epsilon)-RDP statement is a relaxation of epsilon-DP but retains many of its important properties that make RDP particularly well-suited for privacy analysis of DP-SGD. The `alphas` parameter instructs the privacy engine what RDP orders to use for tracking privacy expenditure.
110110

111-
When the privacy engine needs to bound the privacy loss of a training run using (epsilon, delta)-DP for a given delta, it searches for the optimal order from among `alphas`. There’s very little additional cost in expanding the list of orders. We suggest using a list `[1 + x / 10.0 for x in range(1, 100)] + list(range(12, 64))`. You can pass your own alphas by passing `alphas=custom_alphas` when calling `privacy_engine.make_private_with_epsilon`.
111+
When the privacy engine needs to bound the privacy loss of a training run using (epsilon, delta)-DP for a given delta, it searches for the optimal order from among `alphas`. There’s very little additional cost in expanding the list of orders. We suggest using a list `[1 + x / 10.0 for x in range(1, 100)] + list(range(12, 64))`.
112+
<!-- You can pass your own alphas by passing `alphas=custom_alphas` when calling `privacy_engine.make_private_with_epsilon`. -->
112113

113114
A call to `privacy_engine.get_epsilon(delta=delta)` returns a pair: an epsilon such that the training run satisfies (epsilon, delta)-DP and an optimal order alpha. An easy diagnostic to determine whether the list of `alphas` ought to be expanded is whether the returned value alpha is one of the two boundary values of `alphas`.
114115

opacus/accountants/gdp.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ def step(self, *, noise_multiplier: float, sample_rate: float):
4444
else:
4545
self.history = [(noise_multiplier, sample_rate, 1)]
4646

47-
def get_epsilon(self, delta: float, poisson: bool = True) -> float:
47+
def get_epsilon(self, delta: float, poisson: bool = True, **kwargs) -> float:
4848
"""
4949
Return privacy budget (epsilon) expended so far.
5050

opacus/accountants/prv.py

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,12 @@ def step(self, *, noise_multiplier: float, sample_rate: float):
8181
self.history.append((noise_multiplier, sample_rate, 1))
8282

8383
def get_epsilon(
84-
self, delta: float, *, eps_error: float = 0.01, delta_error: float = None
84+
self,
85+
delta: float,
86+
*,
87+
eps_error: float = 0.01,
88+
delta_error: float = None,
89+
**kwargs,
8590
) -> float:
8691
"""
8792
Return privacy budget (epsilon) expended so far.

opacus/accountants/rdp.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,10 @@ def get_privacy_spent(
6868
return float(eps), float(best_alpha)
6969

7070
def get_epsilon(
71-
self, delta: float, alphas: Optional[List[Union[float, int]]] = None
71+
self,
72+
delta: float,
73+
alphas: Optional[List[Union[float, int]]] = None,
74+
**kwargs,
7275
):
7376
"""
7477
Return privacy budget (epsilon) expended so far.

opacus/optimizers/optimizer_fast_gradient_clipping.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414

1515
from __future__ import annotations
1616

17+
import copy
1718
import logging
1819
from typing import Callable, Optional
1920

@@ -112,9 +113,9 @@ def accumulate(self):
112113
"""
113114
for p in self.params:
114115
if p.summed_grad is not None:
115-
p.summed_grad += p.grad
116+
p.summed_grad.add_(p.grad.data)
116117
else:
117-
p.summed_grad = p.grad
118+
p.summed_grad = copy.deepcopy(p.grad.data)
118119

119120
def zero_grad(self, set_to_none: bool = False):
120121
"""

opacus/privacy_engine.py

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,6 @@ def _prepare_optimizer(
136136
loss_reduction=loss_reduction,
137137
generator=generator,
138138
secure_mode=self.secure_mode,
139-
**kwargs,
140139
)
141140

142141
def _prepare_data_loader(
@@ -212,6 +211,26 @@ def _prepare_model(
212211
loss_reduction=loss_reduction,
213212
)
214213

214+
def _prepare_criterion(
215+
self,
216+
*,
217+
module: GradSampleModule,
218+
optimizer: DPOptimizer,
219+
criterion=nn.CrossEntropyLoss(),
220+
loss_reduction: str = "mean",
221+
**kwargs,
222+
) -> DPLossFastGradientClipping:
223+
"""
224+
Args:
225+
module: GradSampleModule used for training,
226+
optimizer: DPOptimizer used for training,
227+
criterion: Loss function used for training,
228+
loss_reduction: "mean" or "sum", indicates if the loss reduction (for aggregating the gradients)
229+
230+
Prepare the DP loss class, which packages the two backward passes for fast gradient clipping.
231+
"""
232+
return DPLossFastGradientClipping(module, optimizer, criterion, loss_reduction)
233+
215234
def is_compatible(
216235
self,
217236
*,
@@ -403,9 +422,14 @@ def make_private(
403422
self.accountant.get_optimizer_hook_fn(sample_rate=sample_rate)
404423
)
405424
if grad_sample_mode == "ghost":
406-
criterion = DPLossFastGradientClipping(
407-
module, optimizer, criterion, loss_reduction
425+
criterion = self._prepare_criterion(
426+
module=module,
427+
optimizer=optimizer,
428+
criterion=criterion,
429+
loss_reduction=loss_reduction,
430+
**kwargs,
408431
)
432+
409433
return module, optimizer, criterion, data_loader
410434

411435
return module, optimizer, data_loader

0 commit comments

Comments
 (0)