Fix typos in DDP tutorial (#438)

Summary:
## Types of changes

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [x] Docs change / refactoring / dependency upgrade

## Motivation and Context / Related issue

I was reading the DDP tutorial and spotted a few typos. Great write-up by the way!

## Checklist

- [x] The documentation is up-to-date with the changes I made.
- [x] I have read the **CONTRIBUTING** document and completed the CLA (see **CONTRIBUTING**).
- [x] All tests passed, and additional code has been covered with new tests.

Pull Request resolved: #438

Reviewed By: karthikprasad

Differential Revision: D37109929

Pulled By: romovpa

fbshipit-source-id: edcc19e066e5836f01449582c95d0f06ca5c7469

Author: tholop

tutorials/ddp_tutorial.ipynb

@@ -15,7 +15,7 @@
    "source": [
     "In this tutorial we'll go over the basics you need to know to start using Opacus in your distributed model training pipeline. As the state-of-the-art models and datasets get bigger, multi-GPU training became the norm and Opacus comes with seamless, out-of-the box support for Distributed Data Parallel (DDP).\n",
     "\n",
-    "This tutorial requires basic knowledge of Opacus and DDP. If you're knew to either of these tools, we suggest to start with the following tutorials: [Building an Image Classifier with Differential Privacy](https://opacus.ai/tutorials/building_image_classifier) and [Getting Started with Distributed Data Parallel](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html)\n",
+    "This tutorial requires basic knowledge of Opacus and DDP. If you're new to either of these tools, we suggest to start with the following tutorials: [Building an Image Classifier with Differential Privacy](https://opacus.ai/tutorials/building_image_classifier) and [Getting Started with Distributed Data Parallel](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html)\n",
     "\n",
     "In Chapter 1 we'll start with a mininmal working example to demonstrate what exactly do you need to do in order to make Opacus work in a distributed setting. This should be enough to get started for most common scenarios.\n",
     "\n",
@@ -35,7 +35,7 @@
    "id": "1089c8c1",
    "metadata": {},
    "source": [
-    "Before we begin, thre are few things we need to mention.\n",
+    "Before we begin, there are a few things we need to mention.\n",
     "\n",
     "First, this tutorial is written to be executed on a single Linux machine with at least 2 GPUs. The general principles remain the same for Windows environment and/or multi-node training, but you'll need to slightly modify the DDP code to make it work.\n",
     "\n",
@@ -409,7 +409,7 @@
    "id": "e8703467",
    "metadata": {},
    "source": [
-    "And, finally, running then script. Notice, that we've initialized our `DataLoader` with `batch_size=200`, which is equivalent to 300 batches on the full dataset (60000 images). \n",
+    "And, finally, running the script. Notice, that we've initialized our `DataLoader` with `batch_size=200`, which is equivalent to 300 batches on the full dataset (60000 images). \n",
     "\n",
     "After passing it to `make_private` on each worker we have a data loader with `batch_size=100` each, but each data loader still goes over 300 batches."
    ]
@@ -458,7 +458,7 @@
    "id": "c86099a0",
    "metadata": {},
    "source": [
-    "**Note**: The following two chapters disucss the advanced usage of Opacus and its implementation details. We strongly recommend to read the tutorial on [Advanced Features of Opacus](https://opacus.ai/tutorials/intro_to_advanced_features) before proceeding.\n",
+    "**Note**: The following two chapters discuss the advanced usage of Opacus and its implementation details. We strongly recommend to read the tutorial on [Advanced Features of Opacus](https://opacus.ai/tutorials/intro_to_advanced_features) before proceeding.\n",
     "\n",
     "Now let's look inside `make_private` method and see what it does to enable DDP processing. And we'll start with the modifications made to the `DataLoader`.\n",
     "\n",
@@ -477,7 +477,7 @@
     "- Distributed, non-private\n",
     "- Distributed, private (with Poisson sampling)\n",
     "\n",
-    "All three are initialized so that the logical batch size is 64. Note, that `DPDataLoader` is initialized with a non"
+    "All three are initialized so that the logical batch size is 64."
    ]
   },
   {
@@ -585,7 +585,7 @@
    "source": [
     "Let's see what happens when we run it - and what exactly does `from_data_loader` factory did.\n",
     "\n",
-    "Notice, that our private DataLoader was initialized with a non-distributed, non-private data loader. And all the basic parameters (per GPU batch size and number of examples per GPU) matches with distributed, non-private data loader."
+    "Notice, that our private DataLoader was initialized with a non-distributed, non-private data loader. And all the basic parameters (per GPU batch size and number of examples per GPU) match with distributed, non-private data loader."
    ]
   },
   {
@@ -613,19 +613,19 @@
    "id": "d938f572",
    "metadata": {},
    "source": [
-    "## Chapter 3: Syncronisation"
+    "## Chapter 3: Synchronisation"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "491a3ed8",
    "metadata": {},
    "source": [
-    "One significant difference between `DDP` and `DPDDP` is how it approaches syncronisation.\n",
+    "One significant difference between `DDP` and `DPDDP` is how it approaches synchronisation.\n",
     "\n",
-    "Normally with Distributed Data Parallel forward and backward passes are syncronisation points, and `DDP` wrapper ensures that the gradients are syncronised across workers at the end of the backward pass.\n",
+    "Normally with Distributed Data Parallel forward and backward passes are synchronisation points, and `DDP` wrapper ensures that the gradients are synchronised across workers at the end of the backward pass.\n",
     "\n",
-    "Opacus, however, need a later syncronisation point. Before we can use the gradients, we need to clip them add noise. This is done in the optimizer, which moves the syncronisation point from the backward pass to the optimization step.\n",
+    "Opacus, however, need a later synchronisation point. Before we can use the gradients, we need to clip them add noise. This is done in the optimizer, which moves the synchronisation point from the backward pass to the optimization step.\n",
     "Additionally, to simplify the calculations, we only add noise on worker with `rank=0`, and use the noise scale calibrated to the combined batch across all workers."
    ]
   },
@@ -698,7 +698,7 @@
    "source": [
     "Now we've initialized `DifferentiallyPrivateDistributedDataParallel` model and `DistributedDPOptimizer` let's see how they work together.\n",
     "\n",
-    "`DifferentiallyPrivateDistributedDataParallel` is a no-op: we only perform model syncronisation on initialization and do nothing on forward and backward passes.\n",
+    "`DifferentiallyPrivateDistributedDataParallel` is a no-op: we only perform model synchronisation on initialization and do nothing on forward and backward passes.\n",
     "\n",
     "`DistributedDPOptimizer`, on the other hand does all the heavy lifting:\n",
     "- It does gradient clipping on each worker independently\n",
@@ -804,7 +804,7 @@
    "id": "a9c95972",
    "metadata": {},
    "source": [
-    "When we run the code, notice that the gradients are not syncronised after `loss.backward()`, but only after `optimizer.step()`. For this example we've set privacy parameters to effectively disable noise and clipping, so the syncronised gradient is indeed the average between individual worker's gradients."
+    "When we run the code, notice that the gradients are not synchronised after `loss.backward()`, but only after `optimizer.step()`. For this example we've set privacy parameters to effectively disable noise and clipping, so the synchronised gradient is indeed the average between individual worker's gradients."
    ]
   },
   {