Fixing Ghost Clipping with Batch Memory Manager

Summary: Ghost Clipping with Batch memory manager had an error, resulting in major accuracy loss. The issue was in the accumulate function, the command `p.summed_grad +=p.grad`, wasn't working as expected, since `p.grad` is modified in every iteration. The fix is to copy it and modify in-place.

Reviewed By: HuanyuZhang

Differential Revision: D67778159

fbshipit-source-id: b103cf95905c0b1feb9745249ec1669c95c11979

Author: EnayatUllah

opacus/optimizers/optimizer_fast_gradient_clipping.py

@@ -14,6 +14,7 @@
 
 from __future__ import annotations
 
+import copy
 import logging
 from typing import Callable, Optional
 
@@ -112,9 +113,9 @@ def accumulate(self):
         """
         for p in self.params:
             if p.summed_grad is not None:
-                p.summed_grad += p.grad
+                p.summed_grad.add_(p.grad.data)
             else:
-                p.summed_grad = p.grad
+                p.summed_grad = copy.deepcopy(p.grad.data)
 
     def zero_grad(self, set_to_none: bool = False):
         """