Add support for Normalized SGD (NSGD) with Perturbation

[ParthS007]

🚀 Feature

Add Normalized SGD (NSGD) with Perturbation support to Opacus with NormalizedSGDPOptimizer that uses gradient normalization instead of hard clipping, providing better convergence properties for non-convex optimization problems.

Motivation

Standard DP-SGD uses hard clipping which can lead to gradient bias and suboptimal convergence, especially in non-convex settings. Normalization-based methods address this by normalizing gradients rather than clipping them, which can provide better gradient approximation and improved convergence properties for non-convex optimization problems.

Pitch

Implement NSGD based on "Differentially Private Non-Convex Optimization": https://arxiv.org/pdf/2206.13033

• Per-sample gradient normalization (not hard clipping)
• Normalization formula: g_normalized = g / (r + ||g||) where r > 0 is a regularization parameter
• Better convergence properties for non-convex optimization compared to clipping-based methods
• Reduces gradient bias introduced by hard clipping thresholds

Usage

privacy_engine = PrivacyEngine()

model, optimizer, dataloader = privacy_engine.make_private(
    module=model,
    optimizer=optimizer,
    data_loader=dataloader,
    noise_multiplier=1.0,
    max_grad_norm=0.1,  # Used as regularization_param if regularization_param not provided
    clipping="normalized",  # New option
    regularization_param=0.1,  # Optional: explicit regularization parameter
)

Alternatives

• Existing DPOptimizer (hard clipping): Uses hard clipping threshold which can introduce bias and suboptimal convergence in non-convex settings
• Existing AdaClipDPOptimizer: Adapts a single global clipping norm over time, but still uses hard clipping which can bias gradients

Additional context

I'm happy to submit a PR for this feature if it aligns with the project's goals. I'm implementing Normalized-SGD (NSGD) with Perturbation as part of my thesis work, and I believe making it available in the main library could benefit the broader Opacus community.

[ParthS007]

cc: @HuanyuZhang @EnayatUllah :)