ROC Optimizer

Radius of Curvature-Based Adaptive Gradient Descent

ROC (Radius of Curvature Optimizer) is a physics-inspired optimization algorithm that dynamically scales the learning rate using a first-order approximation of the radius of curvature of the optimization trajectory.

Unlike Adam or RMSprop, which adapt step sizes based on gradient statistics, ROC adapts based on the geometry of the loss landscape.

---

Report

View complete report here.

---

Motivation

Standard first-order optimizers suffer from a geometric limitation:

• Large learning rates → instability in high-curvature regions
• Small learning rates → slow convergence in flat regions

Second-order methods (e.g., Newton's Method) address this using the Hessian, but incur O(N³) computational cost.

ROC bridges this gap by approximating curvature using only gradient differences, preserving O(N) complexity.

---

Key Idea

From physics, curvature is defined as the rate of change of the tangent vector with respect to arc length.

We approximate the radius of curvature at iteration t as:

$R_t = \frac{||\theta_t - \theta_{t-1}||^2}{||g_t - g_{t-1}||^2 + \epsilon}$

Where:

• $\theta_t$ = parameters
• $g_t = \nabla f(\theta_t)$
• $\epsilon$ = stability constant

Intuition

• Small radius (high curvature) → reduce step size
• Large radius (low curvature) → increase step size

---

Algorithm

Scaling Factor

$S_t = \text{clip}(R_t, S_{min}, S_{max})$

Update Rule

$v_t = \mu v_{t-1} + (\alpha S_t) g_t$

$\theta_t = \theta_{t-1} - v_t$

---

Default Hyperparameters

Parameter	Description	Default
$\alpha$	Base learning rate	0.01
$\mu$	Momentum coefficient	0.9
$S_{min}$	Minimum scaling bound	0.1
$S_{max}$	Maximum scaling bound	10.0
$\epsilon$	Numerical stability	1e-8

---

---

Experimental Results

All experiments averaged over 5 random seeds.

---

1. Well-Conditioned Quadratic (Condition Number = 10)

Iterations to reach $f(x) < 10^{-6}$:

Optimizer	Iterations
ROC	164 ± 16
SGD+Momentum	426 ± 106
Adam	751 ± 279
RMSprop	6104 ± 4771
Adagrad	Did not converge

ROC converged 4.5× faster than Adam

---

2. Ill-Conditioned Quadratic (Condition Number = 100)

Iterations to reach $f(x) < 10^{-6}$:

Optimizer	Iterations
ROC	279 ± 62
SGD+Momentum	611 ± 37
Adam	1906 ± 735
RMSprop	Did not converge
Adagrad	Did not converge

ROC was 6.8× faster than Adam

---

3. Rosenbrock Function

Iterations to reach $f(x) < 10^{-4}$:

Optimizer	Iterations
ROC	4632
SGD+Momentum	9268
Adam	10346
RMSprop	Did not converge
Adagrad	Did not converge

ROC converged ~2× faster than SGD+Momentum and Adam

---

4. MNIST (3-Layer NN: 784-128-64-10)

Test Accuracy After 20 Epochs

Optimizer	Accuracy
ROC (default)	91.63% ± 0.13%
ROC (μ = 0.995)	95.50% ± 0.14%
Adam	95.47% ± 0.45%

Training Time per Epoch

Optimizer	Time
ROC	0.17s ± 0.008s
Adam	0.18s ± 0.012s

Tuned ROC slightly surpassed Adam with lower variance and comparable runtime.

---

---

Why ROC Matters

• Curvature-aware without Hessian
• Maintains O(N) computational complexity
• Adapts globally based on geometric signal
• More stable convergence across random seeds
• Works well in ill-conditioned and non-convex landscapes

ROC can be viewed as a quasi-second-order optimizer with first-order efficiency.

---

Future Work

• Per-parameter curvature scaling
• Theoretical convergence analysis
• Large-scale deep learning benchmarks
• Integration into PyTorch optimizer API

---