feat: add ml/strided/dsgd-trainer-squared-epsilon-insensitive

[nakul-krishnakumar]

Resolves a part of #12875.

Description

What is the purpose of this pull request?

This pull request:

• Adds ml/strided/dsgd-trainer-squared-epsilon-insensitive.

Related Issues

Does this pull request have any related issues?

This pull request has the following related issues:

• [RFC]: Add batch machine learning algorithms in Javascript and C (tracking issue) #12875.

Questions

Any questions for reviewers of this pull request?

No.

Other

Any other information relevant to this pull request? This may include screenshots, references, and/or implementation notes.

No.

Checklist

Please ensure the following tasks are completed before submitting this pull request.

• Read, understood, and followed the contributing guidelines.

AI Assistance

When authoring the changes proposed in this PR, did you use any kind of AI assistance?

• Yes
• No

If you answered "yes" above, how did you use AI assistance?

• Code generation (e.g., when writing an implementation or fixing a bug)
• Test/benchmark generation
• Documentation (including examples)
• Research and understanding

Disclosure

If you answered "yes" to using AI assistance, please provide a short disclosure indicating how you used AI assistance. This helps reviewers determine how much scrutiny to apply when reviewing your contribution. Example disclosures: "This PR was written primarily by Claude Code." or "I consulted ChatGPT to understand the codebase, but the proposed changes were fully authored manually by myself.".

I used both Claude Code and ChatGPT to help me refer the sklearn implementation and write the code.

---

@stdlib-js/reviewers

[nakul-krishnakumar]

I have the following doubts:

1. As we are anyway having branching for penalties and learningRate, I think it would be okay to keep a single API ml/strided/sgd-trainer with branching for lossFunctions as well as instead of having separate sgd trainers per loss function as we had planned on in the earlier meetings (dsgd-trainer-squared-epsilon-insensitive).

2. Should we implement the helper function l1Penalty as a standalone implementation? ( I don't see any reason why we shouldn't )

3. Should the API support all 3 penalties/regularization? (l1, l2 and elasticnet)

   • ml/incr/binary-classification supports only L2 by default.
   • If going for all 3, we will have to implement standalone ml/base/sgd-classification/penalties (enum) (opened a PR feat: add ml/base/sgd-classification/penalties #13623 )
   • I would suggest to have support for all 3.

4. How do we handle returning intercept? We cant just update the user passed intercept as we do for the weight matrix as I believe numbers are not passed by reference unlike arrays.

5. Currently this implementation is not in 1:1 parity with the sklearn API, it skips things like dataset shuffling, learning rate schedulers like PA1, PA2 (they were recently added in sklearn API, so I am skeptical on adding those), early stopping etc. It would be helpful if you could let me know if I am missing anything important here that you think would be helpful to have.

6. Should we have a weight vector implementation similar to how sklearn as well as ml/incr/sgd-regression does?
   sklearn using weight vector in sgd implementation: Ref

7. Tidy up function argument, what all should be passed to base implementation?

sklearn sgd implementation: Ref

cc: @kgryte

[nakul-krishnakumar]

Also please note that I kept some comments only for the sake of understanding, will remove them once we finalize the implementation.

[kgryte]

Quick aside: I am wondering if we should namespace the strided routines. E.g.,

ml/strided/kmeans/ddistance
ml/strided/sgd-trainer/dsquared-epsilon-insensitive
...

The current package base name, as is, is a bit unwieldy. Thoughts?

[kgryte]

For packages which are applicable across ML algos, they can go directly in ml/strided. E.g., hypothetically,

ml/strided/dsquared-epsilon-insensitive

And actually, is squared epsilon insensitive so specific to sgd-trainer that it shouldn't be broadly applicable?

I am just curious where we draw the line.

[nakul-krishnakumar]

Quick aside: I am wondering if we should namespace the strided routines. E.g.,

ml/strided/kmeans/ddistance
ml/strided/sgd-trainer/dsquared-epsilon-insensitive
...

The current package base name, as is, is a bit unwieldy. Thoughts?

Yeah we could do that, we had discussed this earlier but decided not to until we add more packages and it becomes harder to track.

[nakul-krishnakumar]

For packages which are applicable across ML algos, they can go directly in ml/strided. E.g., hypothetically,

ml/strided/dsquared-epsilon-insensitive

And actually, is squared epsilon insensitive so specific to sgd-trainer that it shouldn't be broadly applicable?

I am just curious where we draw the line.

Here, yes it is specific to SGD. Because ml/strided/dsquared-epsilon-insensitive is not actually a strided version of squared-epsilon-insensitive loss function gradient, instead it is a strided version of SGD implementation hard coded with squared-epsilon-insensitive loss function.
Instead what I am saying is lets just have a single ml/strided/dsgd-trainer which uses a package similar to ml/strided/dkmeans-distance and resolve the gradient of the user passed loss function.

function dsgdTrainer( loss, .... ) {
    ...
    g = dsgdLossGradient( loss, x, e, y, p );
    ...
}

This was our first choice but decided on implementing individual sgd-trainer per loss function so that we can optimize it (as this doesn't require resolving the loss function using branching). But now that I look at it, the implementation anyway does branching to resolve learning rate and penalty so maybe branching for resolving loss functions too might not be that costly comparatively.

cc: @kgryte