Skip to content

Add LoadFromDatabase convenience methods (#4182) - #7666

Open
rAJt-11 wants to merge 3 commits into
dotnet:mainfrom
rAJt-11:feature/database-loader-loadfrom
Open

rAJt-11 wants to merge 3 commits into
dotnet:mainfrom
rAJt-11:feature/database-loader-loadfrom

Conversation

@rAJt-11

@rAJt-11 rAJt-11 commented Jul 24, 2026 •

Copy link
Copy Markdown

Adds LoadFromDatabase extension methods on DataOperationsCatalog, mirroring the existing LoadFromTextFile convenience API. This collapses the current three-step database-loading pattern into a single call.

Before:
var loader = mlContext.Data.CreateDatabaseLoader();
var source = new DatabaseSource(SqlClientFactory.Instance, connString, "SELECT * FROM Iris");
var data = loader.Load(source);

After:
var data = mlContext.Data.LoadFromDatabase(SqlClientFactory.Instance, connString, "SELECT * FROM Iris");
LoadFromDatabase(DatabaseSource source)
LoadFromDatabase(DbProviderFactory providerFactory, string connectionString, string commandText)
LoadFromDatabase(DbProviderFactory providerFactory, string connectionString, string commandText, int commandTimeoutInSeconds)

Why this approach
I followed the established LoadFromTextFile pattern in TextLoaderSaverCatalog so the new API stays consistent with the rest of the catalog. The methods simply compose the existing CreateDatabaseLoader() and Load(DatabaseSource) no new loading logic and argument validation is delegated to the existing DatabaseSource constructor.

Tests
Added IrisVectorLightGbmUsingLoadFromDatabase to DatabaseLoaderTests.cs, exercising the new API end-to-end through a LightGBM training pipeline.

Add LoadFromDatabase<TInput> extension methods on DataOperationsCatalog
that combine CreateDatabaseLoader<TInput>() and Load(DatabaseSource) into
a single call, mirroring the existing LoadFromTextFile<TInput> sugar.
@rAJt-11

rAJt-11 commented Jul 24, 2026

Copy link
Copy Markdown
Author

@dotnet-policy-service agree

Replace the redundant LightGBM training test with a focused, self-contained
in-memory SQLite test that verifies LoadFromDatabase loads the expected data
through both the DatabaseSource and DbProviderFactory overloads. This scopes
the test to the new data-loading API and avoids adding a training run to the
test suite.
@rAJt-11

rAJt-11 commented Jul 27, 2026

Copy link
Copy Markdown
Author

The failing DatabaseLoaderTests.Iris* tests (IrisLightGbm, IrisVectorLightGbm, IrisLightGbmWithTimeout, IrisSdcaMaximumEntropy) are failing due to the known LocalDB iris.mdf read-only attach issue tracked in #7657 — they fail independently of this change (my new LoadFromDatabaseLoadsExpectedData test uses in-memory SQLite and is unaffected). Could a maintainer confirm and/or re-run once #7657 is sorted?

@codecov

codecov Bot commented Sep 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.29730% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 69.90%. Comparing base (13dc9c5) to head (95d4583).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...ata/DataLoadSave/Database/DatabaseLoaderCatalog.cs 66.66% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #7666   +/-   ##
=======================================
  Coverage   69.89%   69.90%           
=======================================
  Files        1487     1487           
  Lines      276323   276360   +37     
  Branches    28295    28296    +1     
=======================================
+ Hits       193142   193183   +41     
+ Misses      75694    75689    -5     
- Partials     7487     7488    +1     
Flag Coverage Δ
Debug 69.90% <97.29%> (+<0.01%) ⬆️
production 64.08% <66.66%> (+<0.01%) ⬆️
test 89.84% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
test/Microsoft.ML.Tests/DatabaseLoaderTests.cs 91.98% <100.00%> (+1.07%) ⬆️
...ata/DataLoadSave/Database/DatabaseLoaderCatalog.cs 91.66% <66.66%> (-8.34%) ⬇️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants