feat(content): implement ModDB and CnCLabs content-pipeline integration #204

undead2146 · 2025-12-01T23:45:11Z

This PR integrates ModDBContentProvider and CnCLabsContentProvider for discovering, delivering, resolving content and generating a ContentManifest

…olution.

undead2146 · 2025-12-01T23:52:49Z

@greptile

greptile-apps · 2025-12-01T23:54:35Z

Greptile Overview

Greptile Summary

This PR successfully integrates ModDB and CnCLabs into the content pipeline, enabling discovery, resolution, and delivery of community content. The implementation follows established patterns with discoverers scraping listing pages, resolvers fetching detail pages, and manifest factories generating proper content manifests.

Major Changes

Added 5 new ContentType enum values (Skin, Video, Replay, Screensaver, ModdingTool) to support expanded content types
Implemented complete ModDB pipeline: ModDBDiscoverer for multi-section content discovery, ModDBResolver for HTML parsing, and ModDBManifestFactory for manifest generation using date-based IDs
Extended CnCLabs pipeline with CNCLabsManifestFactory and enhanced constants for mods, downloads, and additional content types
Registered both pipelines in ContentPipelineModule with proper DI configuration

Issues Found

File size parsing in ModDBResolver.cs:264 splits on periods which would break decimal sizes like "2.5 MB"
Fallback to current date in ModDBResolver.cs:218 could cause manifest ID collisions when submission dates are unavailable
Blocking async call with .GetAwaiter().GetResult() in ModDBManifestFactory.cs:142 reduces async benefits

Confidence Score: 4/5

This PR is generally safe to merge with minor issues in error handling and parsing logic
Score reflects solid architectural design and comprehensive implementation, but deducted one point for the file size parsing bug that will cause runtime errors and the date fallback logic that could lead to manifest ID conflicts
Pay close attention to ModDBResolver.cs - the file size parsing logic needs fixing before production use

Important Files Changed

File Analysis

Filename	Score	Overview
GenHub/GenHub/Features/Content/Services/ContentDiscoverers/ModDBDiscoverer.cs	4/5	Implements ModDB content discovery across mods, downloads, and addons sections with filtering support
GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs	4/5	Resolves ModDB content by scraping detail pages - has potential selector issues and fallback date handling
GenHub/GenHub/Features/Content/Services/Publishers/ModDBManifestFactory.cs	4/5	Factory for ModDB manifests using release dates in manifest IDs and comprehensive tagging
GenHub/GenHub/Infrastructure/DependencyInjection/ContentPipelineModule.cs	5/5	Registers ModDB and CnCLabs pipeline services including discoverers, resolvers, and manifest factories

Sequence Diagram

sequenceDiagram
    participant User
    participant Orchestrator as ContentOrchestrator
    participant Provider as ContentProvider
    participant Discoverer as IContentDiscoverer
    participant Resolver as IContentResolver
    participant Factory as ManifestFactory
    participant Deliverer as IContentDeliverer

    Note over User,Deliverer: Content Discovery & Resolution Flow

    User->>Orchestrator: Request Content (query)
    Orchestrator->>Provider: DiscoverAsync(query)
    
    alt ModDB Content
        Provider->>Discoverer: ModDBDiscoverer.DiscoverAsync()
        Discoverer->>Discoverer: Scrape listing pages (mods/downloads/addons)
        Discoverer->>Discoverer: Parse content items with metadata
        Discoverer-->>Provider: Return ContentSearchResult[]
        Provider-->>Orchestrator: Content items with sourceUrl
        
        Orchestrator->>Resolver: ModDBResolver.ResolveAsync(item)
        Resolver->>Resolver: Fetch detail page HTML
        Resolver->>Resolver: ParseModDetailPageAsync()
        Resolver->>Resolver: Extract metadata (author, date, download URL)
        Resolver->>Factory: ModDBManifestFactory.CreateManifest()
        Factory->>Factory: Generate manifest ID with date (1.YYYYMMDD.moddb-author...)
        Factory->>Factory: Add dependencies and tags
        Factory-->>Resolver: ContentManifest
        Resolver-->>Orchestrator: ContentManifest
        
    else CNCLabs Content
        Provider->>Discoverer: CNCLabsMapDiscoverer.DiscoverAsync()
        Discoverer->>Discoverer: Build search URL with filters
        Discoverer->>Discoverer: Parse map listings
        Discoverer-->>Provider: Return ContentSearchResult[]
        Provider-->>Orchestrator: Content items with mapId metadata
        
        Orchestrator->>Resolver: CNCLabsMapResolver.ResolveAsync(item)
        Resolver->>Resolver: Fetch detail page HTML
        Resolver->>Resolver: ParseMapDetailPageAsync()
        Resolver->>Resolver: Extract breadcrumb category
        Resolver->>Factory: CNCLabsManifestFactory.CreateManifest()
        Factory->>Factory: Generate manifest ID (1.0.cnclabs-author...)
        Factory-->>Resolver: ContentManifest
        Resolver-->>Orchestrator: ContentManifest
    end

    Orchestrator->>Deliverer: DeliverAsync(manifest)
    Deliverer->>Deliverer: Download and extract content
    Deliverer->>Deliverer: Store in CAS
    Deliverer-->>Orchestrator: Delivery result
    Orchestrator-->>User: Content ready for installation

greptile-apps

_{11 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs

…Resolver.cs Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

greptile-apps

_{11 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2025-12-03T20:25:42Z

GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs

+        _logger.LogWarning("Could not extract submission date, using current date");
+        return DateTime.UtcNow;


logic: using current date as fallback could cause issues with manifest ID uniqueness and versioning - multiple pieces of content without dates would get the same daily timestamp

Prompt To Fix With AI

This is a comment left during a code review. Path: GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs Line: 218:219 Comment: **logic:** using current date as fallback could cause issues with manifest ID uniqueness and versioning - multiple pieces of content without dates would get the same daily timestamp How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2025-12-03T20:25:43Z

GenHub/GenHub/Features/Content/Services/Publishers/ModDBManifestFactory.cs

+        manifest.AddRemoteFileAsync(
+            fileName,
+            details.DownloadUrl,
+            ContentSourceType.RemoteDownload).GetAwaiter().GetResult();


style: .GetAwaiter().GetResult() blocks async execution - consider making CreateManifest async or restructuring to avoid blocking

Prompt To Fix With AI

This is a comment left during a code review. Path: GenHub/GenHub/Features/Content/Services/Publishers/ModDBManifestFactory.cs Line: 142:145 Comment: **style:** `.GetAwaiter().GetResult()` blocks async execution - consider making `CreateManifest` async or restructuring to avoid blocking How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2025-12-03T20:25:44Z

GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs

+            return 0;
+        }
+
+        var parts = sizeText.Trim().Split(new[] { ' ', ',', '.', }, StringSplitOptions.RemoveEmptyEntries);


logic: splitting on periods could incorrectly parse file sizes like "2.5 MB" - the numeric part 2 and unit 5 would be separated

Prompt To Fix With AI

This is a comment left during a code review. Path: GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs Line: 264:264 Comment: **logic:** splitting on periods could incorrectly parse file sizes like "2.5 MB" - the numeric part `2` and unit `5` would be separated How can I resolve this? If you propose a fix, please make it concise.

RMWMStevens · 2025-12-07T21:07:03Z

GenHub/GenHub/Features/Content/Services/ContentDiscoverers/ModDBDiscoverer.cs

+        var detailUrl = href.StartsWith("http") ? href : ModDBConstants.BaseUrl + href;
+
+        // Extract author
+        var authorLink = item.QuerySelector("a[href*='/members/'], a[href*='/company/']");


This seems familiar to some of the constants I've seen. Do you plan on using such constants here as well?

RMWMStevens · 2025-12-07T21:08:42Z

GenHub/GenHub/Features/Content/Services/ContentResolvers/CNCLabsMapResolver.cs

+/// Parses HTML detail pages and generates content manifests.
 /// </summary>
-public class CNCLabsMapResolver(HttpClient httpClient, IContentManifestBuilder manifestBuilder, ILogger<CNCLabsMapResolver> logger) : IContentResolver
+public partial class CNCLabsMapResolver(


Why is this a partial class now?
Usually that happens by accident in my experience. Did it?

RMWMStevens · 2025-12-07T21:13:44Z

GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs

+    /// <summary>
+    /// Parses the HTML detail page for a ModDB mod and extracts all relevant details.
+    /// </summary>
+    private async Task<GenHub.Core.Models.ModDB.MapDetails> ParseModDetailPageAsync(


RMWMStevens · 2025-12-07T21:16:32Z

GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs

+        // Ensure absolute URL
+        if (!string.IsNullOrEmpty(downloadUrl) && !downloadUrl.StartsWith("http", StringComparison.OrdinalIgnoreCase))
+        {
+            downloadUrl = ModDBConstants.BaseUrl + downloadUrl;


I'm sure you've tested this, but I just wanted to make sure: you're not missing any slashes in between, right?
I'm used to seeing URL things like $"{baseUrl}/{downloadUrl} for example. Again, no criticism, just making sure.

RMWMStevens · 2025-12-07T21:26:00Z

GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs

+            return 0;
+        }
+
+        return unit switch


I've seen this a couple of times as well. Maybe that applies to the whole function. Do you need this to be duplicated?

RMWMStevens · 2025-12-07T21:30:17Z

GenHub/GenHub/Features/Content/Services/Publishers/CNCLabsManifestFactory.cs

+/// <param name="Rating">Content rating.</param>
+/// <param name="TargetGame">Target game (Generals or Zero Hour).</param>
+/// <param name="ContentType">Content type (Map or Mission).</param>
+public record MapDetails(


Why is there a second record MapDetails? They're 90% the same, why not just allow the other two to be null and check inside of the one that allows no nulls on it whether they are or not?
Or, at least change the name to reflect the different implementations.

RMWMStevens · 2025-12-07T21:32:36Z

GenHub/GenHub/Features/Content/Services/Publishers/ModDBManifestFactory.cs

+        manifest.AddRemoteFileAsync(
+            fileName,
+            details.DownloadUrl,
+            ContentSourceType.RemoteDownload).GetAwaiter().GetResult();


Why can't this be awaited? I get that the method is synchronous, and the interface probably forces this, but just make it all async and do return Task.CompletedTask from the classes that don't need it.

RMWMStevens · 2025-12-07T21:35:24Z

GenHub/GenHub/Infrastructure/DependencyInjection/ContentPipelineModule.cs

+        services.AddHttpClient(ModDBConstants.PublisherPrefix, static httpClient =>
+        {
+            httpClient.Timeout = TimeSpan.FromSeconds(45); // ModDB can be slower
+            httpClient.DefaultRequestHeaders.Add("User-Agent", "GenHub/1.0");


Seems like the GenHub version should be a constant, so that you can update it in one go. But that's up to you.

RMWMStevens · 2025-12-07T21:35:37Z

GenHub/GenHub/Infrastructure/DependencyInjection/ContentPipelineModule.cs

+        // Register named HTTP client for ModDB
+        services.AddHttpClient(ModDBConstants.PublisherPrefix, static httpClient =>
+        {
+            httpClient.Timeout = TimeSpan.FromSeconds(45); // ModDB can be slower


Glad to see you went with this route this time! Makes me happy :)

undead2146 added 2 commits November 30, 2025 23:21

feat: add initial CNC Labs content pipeline for map discovery and res…

3b42fee

…olution.

feat(content): Implement ModDB Content-Pipeline integration.

a41ea95

greptile-apps bot reviewed Dec 1, 2025

View reviewed changes

GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDBResolver.cs Outdated Show resolved Hide resolved

Update GenHub/GenHub/Features/Content/Services/ContentResolvers/ModDB…

03dea1c

…Resolver.cs Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

undead2146 marked this pull request as ready for review December 3, 2025 20:19

greptile-apps bot reviewed Dec 3, 2025

View reviewed changes

RMWMStevens reviewed Dec 7, 2025

View reviewed changes

		_logger.LogWarning("Could not extract submission date, using current date");
		return DateTime.UtcNow;

feat(content): implement ModDB and CnCLabs content-pipeline integration #204

Are you sure you want to change the base?

feat(content): implement ModDB and CnCLabs content-pipeline integration #204

Uh oh!

Conversation

undead2146 commented Dec 1, 2025

Uh oh!

undead2146 commented Dec 1, 2025

Uh oh!

greptile-apps bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Overview

Greptile Summary

Major Changes

Issues Found

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

greptile-apps bot commented Dec 1, 2025 •

edited

Loading