Add mechanism to allow excluding sourceset directories

[staktrace]

Sort of related to #208 and #242 . Currently the exclude mechanism allow you to exclude paths, however those paths are still considered as part of the "task input" from gradle's perspective. This is because even if you do something like:

      project.tasks.withType(FormatTask::class.java) { task ->
        task.exclude {
          it.file.path.contains("/generated/")
        }
      }

which excludes all ".../generated/..." files, the Gradle code here still takes the root of the filecollection as a task input.

In Gradle 8+ this can cause issues because it forces task ordering when you have tasks reading/writing the same files. So for example if you apply plugin Foo and it generates files into $buildDir/generated/foo/ then gradle will force you to use dependsOn or mustRunAfter to define an ordering between kotlinter and Foo even if you use the above exclude mechanism, because it still considered kotlinter as "using" the generated folder that Foo is writing to.

After some experimentation, it seems that this can be corrected by applying a filter earlier, on this line by doing something like ... project.provider { directorySet.filterNot { dir -> dir.path.contains("/generated/") } } ... instead of project.provider { directorySet }. This filters out the generated folder that Foo adds to the sourcesets, and so Gradle never sees it as an input to kotlinter at all.

@jeremymailen Would you be open to taking a patch along these lines? I was thinking of (a) adding a boolean flag to the plugin extension like excludeBuildFiles = false and (b) if the flag is set to true, I would modify the project.provider as above but to filter out any source set directories that are inside the project build dir. Of course more complex options are possible by allowing consumers of kotlinter to provide a custom exclusion filter in the extension but I don't know if you want to allow that. The build-dir-exclusion flag would be sufficient for our purposes.

[jeremymailen]

Thanks for the great explanation and I'm always open to contributions.

I take it the main need here is for people to be able exclude source code which is generated by other gradle plugins or frameworks? Is the most common pattern for these frameworks to generate source code which is part of the existing main sourceSet and place those kt files in the gradle build directory for the module? If so, this might be a pragmatic way to have a (I assume global) setting which would do that. Initially I was surprised frameworks added generated code to existing sourceSets instead of creating new ones, but I guess there might be bidirectional dependencies with non-generated source which might make that a challenge.

[staktrace]

Thanks for taking the time to respond!

I take it the main need here is for people to be able exclude source code which is generated by other gradle plugins or frameworks?

Yes, in a way that satisfies the stricter task-ordering requirements in gradle 8+.

Is the most common pattern for these frameworks to generate source code which is part of the existing main sourceSet and place those kt files in the gradle build directory for the module?

I don't know if that's the most common pattern, but I do see it happen with a few different plugins that we're using. In at least one of these, yes, there is a bidirectional dependency with non-generated source. But even if it does end up in a different sourceset, the linting task for that sourceset would still run by default, right? Were you thinking that consumers would disabling the linting task for the generated code sourceset explicitly?

I ran into some minor difficulties with my original approach and am refining it somewhat. Once I have it fully validated I'll put up a PR and we can iterate on it as you see fit.

[jeremymailen]

Ok, this is good information.
Yes, I assumed if the generated source was in a unique sourceSet, people could just disable those tasks, but I don't think frameworks are mostly working this way.

I like the approach you are proposing. It makes a lot of sense we wouldn't want to lint files placed in a build directory. Thanks for the proposal and looking forward to the PR.

[staktrace]

Sorry for the long delay here. In the end I couldn't get this working in a generic enough way that worked for all scenarios. I have a formulation that worked in a lot of cases for us, but there were still some repos where it failed and I was having a really hard time figuring out what gradle was doing to cause it to fail. In the end I didn't have time to keep digging and so worked around it in those projects. I'm providing the formulation here in case it's useful to somebody but at this point I don't have cycles to get it to the point where I could propose a change to kotlinter. For context, this code is applied in one our organization's "convention plugins" (or rough equivalent), so that it overrides behaviour in projects that have kotlinter/ktfmt applied. The idea was to upstream this so we wouldn't need this override.

    project.pluginManager.withPlugin(Plugins.KOTLIN_JVM) {
      val sourceSets = project.extensions.getByType(KotlinProjectExtension::class.java).sourceSets
      sourceSets.configureEach { sourceSet ->
        val directorySet = sourceSet.kotlin
        val sourceFilter = { task : SourceTask ->
          task.logger.debug(
            "Stripping build dir from task ${task.path} for source set ${sourceSet.name}"
          )
          task.setSource(project.provider {
            // Make a copy of the existing source set as best we can, but with the source directories
            // filtered to exclude anything inside the build directory. Note in particular that we
            // do not want this to produce a list of individual files, because ktfmt in particular
            // sets the task output directory to the same as the task source directories, and if that's
            // a list of individual files then Gradle complains. So we want to ensure we filter at
            // the directory level.
            val filteredSourceDirectorySet = project.objects.sourceDirectorySet(directorySet.name, "Filtered copy of ${directorySet.name}")
            filteredSourceDirectorySet.filter.setIncludes(directorySet.filter.includes)
            filteredSourceDirectorySet.filter.setExcludes(directorySet.filter.includes)
            filteredSourceDirectorySet.setSrcDirs(
              directorySet.sourceDirectories.filterNot {
                dir -> dir.absolutePath.startsWith(project.buildDir.absolutePath)
              }
            )
            filteredSourceDirectorySet
          })
        }

        // Note that we use withPlugin blocks here to ensure our withTask(SourceTask) configurer
        // gets registered after the plugin's own task configuration. This is necessary because
        // we are overwriting the plugin's own task configuration and that doesn't happen correctly
        // if we run our configuration block before the plugin's own task configuration.

        project.pluginManager.withPlugin(Plugins.KOTLINTER) {
          val kotlinterSuffix = directorySet.name.split(" ").first().replaceFirstChar(Char::titlecase)
          project.tasks.withType(SourceTask::class.java).matching {
            it.name == "formatKotlin${kotlinterSuffix}" || it.name == "lintKotlin${kotlinterSuffix}"
          }.configureEach { task ->
            sourceFilter(task)
          }
        }

        project.pluginManager.withPlugin(Plugins.KTFMT) {
          val ktfmtSuffix =
            sourceSet.name.split(" ").joinToString("") { it.replaceFirstChar(Char::uppercase) }
          project.tasks.withType(SourceTask::class.java).matching {
            it.name == "ktfmtFormat${ktfmtSuffix}" || it.name == "ktfmtCheck${ktfmtSuffix}"
          }.configureEach { task ->
            sourceFilter(task)
          }
        }
      }
    }

[jeremymailen]

Thank you for the detailed formula. Definitely helpful in thinking about an enhancement.

[andrewaylett]

This becomes more obviously a problem with Gradle 9 and especially with buildSrc projects. Gradle will generate accessors for the Kotlin DSL for plugins referenced by the build, but also for kts plugins defined in the project. And Gradle's configuration cache can't store closures, so most of the "easy" ways to exclude built files throw up configuration cache errors.

Thanks to the hints in this and other threads on the subject I've managed to get things working for my project -- but a mechanism for excluding all generated files would be a lot easier. No promises, but if I get a round tuit then I might see if I can work something out.

[jeremymailen]

It would be nice to add this feature especially alongside Gradle 9 support.
PR contributions are very helpful since my time to work on it right now is a bit thin and I'm not actively working on software day to day that uses the JVM ecosystem right now so I don't stay as up to date as I should.