arch/xtensa/esp32[-s2|-s3]: Modify the method of downloading the repository

[TangMeng12]

Directly downloading the Git repository is inconvenient for local debugging.

Directly downloading the Git repository is inconvenient for local debugging.
This will allow to automatically download external packages from the Internet.
If not set, the repo need to be download will need to provide them manually,
otherwise an error will occur and the build will be aborted.

Add USE_NXTMPDIR_ESP_REPO_DIRECTLY, with this we can use
USE_NXTMPDIR_ESP_REPO_DIRECTLY=y make which can directly use esp-hal-3rdparty
under nxtmpdir without CHECK_COMMITSHA, reset, checkout and update.

Note: Please adhere to Contributing Guidelines.

Summary

Add USE_NXTMPDIR_ESP_REPO_DIRECTLY, with this we can use
USE_NXTMPDIR_ESP_REPO_DIRECTLY=y make which can directly use esp-hal-3rdparty
under nxtmpdir without CHECK_COMMITSHA, reset, checkout and update.

nuttxworkspace
|
|- nuttx
|- apps
|- nxtmpdir

USE_NXTMPDIR_ESP_REPO_DIRECTLY=y make

ARCH
arch/xtensa/src/esp32/Make.defs
arch/xtensa/src/esp32s2/Make.defs
arch/xtensa/src/esp32s3/Make.defs
arch/risc-v/src/common/espressif/Make.defs
arch/risc-v/src/esp32c3-legacy/Make.defs

ARCH
arch/xtensa/src/esp32/Make.defs
arch/xtensa/src/esp32s2/Make.defs
arch/xtensa/src/esp32s3/Make.defs
arch/risc-v/src/common/espressif/Make.defs
arch/risc-v/src/esp32c3-legacy/Make.defs

Impact

There should be no impact.

Testing

CI

[simbit18]

@TangMeng12 Please look here
#13301

The idea behind this logic was to reduce the time needed for the jobs.

With the change you want to make, build times on CI will be longer than before.

[hartmannathan]

@TangMeng12 Please look here #13301

The idea behind this logic was to reduce the time needed for the jobs.

With the change you want to make, build times on CI will be longer than before.

Throughout NuttX, and mostly in the nuttx-apps repo, there are various optional components that might be downloaded.

I think we need to discuss (on the mailing list) and come up with a systematic approach that balances everyone's needs in a suitable way.

Some of those needs are:

• In Kconfig, there should be one permission config "Allow downloads", help text: "Allow the NuttX build system to automatically download external packages from the Internet." Suggested name: CONFIG_ALLOW_DOWNLOADS. In the build system, downloading will occur through one central function that may call wget, curl, git, or other commands as needed, and that function will return an error if CONFIG_ALLOW_DOWNLOADS is not configured, aborting the build. This is important because some developers will want to ensure there are no automatic downloads. If external packages are needed, those developers will need to provide them manually.
• Some developers may allow downloads in general, but will prefer to download some individual package manually. This allows developers to add their own customizations, choose alternate versions, alternate branches, etc. ("Out of tree downloads.") To support this, each package should have its own fine-grained permission Kconfig to enable or disable downloading that package from the internet.
• Some developers will prefer to store external packages at custom path on their workstation. To support this, each package needs a Kconfig to provide the custom path on the developer's workstation. This can be combined with the previous item.
• Some developers will prefer to allow downloads, but customize the URLs. This way, they can control which server the download will come from. The server could be a local server on their own intranet or perhaps a custom fork on the internet. To support this, each package needs a Kconfig to specify the URL.
• The CI system needs to utilize its resources in a smart way, as pointed out by @simbit18 above. So we need a Kconfig to select between downloading the release tarballs or cloning the repository.

Rather than maintaining a hodgepodge of different approaches, a systematic approach will make it easier to add new external dependencies, maintain existing ones, and handle licensing correctly.

[Laczen]

@TangMeng12 Please look here #13301

The idea behind this logic was to reduce the time needed for the jobs.

With the change you want to make, build times on CI will be longer than before.

@simbit18, can you add a line to the documentation on how to use the idea proposed in #13301?

[TangMeng12]

@TangMeng12 Please look here #13301
The idea behind this logic was to reduce the time needed for the jobs.
With the change you want to make, build times on CI will be longer than before.

Throughout NuttX, and mostly in the nuttx-apps repo, there are various optional components that might be downloaded.

I think we need to discuss (on the mailing list) and come up with a systematic approach that balances everyone's needs in a suitable way.

Some of those needs are:

• In Kconfig, there should be one permission config "Allow downloads", help text: "Allow the NuttX build system to automatically download external packages from the Internet." Suggested name: CONFIG_ALLOW_DOWNLOADS. In the build system, downloading will occur through one central function that may call wget, curl, git, or other commands as needed, and that function will return an error if CONFIG_ALLOW_DOWNLOADS is not configured, aborting the build. This is important because some developers will want to ensure there are no automatic downloads. If external packages are needed, those developers will need to provide them manually.
• Some developers may allow downloads in general, but will prefer to download some individual package manually. This allows developers to add their own customizations, choose alternate versions, alternate branches, etc. ("Out of tree downloads.") To support this, each package should have its own fine-grained permission Kconfig to enable or disable downloading that package from the internet.
• Some developers will prefer to store external packages at custom path on their workstation. To support this, each package needs a Kconfig to provide the custom path on the developer's workstation. This can be combined with the previous item.
• Some developers will prefer to allow downloads, but customize the URLs. This way, they can control which server the download will come from. The server could be a local server on their own intranet or perhaps a custom fork on the internet. To support this, each package needs a Kconfig to specify the URL.
• The CI system needs to utilize its resources in a smart way, as pointed out by @simbit18 above. So we need a Kconfig to select between downloading the release tarballs or cloning the repository.

Rather than maintaining a hodgepodge of different approaches, a systematic approach will make it easier to add new external dependencies, maintain existing ones, and handle licensing correctly.

I have updated the commit based on your suggestions. In the commit, the default value of CONFIG_ALLOW_DOWNLOADS is y, which has no impact on the current logic. When developers want to perform local debugging, they can choose not to set CONFIG_ALLOW_DOWNLOADS, and in this case they can configure the repo locally.

[simbit18]

@TangMeng12 , allow me to make a few observations.

As already mentioned #13301, the current system was designed to speed up and reduce errors during the cloning/download step (also possible with Curl downloads if desired) on our CI.

This system on CI makes sense when the packages need to be used in multiple configurations (currently only for Arch Expressif )

jobs Linux (risc-v-04), Linux (xtensa-01), Linux (xtensa-02)

With this simple check

ifneq ($(wildcard $(NXTMPDIR)/.*),)

We can insert any package (if necessary) from the Nuttx or Apps repository into the storage directory.

Storage is enabled with

-S adds the nxtmpdir folder for third-party packages.

in the build.yml workflow on GitHub

nuttx/.github/workflows/build.yml

Line 194 in 201406b

./cibuild.sh -c -A -N -R -S testlist/${{matrix.boards}}.dat

and if desired, you can use it locally
example

./tools/configure.sh -l -S esp32-devkitc:nsh

nuttx/tools/configure.sh

Line 35 in 201406b

-S adds the nxtmpdir folder for third-party packages.

At the moment, simply using CONFIG_ALLOW_DOWNLOADS (must be added to all Expressif configurations!) does not meet the requirements of our CI!

@TangMeng12 Converted back to draft to avoid accidental merging.

The impact must be considered !!!

[acassis]

@TangMeng12 Please look here #13301
The idea behind this logic was to reduce the time needed for the jobs.
With the change you want to make, build times on CI will be longer than before.

Throughout NuttX, and mostly in the nuttx-apps repo, there are various optional components that might be downloaded.

I think we need to discuss (on the mailing list) and come up with a systematic approach that balances everyone's needs in a suitable way.

Some of those needs are:

* In Kconfig, there should be one permission config "Allow downloads", help text: "Allow the NuttX build system to automatically download external packages from the Internet." Suggested name: `CONFIG_ALLOW_DOWNLOADS`. In the build system, downloading will occur through one central function that may call wget, curl, git, or other commands as needed, and that function will return an error if `CONFIG_ALLOW_DOWNLOADS` is not configured, aborting the build. This is important because some developers will want to ensure there are no automatic downloads. If external packages are needed, those developers will need to provide them manually.

* Some developers may allow downloads in general, but will prefer to download some individual package manually. This allows developers to add their own customizations, choose alternate versions, alternate branches, etc. ("Out of tree downloads.") To support this, each package should have its own fine-grained permission Kconfig to enable or disable downloading that package from the internet.

* Some developers will prefer to store external packages at custom path on their workstation. To support this, each package needs a Kconfig to provide the custom path on the developer's workstation. This can be combined with the previous item.

* Some developers will prefer to allow downloads, but customize the URLs. This way, they can control which server the download will come from. The server could be a local server on their own intranet or perhaps a custom fork on the internet. To support this, each package needs a Kconfig to specify the URL.

* The CI system needs to utilize its resources in a smart way, as pointed out by @simbit18 above. So we need a Kconfig to select between downloading the release tarballs or cloning the repository.

Rather than maintaining a hodgepodge of different approaches, a systematic approach will make it easier to add new external dependencies, maintain existing ones, and handle licensing correctly.

@TangMeng12 @simbit18 @hartmannathan another item to include in this listing is the Mirror support. I created the https://github.com/NuttX/apps-mirror-pkgs to have a backup/mirror of the packages that NuttX Apps is using, this way the build never will fail case some URL is momentarily inaccessible or if the site disappears.

[TangMeng12]

As already mentioned 13301, the current system was designed to speed up and reduce errors during the cloning/download step (also possible with Curl downloads if desired) on our CI.

Now, I’ve switched to downloading the zip file, but I still kept the optimizations from 13301.

In this way, it not only retains the optimizations mentioned by @simbit18 in 13301, but also stays consistent with mostly in the nuttx-apps repo, while enabling local configuration for git repository debugging.

[tmedicci]

In summary, I still couldn't figure out why this PR is needed. I think it introduces more maintenance burden and is error-prone by decoupling the git submodules dependencies.