Revert "Enable visionOS xcodebuilds by default in CI" (#3235)

Reverts #3234

This seems to have caused some instability which I'll investigate so we
should revert for now.

Author: rnro

.github/workflows/macos_tests.yml

@@ -101,8 +101,8 @@ on:
         default: false
       visionos_xcode_build_enabled:
         type: boolean
-        description: "Boolean to enable the Xcode build targeting visionOS. Defaults to true."
-        default: true
+        description: "Boolean to enable the Xcode build targeting visionOS. Defaults to false."
+        default: false
       visionos_xcode_test_enabled:
         type: boolean
         description: "Boolean to enable the Xcode test targeting visionOS. Defaults to false."