Simulation-in-loop adversary (#243)

* Create a folder for attack.composer.

* Add composer modules for unbounded patch adversary.

* Add config of Adam optimizer.

* Add LoadCoords for patch adversary.

* Add a config of unbounded patch adversary.

* Add a datamodule config for carla patch adversary.

* Fix the simple Linf projection.

* Add composer module PertImageBase for Lp bounded patch adversary.

* Add config of lp-bounded patch adversary.

* Add a fake renderer composer module.

* Teardown a test dataset gracefully for the rendering-in-loop adversary.

* Add configs of simulation-in-loop adversary.

* Add a datamodule config for CARLA patch rendering.

* Update CarlaDataset config.

* Add a composer.visualize switch to see intermediate images.

* Revert "Teardown a test dataset gracefully for the rendering-in-loop adversary."

This reverts commit a5ffef3.

* Revert "Add a composer.visualize switch to see intermediate images."

This reverts commit a17e224.

Author: mzweilin

mart/attack/composer/patch.py

@@ -14,6 +14,7 @@
     "PertRectSize",
     "PertExtractRect",
     "PertRectPerspective",
+    "FakeRenderer",
     "PertImageBase",
 ]
 
@@ -75,6 +76,29 @@ def forward(self, perturbation, input, coords):
         return perturbation
 
 
+class FakeRenderer(torch.nn.Module):
+    """Replace image with a re-rendered image, but keep the gradient on the perturbation."""
+
+    def forward(self, perturbation, input, renderer):
+        """Use the same perturbation and target.coordinates to re-render input in Simulation.
+
+        perturbation is the rectangular patch.     or a masked frame, with rectangular coordinates
+        in target, so we can extract the rectangle patch for rendering. input is the digitally
+        composed frame. target should include everything that needs to re-render a frame with the
+        perturbation.
+        """
+
+        with torch.no_grad():
+            input_rendered = renderer(perturbation)
+            input_rendered = input_rendered.clamp(0, 255)
+            delta = input_rendered - input
+
+        # Fake differentiable rendering, or BPDA, but keep the mask constraint.
+        input = input + delta
+
+        return input
+
+
 class PertImageBase(torch.nn.Module):
     """Resize an image and add to perturbation."""
 

mart/configs/attack/composer/modules/fake_renderer.yaml

@@ -0,0 +1,2 @@
+fake_renderer:
+  _target_: mart.attack.composer.FakeRenderer

mart/configs/attack/object_detection_lp_patch_adversary_simulation.yaml

@@ -0,0 +1,13 @@
+defaults:
+  - object_detection_lp_patch_adversary
+
+composer:
+  modules:
+    fake_renderer:
+      _target_: mart.attack.composer.FakeRenderer
+
+  sequence:
+    seq060:
+      # Ignore output from overlay.
+      fake_renderer:
+        ["pert_image_base", "pert_rect_perspective", "target.renderer"]

mart/configs/attack/object_detection_patch_adversary_simulation.yaml

@@ -0,0 +1,13 @@
+defaults:
+  - object_detection_patch_adversary
+
+composer:
+  modules:
+    fake_renderer:
+      _target_: mart.attack.composer.FakeRenderer
+
+  sequence:
+    seq060:
+      # Ignore output from overlay.
+      fake_renderer:
+        ["pert_extract_rect", "pert_rect_perspective", "target.renderer"]

mart/configs/datamodule/carla_patch_rendering.yaml

@@ -0,0 +1,36 @@
+defaults:
+  - default.yaml
+
+train_dataset: null
+
+val_dataset: null
+
+test_dataset:
+  _target_: oscar_datagen_tools.dataset.dataset.CarlaDataset
+  simulation_run: ???
+  modality: "rgb"
+  annFile: ${.simulation_run}/kwcoco_annotations.json
+  num_insertion_ticks: 50
+  transforms:
+    _target_: mart.transforms.Compose
+    transforms:
+      - _target_: torchvision.transforms.ToTensor
+      - _target_: mart.transforms.ConvertCocoPolysToMask
+      - _target_: mart.transforms.LoadPerturbableMask
+        perturb_mask_folder: ${....simulation_run}/foreground_mask/
+      - _target_: mart.transforms.LoadCoords
+        folder: ${....simulation_run}/patch_metadata/
+      - _target_: mart.transforms.Denormalize
+        center: 0
+        scale: 255
+      - _target_: torch.fake_quantize_per_tensor_affine
+        _partial_: true
+        # (x/1+0).round().clamp(0, 255) * 1
+        scale: 1
+        zero_point: 0
+        quant_min: 0
+        quant_max: 255
+
+collate_fn:
+  _target_: hydra.utils.get_method
+  path: mart.datamodules.coco.collate_fn