Eagerly processing tree artifacts in manifest.read_entries_from?

[rbeasley-avgo]

While working on PR #928 in service of #927, I discovered #932. Consequently I'm closing PR #928 in favor of providing a more comprehensive fix. That's what this discussion will be about.

So, what if we moved the handling of tree artifacts from each builder's add_tree method to manifest.read_entries_from? Here's the gist:

• When read_entries_from encounters a tree artifact, it will walk the tree artifact and synthesize manifest entries.
  • To enable gradual implementation, synthesis could be (internally) configurable. As builders adopt the option, they can delete their add_tree methods.
  • Care must be taken w/ directories. RPM requires special handling of directories, since any directory defined by an RPM is, in terms of RPM installation lifecycle, owned by the RPM. (That is, uninstalling the RPM would remove the directory.) With that, consider limiting synthesis to files and symlinks (i.e. don't synthesize directories). Alternatively, let the caller choose whether or not the synthesize directories.

Here's a hacky sketch:

def read_entries_from(fh, map_tree_entries: Union[bool, Sequence[str]] = False):
  raw_entries = json.loads(fh.read())
  if not map_tree_entries:
    return [ManifestEntry(**entry) for entry is raw_entries]
  if map_tree_entries == True:
    # We'll map all entry types encountered when walking a tree.
    map_tree_entries = set([ENTRY_IS_FILE, ENTRY_IS_LINK, ENTRY_IS_DIR])
  else:
    map_tree_entries = set(map_tree_entries)

  entries = []
  for raw_entry in raw_entries:
    entry = ManifestEntry(**raw_entry)
    if entry.type == ENTRY_IS_TREE:
      # TODO
      # - Defend against directory collisions, such as when a tree's
      #   destination overlaps another ENTRY_IS_DIR's. Directory entries
      #   from the user's manifest should take precedence over any that
      #   we discover while walking trees.
      for (root, dirs, files) in os.walk(entry.src):
        dirs.sort()  # Otherwise ignored; we'll add each dir as it's visited.
        files.sort()
        if ENTRY_IS_TREE in map_tree_entries:
          entries.append(ManifestEntry(ENTRY_IS_DIR, ...))
        for f in files:
          # if f is a symlink and ENTRY_IS_LINK in map_tree_entries: ...
          # if f is a file and ENTRY_IS_FILE in map_tree_entries: ...
          pass
    else:
      entries.append(entry)

What do folks think?