Unified installer for machine images

[jdpleiness]

Overview

This PR introduces a new unified installer written in Go as outlined in PR-FAQ: Delivering Confidence.
This unified installer aims to replace the following install scripts:

• GCP installer
• GCP init
• AWS latest installer
• AWS latest init
• AWS installer
• AWS init
• General install script

Current project status

While the installer was initially scoped to address problems with the current machine image deployment pipeline, components could be easily changed to fit any new machine image deployment models that may arise in the future. New deployment models with new requirements may also result in being able to drastically simplify the installer code.

Objectives

• Reduce overall complexity by using the same installer across all Machine images (AWS, GCP, AZR).
• Proper support for at least some level of unit and integration testing.
• Reduce overall install time and boot time of images.
• Add logging to install process.
• Deprecate t-shirt image sizes by generating resource limits dynamically.
• Address the following issues:
  • https://github.com/orgs/sourcegraph/projects/316/views/1?pane=issue&itemId=18600414
  • https://github.com/orgs/sourcegraph/projects/316/views/1?pane=issue&itemId=18599170
  • https://github.com/orgs/sourcegraph/projects/316/views/1?pane=issue&itemId=18801339

Initial success at reducing project build times (see below) by roughly 43%. Build times could be reduced further if offline images were no longer needed.

Components

The installer is composed of the following components:

sginstall

The sginstall component is the main installation process. This will do the necessary system configuration (disk, kernel, user, images, containerd, k3s, etc...)

sourcegraphd

The sourcegraphd component is a system daemon that monitors the current version of sourcegraph and if an update is needed. This component could also serve a diagnostics page in the future rather than the current 404 if there is an issue booting the machine image.

sginit

The sginit component is is a systemd oneshot service that will finalize the system configuration and setup upon first boot of the image.

Progress

• Develop initial libraries to support install process (disk, helm, k3s, containerd, etc...)
• Integration tests of each library
• CLI and basic install
• Dynamic resource limits generated based on machine resources available at boot
• Offline image deployment support
• Initial Packer build
• Test with AWS install
• Test with GCP install
• Test generic install
• Build process automation
• Feature flag tests that are integration tests only
• Evaluate and align with new requirements around shifting deployment strategy
• Evaluate if offline install is still a needed feature

Build

At current time, this project utilized Git LFS to handle large files that are embedded in the final Go binary to enable offline installation. Please make sure Git LFS is installed on your system before proceeding.

Building the Go binary:

To build the Go binary, run ./build.sh from the base of the project. This will output three files:

• cmd/install/bin/sg-init
• cmd/install/bin/sourcegraphd
• bin/sginstall

These files will be used by Packer in the next step.

Building the AMI:

To create an AMI that will use the installer, run the following:
packer build -var "instance_version=5.0.4" ./packer/dev/dev-builder.pkr.hcl

This will produce an AMI using Amazon Linux 2023 that is ready to be run.

Note
The current version of the installer is pinned to Sourcegraph 5.0.4.

[jac]

nit: maybe it is just me but it would be nice if these were logically grouped to make it easier to parse related statements
e.g.

func LinkDataVolumes() error {
  {  
    mkdir
    symlink
  }

  {
    mkdir
    mkdir
    symlink
  }
}