S3 Backend for Afero

About

It provides an afero filesystem implementation of an S3 backend.

There are some other alternatives, but this implementation focuses on efficient memory usage by streaming the file download and uploads.

We are open to any improvement through issues or pull-request that might lead to a better implementation or even better testing.

Known limitations

• File appending / seeking for write is not supported because S3 doesn't support it, it could be simulated by rewriting entire files.
• Chtimes is not supported because S3 doesn't support it, it could be simulated through metadata.
• Chmod support is very limited

How to use

Note: Errors handling is skipped for brevity but a complete example is provided in the example folder

package main

import (
	"context"
	"io"
	"log"
	"os"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	s3fs "github.com/contiamo/afero-s3"
)

var (
	region = "us-west-2"
	bucket = "my-bucket"
	key    = "/path/to/file.txt"
	output = "-"
)

func main() {
	ctx := context.Background()

  // initialize the S3 client
	cfg, _ := config.LoadDefaultConfig(ctx,
		config.WithRegion(region),
	)

	s3Client := s3.NewFromConfig(cfg, func(options *s3.Options) {
		options.UsePathStyle = true
	})

	// Initialize the file system
	fs := s3fs.NewFsFromClient(bucket, s3Client)

	// And do your thing
	src, _ := fs.Open(key)
	defer src.Close()

	var out = os.Stdout
	n, _ := io.Copy(out, src)

	log.Printf("copied %d bytes", n)
}

Development

The project uses Taskfile to orchestrate the local development flow.

go install github.com/go-task/task/v3/cmd/task@latest

Install task, and then use

# see the available dev tasks
task --list

To run the test suite:

task test

To run the example code:

task run-example -- --help

Thanks

The initial code (which was massively rewritten) comes from:

• fclairamb's fork
• Which comes from wreulicke's fork
• Itself forked from aviau's fork.
• Initially proposed as an afero PR by rgarcia and updated by aviau.