Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
207 changes: 144 additions & 63 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
[![CircleCI](https://circleci.com/gh/johannesboyne/gofakes3.svg?style=svg)](https://circleci.com/gh/johannesboyne/gofakes3)
[![Codecov](https://codecov.io/gh/johannesboyne/gofakes3/branch/master/graph/badge.svg)](https://codecov.io/gh/johannesboyne/gofakes3)

![Logo](/GoFakeS3.png)
Expand Down Expand Up @@ -26,7 +25,78 @@ For production environments, consider more established solutions. Some recommend

## How to use it?

### Example (aws-sdk-go version 1)
### Example with AWS SDK v2 (Recommended)

```golang
import (
"context"
"crypto/tls"
"net"
"net/http"
"net/http/httptest"
"net/url"
"time"

"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3mem"
)

// Set up gofakes3 server
backend := s3mem.New()
faker := gofakes3.New(backend)
ts := httptest.NewServer(faker.Server())
defer ts.Close()

// Setup AWS SDK v2 config
cfg, err := config.LoadDefaultConfig(
context.TODO(),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("ACCESS_KEY", "SECRET_KEY", "")),
config.WithHTTPClient(&http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
},
}),
config.WithEndpointResolverWithOptions(
aws.EndpointResolverWithOptionsFunc(func(_, _ string, _ ...interface{}) (aws.Endpoint, error) {
return aws.Endpoint{URL: ts.URL}, nil
}),
),
)
if err != nil {
panic(err)
}

// Create an Amazon S3 v2 client, important to use o.UsePathStyle
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.UsePathStyle = true
})

// Create a new bucket
_, err = client.CreateBucket(context.TODO(), &s3.CreateBucketInput{
Bucket: aws.String("newbucket"),
})
if err != nil {
panic(err)
}

// Upload an object
_, err = client.PutObject(context.TODO(), &s3.PutObjectInput{
Body: strings.NewReader(`{"configuration": {"main_color": "#333"}, "screens": []}`),
Bucket: aws.String("newbucket"),
Key: aws.String("test.txt"),
})
if err != nil {
panic(err)
}

// ... accessing of test.txt through any S3 client would now be possible
```

### Example with AWS SDK v1 (Legacy)

```golang
// fake s3
Expand Down Expand Up @@ -68,95 +138,106 @@ _, err = s3Client.PutObject(&s3.PutObjectInput{
// ... accessing of test.txt through any S3 client would now be possible
```

### Example for V2 (aws-sdk-go-v2)

```golang
backend := s3mem.New()
faker := gofakes3.New(s3Backend, gofakes3.WithHostBucket(true))
ts := httptest.NewServer(faker.Server())
defer ts.Close()
Please feel free to check it out and to provide useful feedback (using github
issues), but be aware, this software is used internally and for the local
development only. Thus, it has no demand for correctness, performance or
security.

// Difference in configuring the client
## Connection Options

// Setup a new config
cfg, _ := config.LoadDefaultConfig(
context.TODO(),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("KEY", "SECRET", "SESSION")),
config.WithHTTPClient(&http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
// Override the dial address because the SDK uses the bucket name as a subdomain.
DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) {
dialer := net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}
s3URL, _ := url.Parse(s3Server.URL)
return dialer.DialContext(ctx, network, s3URL.Host)
},
},
}),
config.WithEndpointResolverWithOptions(
aws.EndpointResolverWithOptionsFunc(func(_, _ string, _ ...interface{}) (aws.Endpoint, error) {
return aws.Endpoint{URL: ts.URL}, nil
}),
),
)
There are different ways to connect to your local GoFakeS3 server:

// Create an Amazon S3 v2 client, important to use o.UsePathStyle
// alternatively change local DNS settings, e.g., in /etc/hosts
// to support requests to http://<bucketname>.127.0.0.1:32947/...
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.UsePathStyle = true
})
### 1. Path-Style Addressing (Recommended)

Path-style is the most flexible and least restrictive approach, where the bucket name appears in the URL path:
```
http://localhost:9000/mybucket/myobject
```

With AWS SDK v2, configure this using:
```golang
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.UsePathStyle = true
})
```

Please feel free to check it out and to provide useful feedback (using github
issues), but be aware, this software is used internally and for the local
development only. Thus, it has no demand for correctness, performance or
security.

There are different ways to run locally: e.g., using DNS, using S3 path mode, or V2 setting the ENV-Var:
### 2. Virtual-Hosted Style Addressing

In this mode, the bucket name is part of the hostname:
```
os.Setenv("AWS_ENDPOINT_URL_S3", "http://localhost:9000")
http://mybucket.localhost:9000/myobject
```

S3 path mode is the most flexible and least restrictive, but it does require that you
are able to modify your client code. In Go, the modification would look like so:
This requires DNS configuration. If using `localhost` as your endpoint, add the following to `/etc/hosts` for *every bucket you want to use*:
```
127.0.0.1 mybucket.localhost
```

config := aws.Config{}
config.WithS3ForcePathStyle(true)
With AWS SDK v2, this is the default mode when not setting `UsePathStyle`:
```golang
client := s3.NewFromConfig(cfg)
```

S3 path mode works over the network by default for all bucket names.
### 3. Environment Variable (AWS SDK v2)

If you are unable to modify the code, DNS mode can be used, but it comes with further
restrictions and requires you to be able to modify your local DNS resolution.
With AWS SDK v2, you can also set an environment variable to specify the endpoint:
```
os.Setenv("AWS_ENDPOINT_URL_S3", "http://localhost:9000")
```

If using `localhost` as your endpoint, you will need to add the following to
`/etc/hosts` for *every bucket you want to fake*:
This approach works with code that doesn't directly configure the S3 client.

127.0.0.1 <bucket-name>.localhost

It is trickier if you want other machines to be able to use your fake S3 server
as you need to be able to modify their DNS resolution as well.
## Exemplary usage

### Lambda Example with AWS SDK v3 for JavaScript

## Exemplary usage
```javascript
// Using AWS SDK v3 for JavaScript
import { S3Client, CreateBucketCommand } from '@aws-sdk/client-s3';

// Create an S3 client with custom endpoint
const s3Client = new S3Client({
region: 'us-east-1',
endpoint: 'http://localhost:9000',
forcePathStyle: true, // Required for GoFakeS3
credentials: {
accessKeyId: 'ACCESS_KEY',
secretAccessKey: 'SECRET_KEY'
}
});

// Lambda handler using async/await
export const handler = async (event, context) => {
try {
const command = new CreateBucketCommand({
Bucket: 'my-bucket'
});

const response = await s3Client.send(command);
return response;
} catch (error) {
console.error('Error:', error);
throw error;
}
};
```

### Lambda Example
### Legacy Lambda Example (AWS SDK v2 for JavaScript)

```javascript
var AWS = require('aws-sdk')
var AWS = require('aws-sdk')

var ep = new AWS.Endpoint('http://localhost:9000');
var s3 = new AWS.S3({endpoint: ep});
var s3 = new AWS.S3({
endpoint: ep,
s3ForcePathStyle: true // Recommended for GoFakeS3
});

exports.handle = function (e, ctx) {
s3.createBucket({
Bucket: '<bucket-name>',
Bucket: 'my-bucket',
}, function(err, data) {
if (err) return console.log(err, err.stack);
ctx.succeed(data)
Expand Down
7 changes: 4 additions & 3 deletions backend.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@ package gofakes3

import (
"encoding/hex"
"errors"

"io"
"time"

"github.com/aws/aws-sdk-go/aws/awserr"
)

const (
Expand Down Expand Up @@ -359,7 +359,8 @@ func MergeMetadata(db Backend, bucketName string, objectName string, meta map[st
// get potential existing object to potentially carry metadata over
existingObj, err := db.GetObject(bucketName, objectName, nil)
if err != nil {
if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() != string(ErrNoSuchKey) {
var nsk *ErrorResponse
if errors.As(err, &nsk) && nsk.Code != "NoSuchKey" {
return err
}
}
Expand Down
5 changes: 1 addition & 4 deletions backend/s3afero/single.go
Original file line number Diff line number Diff line change
Expand Up @@ -208,10 +208,7 @@
return nil, err
}

hash, err := hasher.Sum(nil), nil
if err != nil {
return nil, err
}
hash := hasher.Sum(nil)

Check warning on line 211 in backend/s3afero/single.go

View check run for this annotation

Codecov / codecov/patch

backend/s3afero/single.go#L211

Added line #L211 was not covered by tests

return &Metadata{
objectPath,
Expand Down
91 changes: 91 additions & 0 deletions example/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# GoFakeS3 Examples with AWS SDK v2

This directory contains examples that demonstrate how to use GoFakeS3 with AWS SDK v2.

## Examples Included

1. **Integrated Example** (`main.go`) - Creates an in-memory S3 server and a client in the same process
2. **Standalone Server** (`server.go`) - Runs a GoFakeS3 server on a specific port
3. **Client Example** (`client.go`) - Connects to a running GoFakeS3 server

## Example 1: Integrated Server and Client

This example creates an in-memory S3 server and client in the same process:

```sh
go run main.go
```

### What This Demonstrates

1. Setting up a temporary in-memory S3 server
2. Configuring AWS SDK v2 to connect to the server
3. Basic S3 operations: create bucket, upload, download, list, and delete objects

### Expected Output

```
GoFakeS3 server running at: http://127.0.0.1:xxxxx
Created bucket: test-bucket
Uploaded object: test-bucket/hello.txt
Downloaded object content: Hello, GoFakeS3!
Objects in bucket test-bucket:
- hello.txt (size: 15 bytes, last modified: 2023-xx-xx xx:xx:xx)
Deleted object: test-bucket/hello.txt
Example completed successfully!
```

## Example 2: Standalone Server

This example runs a standalone GoFakeS3 server that you can connect to with various clients:

```sh
# Run server on default port (9000)
go run server.go

# Or specify a custom port
go run server.go :8080
```

### Features Demonstrated

1. Running GoFakeS3 as a standalone service
2. Auto-creating buckets (optional feature)
3. Graceful shutdown handling
4. Creating a default "example-bucket" on startup

## Example 3: Client Example

This example connects to a running GoFakeS3 server:

```sh
# First start the server in another terminal
go run server.go

# Then run the client (connects to localhost:9000 by default)
go run client.go

# Or connect to a custom URL
go run client.go http://localhost:8080
```

### What This Demonstrates

1. Connecting to an external GoFakeS3 server
2. Configuring AWS SDK v2 clients
3. Basic S3 operations: upload, download, and list objects

## Using with Your Own Applications

To use GoFakeS3 with your own applications:

1. Start the standalone server:
```
go run server.go
```

2. Configure your AWS SDK clients to use the server URL (e.g., http://localhost:9000)
with path-style addressing enabled.

3. Use the same credentials provided in the client examples, or configure GoFakeS3
to use your own credentials.
Loading