Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add local S3 support with MinIO #54

Merged
merged 10 commits into from
Apr 2, 2024
Merged

Conversation

chrisr-st
Copy link
Contributor

@chrisr-st chrisr-st commented Mar 20, 2024

MinIO allows you to run an S3 service locally. Access to AWS is not needed.

After running docker-compose for the data source services.

Edit the /etc/hosts file on your system and make a reference for

127.0.0.1 s3.docker

Open your browser and navigate to the admin console http://s3.docker:10001 and login to the MinIO admin console.

username: root
password: password
  1. Create an appropriate bucket for each project as needed.
  2. Modify your project's .env file to match the following:
FILESYSTEM_DRIVER=s3

AWS_ACCESS_KEY_ID=root
AWS_SECRET_ACCESS_KEY=password
AWS_DEFAULT_REGION=us-east-1
AWS_BUCKET=[project-bucket-name]
AWS_ENDPOINT=http://s3.docker: 10000
AWS_USE_PATH_STYLE_ENDPOINT=true

Copy link
Member

@iBotPeaches iBotPeaches left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks cool

README.md Outdated Show resolved Hide resolved
docker/data-source-services/docker-compose.yml Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
@iBotPeaches
Copy link
Member

I finally figured out why my xdebug stopped working. We stole port 9000 from it. I guess ultimately we should have moved to 9003 when 3 came out, but we didn't.

So now I want to audit our ports to see where this can fit that doesn't collide with other stuff.

@iBotPeaches
Copy link
Member

I kept spending more time on this trying to figure out why I couldn't just use localhost with a port.

  • the mix between port 9090/9000 confused me, 9000 is xdebug so we gotta figure out a way to move that.
  • our container names having underscores by default prevents us from using default container networking (ie something like sourcetoad_minio), @erik-perri figured this out first
  • Saying all of that, we hit the temporary url thing you linked out.

So I think I'm happy with just finding a new port for 9000 and this might be good to go.

@chrisr-st
Copy link
Contributor Author

chrisr-st commented Mar 26, 2024

I kept spending more time on this trying to figure out why I couldn't just use localhost with a port.

* the mix between port 9090/9000 confused me, 9000 is xdebug so we gotta figure out a way to move that.

* our container names having underscores by default prevents us from using default container networking (ie something like `sourcetoad_minio`), @erik-perri figured this out first

* Saying all of that, we hit the temporary url thing you linked out.

So I think I'm happy with just finding a new port for 9000 and this might be good to go.

Moved it to 10000 and 10001. I didn't see anything using those, and after some Googling, it didn't look like anything we may potentially use in the future would use them.

Copy link
Member

@iBotPeaches iBotPeaches left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-ran my tests and my xdebug works. This looks good to me.

* `AWS_ACCESS_KEY_ID=root`
* `AWS_SECRET_ACCESS_KEY=password`
* `AWS_DEFAULT_REGION=us-east-1`
* `AWS_BUCKET=[project-bucket-name]`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if DevopsToolKit is the right place for this, or if it'd be better in project initial setup or something once this gets added to template, but it'd be nice if this could spin up the initial bucket based on the env configuration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've set it up like that for previous projects, so if that's a direction we want to move, it shouldn't be an issue.

@iBotPeaches iBotPeaches requested review from Vacoris and Bankamah April 1, 2024 12:55
@iBotPeaches iBotPeaches changed the title Add local s3 support with minIO Add local S3 support with MinIO Apr 2, 2024
@iBotPeaches iBotPeaches enabled auto-merge (squash) April 2, 2024 15:06
@iBotPeaches iBotPeaches merged commit c48098a into master Apr 2, 2024
9 checks passed
@iBotPeaches iBotPeaches deleted the feature/minIO-to-mock-S3 branch April 2, 2024 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

7 participants