Garage seems like a viable alternative.
Garage has been great in my homelab. It’s not quite as 1:1 with S3 but it does all the basics with some really nice features.
Where they lose comparability they have a good explanation too
+1 to Garage being great! I used it for a personal project and it worked really well. A lot of S3 data browsing clients also support it natively or just through API compatibility too
Garage has one big flaw. Access control tools are nearly non existent.
Of course their closed source alternative is called AIStor and it is crazy expensive because everyone now needs to pivot to AI
S3 compatibility is nice I guess if you need S3 compatibility but also… why would you need that?
sshfs does everything I need and compatibility is almost native.
SSHFS is a hack and has nothing to do with the proposal of S3 compatible backends
So enlighten me then, save me from my terrible hack that is working fine for me and tell me what it DOES have to do with. I thought S3 was a remote filesystem you can use, essentially Amazon’s proprietary version of webdav where you get a http bucket you can only access with aws proprietary tools. What’s the attraction? Clearly it seems like people love it, and I am getting dunked on for asking an honest question, which feels a bit unhealthy and unpleasant for the self-hosting community.
Am I supposed to be familiar with AWS infrastructure as a prerequisite for being here?
ok, to start with, if you need a POSIX interface to the filesystem, you already have an SSH connection to that server, and don’t need much stability across multiple clients, SSHFS may do just fine. For a homelab, that is likely the case.
now, if you’re hosting a web server that needs data distributed across drives/nodes, data redundancy, and the usage is primarily programmatic, closer to a CDN’s or machine learning pipeline than a single user browsing files; then you want an S3-compatible solution. The S3 API makes it easier to plug it into your application, while allowing you to migrate to a different one - which I’m actually currently doing for a MinIO deployment at work.
S3 is designed for being used by applications via API, for example you can easily save and retrieve files from it even with a JavaScript application. It is much more difficult to do the same with sshfs
If instead you use it mounted on a computer, S3 is worse because each time you need to list its contents that’s an API request, if you have hundreds of thousands of files then it’s thousands of API reuqests
Sshfs has way more overhead and doesn’t do remotely the same thing
Kubernetes storage is the reason I was looking at Minio in the past.
I mean even more options there like Longhorn or Rook-Ceph
Many cloud providers offer S3-compatible storage, so it’s a common protocol to use in applications. There are even some databases like SlateDB that fully rely on object storage for everything. Supporting more API’s is extra work (unless you’re using OpenDAL) so most people pick S3 compatible API’s because they’re the most widely supported across all cloud platforms.
why would you need that?
So you can switch to S3 if needed? Using compatible solutions means you have choice. Choice is good.
I’ve honestly never understood the need for s3 buckets. WebDAV satisfies my needs. I’m sure there are some use cases that require S3, but for the life of me I can’t think of one off the top of my head right now.
Many cloud providers offer S3-compatible storage, so it’s a common protocol to use in applications. There are even some databases like SlateDB that fully rely on object storage for everything. Being able to have local S3 compatible storage is useful if you want the storage of your local machine while still doing so over a widely compatible protocol.
A quick web search shows slatedb supports WebDAV through Rust’s object_store interface, or at least it does at first glance.
WebDAV is a wonderful standard and it is compatible with all kinds of things that seem to be overlooked. S3 has turned into this monster of a thing that’s “owned” by AWS vs a nice usable RFC that anybody can implement and know if it actually changes.
object_store does indeed also support WebDAV among a variety of other protocols, Apache Druid or Apache Pinot probably would be better examples. My only experience with WebDAV is with Nextcloud and hasn’t been that great because it has been very slow, probably should look into it sometime.
EDIT: Apparently it supports CAS, and even has a locking mechanism
I use an external S3 provider so that I only pay for the storage I use for the services I host. It’s dirt cheap, 0.00002750€ per GiB hour (excluding tax). Self-hosting something like MinIO for your app gives you the option of switching to an external provider later on, and it gives you flexibility in the location of the storage.
Which one?
In my experience I’ve mostly seen it used for a local equivalent of S3 to plop in your dev environment. It’s pretty good if your prod depends on S3 and you don’t want to deal with the cost and latency of using actual S3 buckets during development.
Shit, I am actually building a webtool and thought Minio could be a good part to be a file storage in it. What’s an good alternative?
Edit: I try “garage”
…a hard disk? you can just write data to a file
Not if you want to validate S3 compatibility for an actual future use case or, * can you imagine*, just for the fun of it.
i’ll give you the second case, but nobody should plan for putting stuff on aws with the world as it looks right now…
I use S3 with OVH at my workplace. So it’s not just aws / google.
Genuine question, what are the alternatives not called Azure/GCP?
S3 isn’t just an AWS thing anymore. It has kind of become the standard object storage protocol, and almost every cloud provider uses it aside from a few the made their own API’s (e.g. Azure Blob storage)
any european ones?
IONOS and ImpossibleCloud for instance are ones we use.
I think Hetzner and OVH also offer S3 buckets.Scaleway, Exoscale, Cyso, Contabo, UpCloud, and others too
There are numerous other vendors with S3 API-compatibility.
There’s also SeaweedFS that I’ve used as an S3 compatible fileserver
I hope chainguard keeps maintaining their fork https://github.com/chainguard-forks/minio
Oddly enough right before I saw this post some alternatives were in my news feed.
https://itsfoss.com/news/minio-moves-away-from-open-source/
Highlights are:
SeaweedFS
Garage
RustFS
Sadly I only recognized it as “the thing you put in your docker compose for integration tests”.
