I am looking for open source software that can be used for expose a file system
ID: 656612 • Letter: I
Question
I am looking for open source software that can be used for expose a file system tree as a service with an S3 (subset) compatible API.
I need something that allows software (i.e. GenomeSpace which has has S3 client capability) to remotely read and write files from the file system of my server as if my server was a full-blown S3 server.
Redundancy, etcetera is not a requirement. This is not full-function Object Storage: it is simply making files and directories look like object storage to an S3 client running on another machine (probably in the same data centre).
I should have mentioned that my server will run some flavour of Linux.
Explanation / Answer
You can try s3fs for Linux, or possibly wins3fs for Windows (though its last update is forever old).
There are also commercial alternatives like ExpanDrive, but these fail your "open source" criteria.
I would caution you to test whatever solution you pick rather carefully. I have used both open and proprietary approaches to "S3 as a file system," and have never been satisfied with their reliability. Trying to couple what fundamentally is a remote, high-latency resource with a software system (the file system) that expects a very local, low-latency resource--well, the two often don't match up very well. I've found that I can copy files beautifully to and from S3...and then a few minutes later, when I try to copy more files, the connection is somehow broken. This is an an environment with 50 Mbps network connections, in which software like ssh and scp have no problems whatever. My experience is that trying to treat S3 as a file system seemed like the very simplest and most fool-proof approach...but in practice, it often did not work robustly, and I spent more time restarting my S3 file system connection than it was worth. I have since switched to command-line tools like s3cmd which do not suffer the same drop-outs. YMMV.
Related Questions
Navigate
Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.