User:Davidchen55/sandbox

SeaweedFS is a simple and highly scalable distributed file system, to store and serve billions of files fast! SeaweedFS implements an object storage with O(1) disk seek, transparent cloud integration, and an optional Filer with POSIX interface, supporting S3 API, Rack-Aware Erasure Coding for warm storage, FUSE mount, Hadoop compatible, WebDAV.

Object storage
SeaweedFS started as a distributed object storage to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages file volumes, and it lets these volume servers manage files and their metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers, allowing faster file access (O(1), usually just one disk read operation). Client applications directly read or write with the object-based storage system via HTTP REST APIs.

File system
SeaweedFS' file system runs on top of the object storage system. The SeaweedFS Filer server maps the directories and file names of the file system to objects stored within SeaweedFS volume servers. Filer is a separate linearly-scalable stateless server with customizable metadata stores on an embedded LevelDB, or other external data stores, such as MySql, Postgres, Redis, Etcd, Cassandra, LevelDB, MemSql, TiDB, TiKV, CockroachDB, etc. Metadata can be exported and imported between metadata stores. This provides flexible options to balance performance, atomicity, scalability, and maintenance. Clients can access the Filer via HTTP REST APIs, or mount the POSIX-compatible file system using FUSE, or via APIs for Hadoop Compatible File System to store data on Hadoop, HBase, Presto, etc.

Transparent Cloud Integration
SeaweedFS can optionally move warm data on to cloud. Hot data can be accessed with fast local access, and warm data can be stored with the cloud storage's elastic capacity. With fixed number of servers, this transparent cloud integration literally gives SeaweedFS unlimited capacity.

History
SeaweedFS started by implementing Facebook's Haystack design paper. Also, SeaweedFS implements erasure coding with ideas from f4: Facebook’s Warm BLOB Storage System.