User:Gonnabee/sandbox/Storj DCS

To keep things organized, please use following links:

Sources For links to sources Discussion For discussion, only article related Gonnabee (Talk) 14:50, 23 January 2022 (UTC)

Storj DCS formerly known as Tardigrade (named after the famous and basically indestructible water bears) is a cloud service offered by Storj Labs Inc. that provides an Amazon S3 compatible decentralized object storage. DCS stands for Decentralized Cloud Storage.

In contrast to conventional usually centralized cloud storage providers, Storj Labs Inc. does not own or operate any data center to store its customers data. Instead, the Storj DCS object storage space consists of a network of currently approximately 12.800 single and geographically divers storage nodes. These nodes are operated by independent individuals called Storage Node Operators (SNO) who get paid by Storj Labs Inc. for providing storage space and bandwidth for Storj DCS.

Storj Labs Inc.launched the current version (V3) of its decentralized storage initially as Tardigrade on 03/19/2020. In 2021 the service has been rebranded and re-launched as Storj DCS on 04/20/2021 with a new pricing structure and a new corporate identity.

Inception
Storj was founded in 2014 by Shawn Wilkinson after sparking an idea about disrupting cloud storage with Bitcoin technology. Together with co-founder James Lowry their presentation of a working proof of concept software demo has won 1st place in the Texas Bitcoin Conference 2014 hackathon contest and $250.000 as prize.

Initial idea
Similar to Uber or Airbnb, Storj is a player in the sharing economy space. Like Uber drivers chauffering people in their private cars or Airbnb users renting out private rooms, Storage Node Operators rent out unused hard disk space and bandwidth of their existing predominantly residential computer and internet infrastructure.

Similar services at inception❌
By the time of Storjs inception, the idea of a distributed storage cloud operated by home users had already been adopted by:


 * Space Monkey, a startup founded in 2011 that offered its customers a 1 TB hard disk as local storage for rent. When the drive was connected to the internet, it became connected to all other Space Monkey hard drives forming a distributed storage network in which all user files placed onto the local drive would get encrypted and replicated to other remote drives, which safeguarded from data loss in case of hardware failures or disaster. Space Monkey was acquired by Vivint Smart Home Inc. in 2014.


 * Vivint Smart Home Inc., a public smart home company in the United States and Canada, used the Space Monkey technology after its acquisition in 2014 for storing those large amounts of data that accrue in smart homes from IoT devices and camera recordings, such as surveillance or doorbell cams.

Tardigrade V2
Prior to the actual V3 version of the Storj DCS network, extensive experience has been gathered with version V2 in which node operators were called farmers while storage space customers were called renters. Back then the Storj storage network consisted of a system for contracting and negotiating on top of a distributed hash table (Kademlia). On this permissionless platform renters and farmers would broadcast their contract offers and then negotiate directly with interested parties to enter into a contract and execute it.

Farmers were paid through Storj token (SJCX), which at that time was a Counterparty token on top of the Bitcoin blockchain.

After extensive alpha and beta testing, developers at Storj Labs Inc. came to the conclusion to drop the existing platform and to completely rebuild the system with an improved concept. Due to a massive increase in Bitcoin fees, payment was moved away from Counterparty token to Storj ERC-20 token (STORJ) on Ethereum. This switch was made with the intention to assure the ability to process mass micropayments to a large number of node operators in the future.

Actual Product (V3) ❌
The current version V3 is a complete rebuild after extensive experience has been gathered with version V2. The new concept geared towards better scaling and to provide Amazon S3 compatibility.

This area could include:


 * Differences to conventional centralized providers
 * Benefits, advantages


 * Maybe also something about incidents and if files have been lost so far
 * Problems, issues, disadvantages, e.g. Problem Eth fees

General overview
The Storj DCS object storage is a network of computers with different roles to allow uploads and downloads and to maintain a zero knowledge state as well as to ensure the durability and integrity of the uploaded data.

Independent node operators provide the entire storage space and up- and download bandwidth. High availability database clusters, called satellites, keep track of all data in the distributed network and of the availability and integrity of the data by constantly auditing the nodes. Repair workers maintain the redundancy of data in the network by recreating and redistributing data pieces whenever the availability falls below a specified threshold.

Upload
Before an object gets uploaded to the network, it gets encrypted with AES-256-GCM symmetric encryption and split into segments. Each segment gets split into 80 pieces (shards) that are erasure coded with the Reed Solomon erasure coding scheme. These shards get uploaded to 80 different accidentally selected nodes which are located all over the world.

Download
Whenever a client requests to download data from Storj DCS, then it will download the erasure coded pieces, reconstruct segments from them and rebuild the file from the segments. Because the pieces are erasure coded, 29 shards out of 80 are sufficient to fully rebuild a complete segment.

Storage
The erasure coded pieces are stored on nodes operated by independent node operators (SNO). The node operators provide disk space and bandwidth for up- and download and get paid with Storj ERC-20 tokens for providing this service. To maintain data availability and integrity, Storj Labs Inc. is running a ledger on so called satellites, which are high availability database clusters which are located independently from each other in different areas of the world. The satellites constantly check the nodes online status, the availability and integrity of single pieces via audits to make sure that there are always enough intact pieces available to rebuild any given data segment at all times. If the number of pieces fall below a certain threshold, the repair worker servers will get triggered for re-creation and re-distribution of pieces.

Repair
To safeguard customer data against node churn, node unavailability or nodes losing data permanently repair workers will recreate and redistribute pieces, whenever the ledger suggests, that the number of pieces for a specific segment has fallen below a certain threshold. Then the repair worker server will reconstruct that segment and recreate new pieces from it which will then get redistributed to new nodes. This process assures that there are enough pieces for reconstruction for any segment any time.

Redundancy
Storj DCS ensures redundancy by erasure coding and not by data replication. Every file gets encrypted and split into segments. Every segment gets split into 80 unique erasure coded pieces from which any 29 pieces are sufficient to reconstitute the whole segment. 29 out of 80 results in an expansion factor of 2.75.

Amazon S3 compatibility ❌
Amazon S3 API, but also compatibility in terms of performance, durability, functionality (such as native streaming), security, and privacy.

Security Model ❌

 * Access Grants
 * Zero trust and zero knowledge Architecture ❌
 * Zero knowledge: Nodes don't see or impact the data in any way
 * Zero trust: Anyone trying to access data is authenticated every single time they make a request

Programming language and license ❌
The Storj DCS code is Open Source and written in Golang.

Common misconceptions about Storj DCS ❌
Storj DCS does not operate a blockchain.

The Storj token is only used to transfer payments to node operators.

No data gets stored in any blockchain in any way.

There is no data replication, all stored pieces are unique. Redundancy is achieved through erasure coding.

Use Cases
Backups, research data sets, video streaming , large file transfer , multimedia storage , persistent storage

Adoption
Slow for enterprises therefore aiming at developers

Network statistics
(At the time of writing Storj DCS consists of 12.767 active nodes, a total network capacity of 15,61 Petabytes whereas 6,14 Petabytes are available. 401 million objects are being stored resulting in 31 billion pieces (shards). ) Redundant to the table and will be removed if table remains.

Aggregated real time data about the Storj DCS network is available on the Storj homepage. On 08/24/2021 Storj launched the Storj DCS Public Network Statistics web page where various statistics about its Storj DCS service gets periodically updated. Independent web pages use this data to publish statistical data about Storj DCS.

Customers
Storj DCS is currently particularly aimed at developers to have apps and services built on top of it. There are all kind of customers which includes: The Internet Archive, CIMMYT, Boonji Project, Pocket Network, PixelExperience, Filebase , tribesocial

Integrations ❌
Storj offer various integrations: Filezilla, Fastly, Duplicati

Regulatory constraints ❌
Regulations maybe not ready for web3 style distributed and encrypted zero knowledge storage.

Decentralized storage providers with own technology
Other decentralized cloud storage providers are:


 * Arweave
 * CrowdStorage
 * Filecoin
 * Maidsafe
 * Sia / Skynet

Add IPFS too?

Decentralized providers based on Storj V2 technology

 * Internxt

Centralized storage providers
Other centralized cloud storage providers are:


 * Alibaba Cloud
 * Amazon S3
 * Backblaze
 * DigitalOcean Spaces
 * Google Cloud
 * IBM Cloud Object Storage
 * Microsoft Azure
 * Open Telekom Cloud
 * OVHCloud
 * Rackspace Cloud
 * Wasabi

See also ❌

 * Amazon S3
 * Cloud computing
 * Distributed storage
 * Object Storage
 * Web3

External links ❌

 * Official website


 * Sintel on Storj DCS: https://link.us1.storjshare.io/jvvgukpaalhupsibaqgj5iophx7a/sintel%2FSintel.2010.1080p.mkv


 * Storj DCS: How it works: https://link.ap1.storjshare.io/s/jxqy2wdbf5qeaxo4a2eq6yw3gthq/videos/Storj%20Final.mp4


 * Big Buck Bunny 4K on Storj DCS: https://link.eu1.storjshare.io/jxev3kllqqvkhckzs7uoztcldu2q/bucks5%2FBig_Buck_Bunny_4K.webm


 * Tears of Steel on Storj DCS: https://link.eu1.storjshare.io/jvq5xxvqs6333dhnq4hl5rkhwaoa/tearsofsteel%2Ftearsofsteel_4k.mov


 * Tears of Steel Digital Cinema Package (DCP) (zipped): https://link.eu1.storjshare.io/juxcnobnh743wxiif3j2ksdckrya/tearsofsteel%2Ftearsofsteel_4k_DCP.zip

Category:Distributed data storage Category:Web3 Category:Decentralization Category:Cloud Storage