README
¶
go-ds-crdt
A distributed go-datastore implementation using Merkle-CRDTs.
go-ds-crdt is a key-value store implementation using Merkle CRDTs, as
described in
the paper by Héctor Sanjuán, Samuli Pöyhtäri and Pedro Teixeira.
It satisfies the
Datastore,
ThreadSafeDatastore
and Batching
interfaces from go-datastore.
This implementation Work In Progress and highly experimental at the moment.
Internally it uses a delta-CRDT Add-Wins Observed-Removed set. The current value for a key is the one with highest priority. Priorities are defined as the height of the Merkle-CRDT node in which the key was introduced.
Usage
go-ds-crdt needs:
- A user-provided, thread-safe,
go-datastoreimplementation to be used as permanent storage. - A user-defined
Broadcastercomponent to broadcast and receive updates from a set of replicas. - A user-defined
DAGSyncercomponent to publish and retrieve Merkle DAGs
The permanent storage layout is optimized for KV stores with fast indexes and key-prefix support.
See https://godoc.org/github.com/ipfs/go-ds-crdt for more information.
Captain
This project is captained by @hsanjuan.
License
This library is dual-licensed under Apache 2.0 and MIT terms.
Copyright 2019. Protocol Labs, Inc.
Documentation
¶
Overview ¶
Package crdt provides a replicated go-datastore implementation using Merkle-CRDTs built with IPLD nodes. This Datastore is agnostic to how new MerkleDAG roots are broadcasted to the rest of replicas (Broadcaster component) and to how the IPLD nodes are made discoverable and retrievable to by other replicas (DAG-Syncer component).
Index ¶
- Variables
- type Broadcaster
- type DAGSyncer
- type Datastore
- func (store *Datastore) Batch() (ds.Batch, error)
- func (store *Datastore) Close() error
- func (store *Datastore) Delete(key ds.Key) error
- func (store *Datastore) Get(key ds.Key) (value []byte, err error)
- func (store *Datastore) GetSize(key ds.Key) (size int, err error)
- func (store *Datastore) Has(key ds.Key) (exists bool, err error)
- func (store *Datastore) PrintDAG() error
- func (store *Datastore) Put(key ds.Key, value []byte) error
- func (store *Datastore) Query(q query.Query) (query.Results, error)
- type Options
- type PubSubBroadcaster
Constants ¶
This section is empty.
Variables ¶
var (
ErrNoMoreBroadcast = errors.New("receiving blocks aborted since no new blocks will be broadcasted")
)
Common errors.
Functions ¶
This section is empty.
Types ¶
type Broadcaster ¶
type Broadcaster interface {
// Send payload to other replicas.
Broadcast([]byte) error
// Obtain the next payload received from the network.
Next() ([]byte, error)
}
A Broadcaster provides a way to send (notify) an opaque payload to all replicas and to retrieve payloads broadcasted.
type DAGSyncer ¶
type DAGSyncer interface {
ipld.DAGService
// Returns true if the block is locally available (therefore, it
// is considered processed).
HasBlock(c cid.Cid) (bool, error)
}
A DAGSyncer is an abstraction to an IPLD-based p2p storage layer. A DAGSyncer is a DAGService with the ability to publish new ipld nodes to the network, and retrieving others from it.
type Datastore ¶
type Datastore struct {
// contains filtered or unexported fields
}
Datastore makes a go-datastore a distributed Key-Value store using Merkle-CRDTs and IPLD.
func New ¶
func New( store ds.Datastore, namespace ds.Key, dagSyncer DAGSyncer, bcast Broadcaster, opts *Options, ) (*Datastore, error)
New returns a Merkle-CRDT-based Datastore using the given one to persist all the necessary data under the given namespace. It needs a DAG-Syncer component for IPLD nodes and a Broadcaster component to distribute and receive information to and from the rest of replicas. Actual implementation of these must be provided by the user.
func (*Datastore) Batch ¶
Batch implements batching for writes by accumulating Put and Delete in the same CRDT-delta and only applying it and broadcasting it on Commit().
func (*Datastore) Get ¶
Get retrieves the object `value` named by `key`. Get will return ErrNotFound if the key is not mapped to a value.
func (*Datastore) GetSize ¶
GetSize returns the size of the `value` named by `key`. In some contexts, it may be much cheaper to only get the size of the value rather than retrieving the value itself.
func (*Datastore) Has ¶
Has returns whether the `key` is mapped to a `value`. In some contexts, it may be much cheaper only to check for existence of a value, rather than retrieving the value itself. (e.g. HTTP HEAD). The default implementation is found in `GetBackedHas`.
func (*Datastore) PrintDAG ¶
PrintDAG pretty prints the current Merkle-DAG using the given printFunc
func (*Datastore) Query ¶
Query searches the datastore and returns a query result. This function may return before the query actually runs. To wait for the query:
result, _ := ds.Query(q)
// use the channel interface; result may come in at different times
for entry := range result.Next() { ... }
// or wait for the query to be completely done
entries, _ := result.Rest()
for entry := range entries { ... }
type Options ¶
type Options struct {
Logger logging.StandardLogger
RebroadcastInterval time.Duration
// The PutHook function is triggered whenever an element
// is successfully added to the datastore (either by a local
// or remote update), and only when that addition is considered the
// prevalent value.
PutHook func(k ds.Key, v []byte)
// The DeleteHook function is triggered whenever a version of an
// element is successfully removed from the datastore (either by a
// local or remote update). Unordered and concurrent updates may
// result in the DeleteHook being triggered even though the element is
// still present in the datastore because it was re-added. If that is
// relevant, use Has() to check if the removed element is still part
// of the datastore.
DeleteHook func(k ds.Key)
}
Options holds configurable values for Datastore.
func DefaultOptions ¶
func DefaultOptions() *Options
DefaultOptions initializes an Options object with sensible defaults.
type PubSubBroadcaster ¶ added in v0.0.3
type PubSubBroadcaster struct {
// contains filtered or unexported fields
}
PubsubBroadcaster implements a Broadcaster using libp2p PubSub.
func NewPubSubBroadcaster ¶ added in v0.0.3
func NewPubSubBroadcaster(ctx context.Context, psub *pubsub.PubSub, topic string) (*PubSubBroadcaster, error)
NewPubsubBroadcaster returns a new broadcaster using the given PubSub and a topic to subscribe/broadcast to. The given context can be used to cancel the broadcaster. Please register any topic validators before creating the Broadcaster.
func (*PubSubBroadcaster) Broadcast ¶ added in v0.0.3
func (pbc *PubSubBroadcaster) Broadcast(data []byte) error
Broadcast publishes some data.
func (*PubSubBroadcaster) Next ¶ added in v0.0.3
func (pbc *PubSubBroadcaster) Next() ([]byte, error)
Next returns published data.
Source Files
Directories