Plugins¶
Implementing a plugin is a way to extend Wildland capabilities by adding a support for a new type of storage. Wildland exposes an API to register new storage drivers (a.k.a. plugins). At the time of writing there are plugins for Bear, Dropbox, IMAP, IPFS, S3 and WebDAV.
API internals¶
All of the plugins live in their own installable Python modules, in plugins/
directory. Each
plugin is self-contained and needs to implement some of the filesystem operations, manifest schema and command line argument handling for wl storage create command.
To add a new plugin, create a Python package and make sure it has a right entry point in
setup.py
. A good way to start may be studying existing plugins.
FUSE callbacks¶
Wildland is built on top of FUSE (Filesystem in Userspace). To put it more precisely, FUSE
consists of a kernel module and a userspace libfuse
library that abstracts away the low-level
interface. Wildland directly utilizes python-fuse which is a Python interface to libfuse
.
Storage backend¶
Every plugin needs to implement StorageBackend
which is an
abstract base class exposing an interface similar to the one being used by python-fuse
. The
following is the list of the methods you typically need to implement:
mount()
- Called when mounting a container. Initializes the storage, e.g. establishing a connection with a server.unmount()
- Called when unmounting a container. Cleans up the resources, e.g. closing a connection with a server.open()
- Based on the givenpath
, returns aFile
representing a file being opened. Object that is returned from this method wrapsread()
andwrite()
operations amongst the others, therefore you typically shouldn’t implementStorageBackend
’sread()
andwrite()
which just call respective methods fromFile
object.getattr()
- Gets attributes of the given file: its size, timestamp and permissions. ReturnsAttr
object or backend-specific one, inheriting from it (e.g.DropboxFileAttr
).create()
- Creates empty file with the given permissions.unlink()
- Removes (deletes) the given file.mkdir()
- Creates empty directory with given permissions.rmdir()
- Removes the given directory. This should succeed only if the directory is empty.readdir()
- Lists the given directory.
There are many other FUSE callbacks that, depending on the needs, you should or should not
implement. For full list, refer to StorageBackend
class.
All storage backends can use a persistent key-value object store:
KVStore
. Any instance of
StorageBackend
can access that store (which is backed
by an SQLite DB) by the property
persistent_db
. For an example of how this
can be used, see the IMAP plugin: ~plugins.imap.wildland_imap.ImapClient.LocalCache. There,
the persistent store is used to cache message metadata so constructing the FS structure is faster
on subsequent mounts.
The following are examples of the classes inheriting from
StorageBackend
. You can refer to them to see how they use
storage primitives.
BaseCached
- Cached storage backed by local files.TimelineStorageBackend
- Proxy storage that re-organizes the files into directories based on their modification date.DelegateProxyStorageBackend
- Proxy storage that exposes a subdirectory of another container.DummyStorageBackend
- Dummy storage.LocalCachedStorageBackend
- Cached storage that usesinfo_all()
.LocalDirectoryCachedStorageBackend
- Cached storage that usesinfo_dir()
.LocalStorageBackend
- Local, file-based storage.
Command line and manifest¶
Besides the above mentioned methods that are all strictly related to handling filesystem operations, you need to also implement:
cli_options()
,cli_create()
that are responsible for parsing command line input.SCHEMA
that defines storage manifest schema.
Storage Mixins¶
Instead of implementing all of the FUSE callbacks yourself, you can use one of the mixins available. They provide higher abstraction primitives optimized for different scenarios.
The following is the list of all of the available mixins at the time of writing:
DirectoryCachedStorageMixin
- Helps caching file’s attributes and directory listings. It implements bothreaddir()
andgetattr()
for you by utilizing a cache. You just need to implementinfo_dir()
which is being used by both of those methods. Make sure to callclear_cache()
whenever directory content or any of the files’ attributes may change to not allow cache to serve outdated data.CachedStorageMixin
- Similar toDirectoryCachedStorageMixin
but caches whole storage instead of just a single directory. It implements bothreaddir()
andgetattr()
for you by utilizing a cache. You just need to implementinfo_dir()
which is being used by both of those methods. You should not use this mixin unless you are operating on relatively small tree directory.GeneratedStorageMixin
- Helps you with creating, auto-generated storage.readdir()
,getattr()
,open()
are implemented for you. You just need to implementget_root()
method. This mixin does not support cache (yet).FileChildrenMixin
- Special type of mixin, providing support for subcontainers and manifests catalog containers specified through flat file lists or glob expressions.
Proxy backends¶
Sometimes you might want to utilize other storage backend from your own. Examples include following classes, working with inner storage in very different ways.
DelegateProxyStorageBackend
- a simple and clean example, accesses inner storage directly.TimelineStorageBackend
- manipulates paths to create a timeline view of container contents.EncryptedStorageBackend
- utilizes access to inner storage directly and via FUSE.
When working with inner backend, consider what could the worst case look like. One example - encrypted backend attempts to write down a configuration file for gocryptfs and does not call flush to make sure that data is written to permanent storage. Since the inner storage is CachedStorageMixin, few moments later gocryptfs attempts to read its configuration and fails. A data race.
Installation¶
To install your all of the plugins available, run:
python3 -m venv env/
. ./env/bin/activate
pip install -r requirements.dev.txt
pip install -e . plugins/*
To check whether your newly implemented plugin was registered correctly, run:
wl storage list