Plugins

Implementing a plugin is a way to extend Wildland capabilities by adding a support for a new type of storage. Wildland exposes an API to register new storage drivers (a.k.a. plugins). At the time of writing there are plugins for Bear, Dropbox, IMAP, IPFS, S3 and WebDAV.

API internals

All of the plugins live in their own installable Python modules, in plugins/ directory. Each plugin is self-contained and needs to implement some of the filesystem operations, manifest schema and command line argument handling for wl storage create command.

To add a new plugin, create a Python package and make sure it has a right entry point in setup.py. A good way to start may be studying existing plugins.

FUSE callbacks

Wildland is built on top of FUSE (Filesystem in Userspace). To put it more precisely, FUSE consists of a kernel module and a userspace libfuse library that abstracts away the low-level interface. Wildland directly utilizes python-fuse which is a Python interface to libfuse.

Storage backend

Every plugin needs to implement StorageBackend which is an abstract base class exposing an interface similar to the one being used by python-fuse. The following is the list of the methods you typically need to implement:

• mount() - Called when mounting a container. Initializes the storage, e.g. establishing a connection with a server.

• unmount() - Called when unmounting a container. Cleans up the resources, e.g. closing a connection with a server.

• open() - Based on the given path, returns a File representing a file being opened. Object that is returned from this method wraps read() and write() operations amongst the others, therefore you typically shouldn’t implement StorageBackend’s read() and write() which just call respective methods from File object.

  Note

  Typically you should not inherit directly from File as there are classes built on it to optimize read/writes by utilizing buffering. See: PagedFile.

• getattr() - Gets attributes of the given file: its size, timestamp and permissions. Returns Attr object or backend-specific one, inheriting from it (e.g. DropboxFileAttr).

• create() - Creates empty file with the given permissions.

• unlink() - Removes (deletes) the given file.

• mkdir() - Creates empty directory with given permissions.

• rmdir() - Removes the given directory. This should succeed only if the directory is empty.

• readdir() - Lists the given directory.

There are many other FUSE callbacks that, depending on the needs, you should or should not implement. For full list, refer to StorageBackend class.

All storage backends can use a persistent key-value object store: KVStore. Any instance of StorageBackend can access that store (which is backed by an SQLite DB) by the property persistent_db. For an example of how this can be used, see the IMAP plugin: ~plugins.imap.wildland_imap.ImapClient.LocalCache. There, the persistent store is used to cache message metadata so constructing the FS structure is faster on subsequent mounts.

The following are examples of the classes inheriting from StorageBackend. You can refer to them to see how they use storage primitives.

• BaseCached - Cached storage backed by local files.

• TimelineStorageBackend - Proxy storage that re-organizes the files into directories based on their modification date.

• DelegateProxyStorageBackend - Proxy storage that exposes a subdirectory of another container.

• DummyStorageBackend - Dummy storage.

• LocalCachedStorageBackend - Cached storage that uses info_all().

• LocalDirectoryCachedStorageBackend - Cached storage that uses info_dir().

• LocalStorageBackend - Local, file-based storage.

Command line and manifest

Besides the above mentioned methods that are all strictly related to handling filesystem operations, you need to also implement:

• cli_options(), cli_create() that are responsible for parsing command line input.

• SCHEMA that defines storage manifest schema.

Storage Mixins

Instead of implementing all of the FUSE callbacks yourself, you can use one of the mixins available. They provide higher abstraction primitives optimized for different scenarios.

The following is the list of all of the available mixins at the time of writing:

• DirectoryCachedStorageMixin - Helps caching file’s attributes and directory listings. It implements both readdir() and getattr() for you by utilizing a cache. You just need to implement info_dir() which is being used by both of those methods. Make sure to call clear_cache() whenever directory content or any of the files’ attributes may change to not allow cache to serve outdated data.

• CachedStorageMixin - Similar to DirectoryCachedStorageMixin but caches whole storage instead of just a single directory. It implements both readdir() and getattr() for you by utilizing a cache. You just need to implement info_dir() which is being used by both of those methods. You should not use this mixin unless you are operating on relatively small tree directory.

• GeneratedStorageMixin - Helps you with creating, auto-generated storage. readdir(), getattr(), open() are implemented for you. You just need to implement get_root() method. This mixin does not support cache (yet).

• FileChildrenMixin - Special type of mixin, providing support for subcontainers and manifests catalog containers specified through flat file lists or glob expressions.

Proxy backends

Sometimes you might want to utilize other storage backend from your own. Examples include following classes, working with inner storage in very different ways.

• DelegateProxyStorageBackend - a simple and clean example, accesses inner storage directly.

• TimelineStorageBackend - manipulates paths to create a timeline view of container contents.

• EncryptedStorageBackend - utilizes access to inner storage directly and via FUSE.

When working with inner backend, consider what could the worst case look like. One example - encrypted backend attempts to write down a configuration file for gocryptfs and does not call flush to make sure that data is written to permanent storage. Since the inner storage is CachedStorageMixin, few moments later gocryptfs attempts to read its configuration and fails. A data race.

Installation

To install your all of the plugins available, run:

python3 -m venv env/
. ./env/bin/activate
pip install -r requirements.dev.txt
pip install -e . plugins/*

To check whether your newly implemented plugin was registered correctly, run:

wl storage list