Internals

Client interface

class wildland.session.Session(sig)

A low-level interface for saving Wildland objects.

dump_object(obj, path=None)

Create a signed manifest out of a Bridge/Container/Storage object; if the object was created from a signed manifest and did not change, returns that object, otherwise creates and signs the manifest.

Return type

bytes

dump_user(user, path=None)

Create a signed manifest out of a User object.

Return type

bytes

class wildland.client.Client(base_dir=None, sig=None, config=None, load=True, **config_kwargs)

A high-level interface for operating on Wildland objects.

add_storage_to_container(container, storage, inline=True, storage_name=None)

Add storage to container, save any changes. If the given storage exists in the container (as determined by backend_id), it gets updated (if possible). If not, it is added. If the passed Storage exists in a container but is referenced by an url, it can only be saved for file URLS, for other URLs a WildlandError will be raised. :type container: Container :param container: Container to add to :type storage: Storage :param storage: Storage to be added :type inline: bool :param inline: add as inline or standalone storage (ignored if storage exists) :type storage_name: Optional[str] :param storage_name: optional name to save storage under if inline == False

static all_storages(container, *, predicate=None)

Return (and load on returning) all storages for a given container.

In case of proxy storage, this will also load a reference storage and inline the manifest.

Return type

Iterator[Storage]

all_subcontainers(container)

List subcontainers of this container.

This takes only the first backend that is capable of sub-containers functionality. :type container: Container :param container: :rtype: Iterator[Union[Container, Bridge]] :return:

auto_import_users

save (import) users encountered while traversing WL paths

cache_storage(container)

Return cache storage for the given container.

Return type

Optional[Storage]

clear_cache()

Clear cache: users, bridges and reference storages.

connect_sync_daemon()

Connect to the sync daemon. Starts the daemon if not running.

do_sync(container_name, job_id, source, target, one_shot, unidir, active_events=None)

Start sync between source and target storages

Return type

str

ensure_mount_reference_container(containers, callback_iter_func=<built-in function iter>)

Ensure that for any storage with MOUNT_REFERENCE_CONTAINER corresponding reference_container appears in sequence before the referencer.

Return type

Tuple[List[Container], str]

find_local_manifest(object_type, name, allow_local_paths=True)

Find local manifest based on a (potentially ambiguous) name. Names can be aliases, user fingerprints (for users), name of the file, part of the file name, or complete file path. If allow_local_paths is False, no filename/local file path is allowed (needed for transitional period for WL Cargo refactor) allow_local_paths should be removed with https://gitlab.com/wildland/wildland-client/-/issues/718

Return type

Optional[Path]

find_storage_usage(storage_id)

Find containers which can use storage given by path or backend-id.

Parameters

storage_id (Union[Path, str]) – storage path or backend_id

Return type

List[Tuple[Container, Union[Path, str]]]

Returns

list of tuples (container, storage_url_or_dict)

find_user_manifest_within_catalog(user)

Mounts containers of the given user’s manifests-catalog and attempts to find that user’s manifest file within that catalog. The user manifest file is expected to be named ‘forest-owner.user.yaml’ and be placed in the root directory of a storage. :type user: User :param user: User :rtype: Optional[Tuple[Storage, PurePosixPath]] :return: tuple of Storage where the user manifest was found and PurePosixPath path pointing at that manifest in the storage

get_all_storages(container, excluded_storage=None, only_writable=False)

List of all storages (including cache storages) for the provided container.

If excluded_storage is provided, it will filter out the corresponding Storage object from the resulting list.

If only_writable is provided as True, it will filter out storages which are read only.

Parameters
  • container (Container) – Container object

  • excluded_storage (Optional[str]) – Storage backend_id

  • only_writable (bool) – Bool

Returns

List of Storage objects

get_bridge_paths_for_user(user, owner=None)

Get bridge paths to the user using bridges owned by owner, including multiple hops. If owner is not specified, use default user (@default).

Parameters
  • user (Union[User, str]) – user to look paths for (user id is accepted too)

  • owner (Optional[User]) – owner of bridges to consider

Return type

Iterable[Iterable[PurePosixPath]]

Returns

list of paths collected from bridges - each result is a list of bridge paths, to be concatenated

get_local_bridges(reload=False)

List of local bridges (loaded from the appropriate directory).

Loads local bridges and caches. :@param reload: if False, load only new manifests and use the cache for the rest; if True, reload cached manifests and load new ones but DO NOT reload broken manifests, as we do not expect these manifests to fix themselves. In short, if the manifest fails to load, do not try to reload it. To reload the broken manifest, we need to use clear_cache() first.

get_local_storage(container, local_storage=None, excluded_storage=None, only_writable=False)

Get first local Storage found for the provided container.

If local_storage is provided as backend_id, it will return the corresponding Storage object if it exists.

If excluded_storage is provided, it will filter out the corresponding Storage object from the result.

Parameters
  • container (Container) – Container object

  • local_storage (Optional[str]) – Storage backend_id

  • excluded_storage (Optional[str]) – Storage backend_id

  • only_writable (bool) – Bool

Returns

Storage object

get_local_storages(container, excluded_storage=None, only_writable=False)

List of Storage object representing all local storages (as defined by self.is_local_storage) for the provided container.

If excluded_storage is provided, it will filter out the corresponding Storage object from the resulting list.

Parameters
  • container (Container) – Container object

  • excluded_storage (Optional[str]) – Storage backend_id

  • only_writable (bool) – Bool

Returns

List of storages objects

get_local_users(reload=False)

List of local users (loaded from the appropriate directory).

Loads local users and caches. :@param reload: if False, load only new manifests and use the cache for the rest; if True, reload cached manifests and load new ones but DO NOT reload broken manifests, as we do not expect these manifests to fix themselves. In short, if the manifest fails to load, do not try to reload it. To reload the broken manifest, we need to use clear_cache() first.

get_remote_storage(container, remote_storage=None, excluded_storage=None, only_writable=False)

Get first remote Storage found for the provided container.

If remote_storage is provided as backend_id, it will return the corresponding Storage object if it exists.

If excluded_storage is provided, it will filter out the corresponding Storage object from the result.

Parameters
  • container (Container) – Container object

  • remote_storage (Optional[str]) – Storage backend_id

  • excluded_storage (Optional[str]) – Storage backend_id

  • only_writable (bool) – Bool

Returns

Storage object

get_remote_storages(container, excluded_storage=None, only_writable=False)

List of Storage object representing all remote storages (as defined as being not self.is_local_storage) for the provided container.

If excluded_storage is provided, it will filter out the corresponding Storage object from the resulting list.

Parameters
  • container (Container) – Container object

  • excluded_storage (Optional[str]) – Storage backend_id

  • only_writable (bool) – Bool

Returns

List of storages objects

get_storages_to_mount(container)

Return valid, mountable storages for the given container

Return type

List[Storage]

get_sync_event()

Wait for sync event and return it.

Return type

Iterator[SyncEvent]

get_sync_job_state(job_id)

Return state of a sync job or None if the job doesn’t exist. Returned tuple contains SyncState and a human-readable description.

Return type

Optional[Tuple[SyncState, str]]

static is_local_storage(storage)

Check if the given storage is local. Currently checks for TYPE matching local, local-cached or local-dir-cached.

static is_local_url(url)

Check if the provided url is a local url.

Return type

bool

static is_url(s)

Check if string can be recognized as URL.

is_url_file_path(url)

Determines if the given URL is an URL to file or to a Wildland bridge. :type url: str :param url: str

load_all(object_type, decrypt=True, base_dir=None, quiet=False)

Load object manifests from the appropriate directory.

load_containers_from(name, aliases=None, bridge_placeholders=True, include_manifests_catalog=False)

Load a list of containers. Currently supports WL paths, glob patterns (*) and tilde (~), but only in case of local files.

Parameters
  • name (Union[str, WildlandPath]) – containers to load - can be a local path (including glob) or a Wildland path

  • aliases (Optional[dict]) – aliases to use when resolving a Wildland path

  • bridge_placeholders (bool) – include bridges as placeholder containers

Return type

Iterator[Container]

Load a Link object from a dictionary

Return type

Link

load_local_storage_cache()

Load local cache storages from manifests to memory for fast access.

load_object_from_bytes(object_type, data, allow_only_primary_key=None, file_path=None, trusted_owner=None, local_owners=None, decrypt=True, expected_owner=None)

Load and return a Wildland object from raw bytes. :type data: bytes :param data: object bytes :type object_type: Optional[Type] :param object_type: expected object type; if not provided, will be guessed based on ‘object’ field. If provided, must match data, or a WildlandError will be raised. :type allow_only_primary_key: Optional[bool] :param allow_only_primary_key: can the object be signed by any of its owner’s keys, or just by the primary key. If omitted, assumed to be True for USER objects and False for all other. :type file_path: Optional[Path] :param file_path: path to local manifest file, if exists :type trusted_owner: Optional[str] :param trusted_owner: accept signature-less manifests from this owner :type local_owners: Optional[List[str]] :param local_owners: owners allowed to access local storages :type decrypt: bool :param decrypt: should we attempt to decrypt the bytes :type expected_owner: Optional[str] :param expected_owner: if object received has a different owner than that, raise an ManifestError

load_object_from_dict(object_type, dictionary, owner, expected_owner=None, container=None)

Load Wildland object from a dict. :type dictionary: dict :param dictionary: dict containing object data :type object_type: Optional[Type] :param object_type: expected type of object; if None, will use dict ‘object’ field. On mismatch of expected and actual type, a WildlandError will be raised. :type owner: str :param owner: owner in whose context we load the dict :type expected_owner: Optional[str] :param expected_owner: expected owner. On mismatch of expected and actual owner, a WildlandError will be raised. :type container: Optional[Container] :param container: used if object is STORAGE. Ignored otherwise.

load_object_from_file_path(object_type, path, decrypt=True)

Load and return a Wildland object from local file path (not in URL form). :type path: Path :param path: local file path :type object_type: Type :param object_type: expected type of object :type decrypt: bool :param decrypt: should we attempt to decrypt the object (default: True)

load_object_from_name(object_type, name, allow_local_paths=True)

Load a Wildland object from ambiguous name. The name can be a local filename, part of local filename (will attempt to look for the object in appropriate local directory), a WL URL, another kind of URL etc. :type object_type: Type :param object_type: expected object type :type name: str :param name: ambiguous name :type allow_local_paths: bool :param allow_local_paths: should accessing local file paths / filenames be allowed; needed before cargo refactor is finished, should be removed with https://gitlab.com/wildland/wildland-client/-/issues/718

load_object_from_url(object_type, url, owner, expected_owner=None)

Load and return a Wildland object from any URL, including Wildland URLs. :type url: str :param url: URL. must start with protocol (e.g. wildland: or https: :type object_type: Optional[Type] :param object_type: expected object type. If not provided, will try to guess it based on data (although this will not be successful for WL URLs to containers.). If provided will raise an exception if expected type is different than received type. :type owner: str :param owner: owner in whose context we should resolve the URL :type expected_owner: Optional[str] :param expected_owner: expected owner. Will raise a WildlandError if receives a different owner.

load_object_from_url_or_dict(object_type, obj, owner, expected_owner=None, container=None)

A convenience wrapper for loading objects from either URL or dict. Returns a Wildland object. :type obj: Union[str, dict] :param obj: URL or dict to be turned into WL object :type object_type: Optional[Type] :param object_type: expected object type :type owner: str :param owner: owner in whose context we should resolve URLs :type expected_owner: Optional[str] :param expected_owner: expected owner :type container: Optional[Container] :param container: used if loading a STORAGE object of dict type

load_pubkeys_from_field(input_list, default_owner=None)

Load and set pubkeys from provided access list. Default owner is provided in order to load user access provided as WLPath.

load_subcontainer_object(container, storage, subcontainer_obj)

Transform a Link or ContainerStub into a real Container or Bridge. Fill remaining fields of the subcontainer and possibly apply transformations.

Return type

Union[Container, Bridge]

load_users_with_bridge_paths(only_default_user=False)

Helper method to return users with paths from bridges leading to those users.

Return type

Iterable[Tuple[User, Optional[List[PurePosixPath]]]]

local_url(path)

Convert an absolute path to a local URL.

Return type

str

new_path(manifest_type, name, skip_numeric_suffix=False, base_dir=None)

Create a path in Wildland base_dir to save a new object of type manifest_type and name name. It follows Wildland conventions. :type manifest_type: Type :param manifest_type: ‘user’, ‘container’, ‘storage’, ‘bridge’ or ‘set’ :type name: str :param name: name of the object :type skip_numeric_suffix: bool :param skip_numeric_suffix: should the path be extended with .1 etc. numeric suffix if first inferred path already exists :type base_dir: Optional[Path] :param base_dir: override base directory if present :rtype: Path :return: Path

parse_file_url(url, owner)

Retrieve path from a given file URL, if it’s applicable. Checks the ‘local_hostname’ and ‘local_owners’ settings.

Return type

Optional[Path]

read_bridge_from_url(url, use_aliases=False)

Return an iterator over all bridges encountered on a given Wildland path.

Return type

Iterable[Bridge]

read_from_url(url, owner=None, use_aliases=False)

Retrieve data from a given URL. The local (file://) URLs are recognized based on the ‘local_hostname’ and ‘local_owners’ settings.

Return type

bytes

recognize_users_and_bridges(users=None, bridges=None)

Load users and recognize their keys from the users directory or a given iterable. This function loads also all (local) bridges, so it’s possible to find paths for the users.

Recognize users and bridges encountered while traversing WL path. If self.auto_import_users is set, bridge and user manifests are saved into ~/.config/wildland as is - without changing bridge owner or its paths (contrary to wl user import command).

Parameters

final_step – final step returned by Search.resolve_raw()

Returns

run_sync_command(name, **kwargs)

Run sync command (through the sync daemon).

Return type

Any

save_new_object(object_type, object_, name=None, path=None, enforce_original_bytes=False)

Save a new object in appropriate directory. Use the name as a hint for file name.

save_object(object_type, obj, path=None, storage_driver=None, enforce_original_bytes=False)

Save an existing Wildland object and return the path it was saved to. :param obj: Object to be saved :type object_type: Type :param object_type: type of object to be saved :type path: Optional[Path] :param path: (optional), path to save the object to; if omitted, object’s local_path will be used. :type storage_driver: Optional[StorageDriver] :param storage_driver: if the object should be written to a given StorageDriver :type enforce_original_bytes: bool :param enforce_original_bytes: should the object be written as-it-was, or should any changes, updated fields etc. be used.

Return type

Path

select_reference_storage(container_url_or_dict, owner, trusted)

Select a “reference” storage and default container path based on URL or dictionary. This resolves a container specification and then selects storage for the container.

Return type

Optional[Tuple[PurePosixPath, Dict]]

select_storage(container, *, predicate=None)

Select and load a storage to mount for a container.

In case of proxy storage, this will also load an reference storage and inline the manifest.

Return type

Storage

start_sync_daemon()

Start the sync daemon.

stop_sync(job_id)

Stop a sync job by job ID.

Return type

str

sub_client_with_key(pubkey)

Create a copy of the current Client, with a public key imported. Returns a tuple (client, owner).

Return type

Tuple[Client, str]

wait_for_sync(job_id, stop_on_finish=True)

Wait for a sync job to complete (state: SYNCED). Returns response messages and a bool meaning if the operation succeeded.

Return type

Tuple[str, bool]

Storage backends

class wildland.storage_backends.base.StorageBackend(*, params=None, read_only=False, **_kwds)

Abstract storage implementation.

Any implementation should inherit from this class.

This abstract class exposes the following constants.

TYPE:           str    (required) - a unique name for backend's type (eg. dropbox, s3,
                                    local)
SCHEMA:         Schema (optional) - a jsonschema compatible Schema object used to validate
                                    storage manifest syntax and accepted values
LOCATION_PARAM: str    (optional) - the key in storage params which holds a path or uri that
                                    points to a location in storage backend.

                                    Some backends (eg. dateproxy) don't specify any
                                    locations as they are merely proxying actual backends.
                                    In those cases this constant should be omitted.

                                    Examples:
                                    - `location` for `local` storage as it points to a
                                      directory in `local` storage
                                    - `s3_url` for `s3` storage as it's the s3's location
                                      identifier

Currently the storage should implement an interface similar to FUSE. This implementation detail might change in the future.

Although FUSE allows returning an error value (like -errno.ENOENT), the storage should always raise an exception if the operation fails, like so:

raise FileNotFoundError(str(path))

Or:

raise OSError(errno.ENOENT, str(path))

See also Python documentation for OS exceptions: https://docs.python.org/3/library/exceptions.html#os-exceptions

add_child(client, wl_object)

Add a Wildland Object to this storage.

If given object is already a child of this storage, subcontainer info will be updated.

property can_have_children: bool

Check if storage can have subcontainers.

If False get_children have to return empty collection or raise error. If True get_children can return an empty or non-empty collection.

Return type

bool

chmod(path, mode)

Set file/directory access mode. Optional.

Return type

None

chown(path, uid, gid)

Set file/directory owner. Optional.

Return type

None

clear_cache()

Clear cache, if any.

Return type

None

classmethod cli_create(data)

Convert provided command-line arguments to a list of storage parameters. If using mixins, check if a super() call is needed.

Return type

Dict[str, Any]

classmethod cli_options()

Provide a list of command-line options needed to create this storage. If using mixins, check if a super() call is needed.

Return type

List[Option]

create(path, flags, mode=438)

Create and open a file. If the file does not exist, first create it with the specified mode, and then open it. Flags are expressed in POSIX style (see os module flag constants). They might be ignored by backend implementation, depending on backend specifics.

static from_params(params, read_only=False, deduplicate=False)

Construct a StorageBackend from fields originating from manifest.

Assume the fields have been validated before.

Parameters

deduplicate – return cached object instance when called with the same params

Return type

StorageBackend

fsync(_path, isfsyncfile, obj)

Flush dirty information about the file to disk. If isfsyncfile is True, only data needs to be flushed (without metadata).

Return type

None

static generate_hash(params)

Returns hash for the given params. May be used eg. for caching.

Return type

str

get_children(client=None, query_path=PurePosixPath('*'))

List all subcontainers provided by this storage.

This method should provide an Iterable of tuples: - PurePosixPath to object (needed for search) - Link or ContainerStub of the object

Storages of listed containers, when set as ‘delegate’ backend, may reference parent (this) container via Wildland URL: wildland:@default:@parent-container:

Return type

Iterable[Tuple[PurePosixPath, Union[Link, ContainerStub]]]

get_hash(path)

Return (and, if get_file_token is implemented, cache) sha256 hash for object at path.

Return type

Optional[str]

get_subcontainer_watch_pattern(query_path)

This function must return pattern usable by mount-watch and by search’s resolve mechanism.

getattr(path)

Get file attributes.

Return type

Attr

has_child(wl_object_uuid_path)

Check if the given Wildland Object is a subcontainer of this storage.

Return type

bool

static is_type_supported(storage_type)

Check if the storage type is supported.

property location

Returns location of the storage backend or None if it is not defined.

mkdir(path, mode=511)

Create a directory with the given name. The directory permissions are encoded in mode.

Return type

None

mount()

Initialize. Called when mounting.

Backend should implement this method if necessary. External callers should use request_mount instead.

Return type

None

abstract open(path, flags)

Open a file with given path. Access flags should be used to check if the operation is permitted. Flags are expressed in POSIX style (see os module flag constants). Caution: flags such as read-only may be ignored by backends, this is not checked anywhere in base storage backend. For example of testing if flags are respected, see test_backend.test_read_only_flags.

Return type

File

open_for_safe_replace(path, flags, original_hash)

This should implement a version of compare-and-swap: open a file, write data as needed, but apply those changes (on release) only if file hash before the changes matched original_hash. Should throw HashMismatchError if hashes are mismatched (both at open and at release).

Return type

File

property persistent_db: wildland.storage_backends.kv_store.KVStore

Returns instance of a persistent key-value DB for use by the backend.

Return type

KVStore

read(_path, length, offset, obj)

Read data from an open file.

Read should return exactly the number of bytes requested except on EOF or error, otherwise the rest of the data will be substituted with zeroes. An exception to this is when the direct_io mount option is specified, in which case the return value of the read system call will reflect the return value of this operation.

This method is proxied to wildland.storage_backends.base.File.read().

Return type

bytes

readdir(path)

Return iterable of files’ names living in the given directory.

Return type

Iterable[str]

remove_child(client, wl_object)

Remove a Wildland Object from this storage.

If given object is not a child of that storage, nothing happens.

rename(move_from, move_to)

Move file/directory. Optional.

Return type

None

request_mount()

Request storage to be mounted, if not mounted already.

Return type

None

request_unmount()

Request storage to be unmounted, if not used anymore.

Return type

None

retrieve_hash(path)

Get cached hash, if possible; priority is given to local dict, then to permanent storage.

Return type

Optional[HashCache]

rmdir(path)

Remove a directory.

Return type

None

set_config_dir(config_dir)

Set path to config dir. Used to store hashes in a local sqlite DB.

start_bulk_writing()

Indicates that we want to do many writing/creation operation. Writing may be cached (or just invisible in local cache) so from now on reading what we are writing/updating is undefined (files may or may not be updated). Reading untouched files should works as usual.

Return type

None

stop_bulk_writing()

Indicates that we end up bulk writing. This method should accomplish cached writing operations (like the flush method). The cache should be (possibly) invalidated if applicable.

Return type

None

store_hash(path, hash_cache)

Store provided hash in persistent (if available) storage and in local dict.

Return type

None

property supports_publish: bool

Check if storage handles subcontainers.

Return type

bool

to_str(include_sensitive=False)

Return string representation

truncate(path, length)

Truncate or extend the given file so that it is precisely length bytes long.

Return type

None

static types()

Lazily initialized type -> storage class mapping.

Return type

Dict[str, Type[StorageBackend]]

Remove a file.

Return type

None

unmount()

Clean up. Called when unmounting.

Backend should implement this method if necessary. External callers should use request_unmount instead.

Return type

None

utimens(path, atime, mtime)

https://github.com/libfuse/python-fuse/blob/6c3990f9e3dce927c693e66dc14138822b42564b/fuse.py#L474

Parameters
  • atime – Timespec access time

  • mtime – Timespec modification time

Return type

None

static validate_manifest(manifest)

Validate manifest, assuming it’s of a supported type.

Return type

None

walk(directory=PurePosixPath('.'))

A simplified variant of os.walk : returns a generator of paths of all objects in the given directory (or the whole storage if none given). Assumes the results will be given depth-first.

Return type

Iterable[Tuple[PurePosixPath, Attr]]

watcher()

Create a StorageWatcher (see watch.py) for this storage, if supported. If the storage manifest contains a watcher-interval parameter, SimpleStorageWatcher (which is a naive, brute-force watcher that scans the entire storage every watcher-interval seconds) will be used. If a given StorageBackend provides a better solution, it’s recommended to overwrite this method to provide it. It is recommended to still use SimpleStorageWatcher if the user explicitly specifies watcher-interval in the manifest. See local.py for a simple super() implementation that avoids duplicating code.

Note that changes originating from FUSE are reported without using this mechanism.

write(_path, data, offset, obj)

Write data to an open file.

Write should return exactly the number of bytes requested except on error. An exception to this is when the direct_io mount option is specified.

This method is proxied to wildland.storage_backends.base.File.write().

Return type

int

class wildland.storage_backends.kv_store.KVStore(base_dir, backend_id)

Persistent key-value store backed by SQLite.

del_object(key)

Delete stored object.

get_all_keys()

Get all keys for given backend ID.

Return type

Set[str]

get_object(key)

Retrieve object by key.

Return type

Any

store_object(key, value)

Store object by key.

class wildland.storage_backends.local_cached.BaseCached(**kwds)

A cached storage backed by local files. Used mostly to test the caching scheme.

This backend should emulate “cloud” backends, therefore, we don’t keep open file handles, but perform read()/write() operations opening the file each time.

chmod(path, mode)

Set file/directory access mode. Optional.

chown(path, uid, gid)

Set file/directory owner. Optional.

create(path, flags, mode=438)

Create and open a file. If the file does not exist, first create it with the specified mode, and then open it. Flags are expressed in POSIX style (see os module flag constants). They might be ignored by backend implementation, depending on backend specifics.

mkdir(path, mode=511)

Create a directory with the given name. The directory permissions are encoded in mode.

mount()

Initialize. Called when mounting.

Backend should implement this method if necessary. External callers should use request_mount instead.

Return type

None

open(path, flags)

Open a file with given path. Access flags should be used to check if the operation is permitted. Flags are expressed in POSIX style (see os module flag constants). Caution: flags such as read-only may be ignored by backends, this is not checked anywhere in base storage backend. For example of testing if flags are respected, see test_backend.test_read_only_flags.

Return type

File

rename(move_from, move_to)

Move file/directory. Optional.

rmdir(path)

Remove a directory.

truncate(path, length)

Truncate or extend the given file so that it is precisely length bytes long.

Return type

None

Remove a file.

utimens(path, atime, mtime)

https://github.com/libfuse/python-fuse/blob/6c3990f9e3dce927c693e66dc14138822b42564b/fuse.py#L474

Parameters
  • atime – Timespec access time

  • mtime – Timespec modification time

watcher()

If manifest explicitly specifies a watcher-delay, use default implementation. If not, we can use the smarter LocalStorageWatcher.

class wildland.storage_backends.timeline.TimelineStorageBackend(**kwds)

A proxy storage that re-organizes the files into directories based on their modification date.

All files will have a ‘year/month/day’ prefix prepended to their path. Directory timestamps will be ignored, and empty directories will not be taken into account.

The ‘reference-container’ parameter specifies inner container, either as URL, or as an inline manifest. When creating the object instance:

1. First, the storage parameters for the inner container will be resolved (see Client.select_storage()),

2. Then, the inner storage backend will be instantiated and passed as params[‘storage’] (see StorageBackend.from_params()).

property can_have_children: bool

Check if storage can have subcontainers.

If False get_children have to return empty collection or raise error. If True get_children can return an empty or non-empty collection.

Return type

bool

clear_cache()

Invalidate cache.

get_children(client=None, query_path=PurePosixPath('*'))

List all subcontainers provided by this storage.

This method should provide an Iterable of tuples: - PurePosixPath to object (needed for search) - Link or ContainerStub of the object

Storages of listed containers, when set as ‘delegate’ backend, may reference parent (this) container via Wildland URL: wildland:@default:@parent-container:

Return type

Iterable[Tuple[PurePosixPath, ContainerStub]]

info_all()

Retrieve information about all files in the storage.

Return type

Iterable[Tuple[PurePosixPath, Attr]]

mount()

Initialize. Called when mounting.

Backend should implement this method if necessary. External callers should use request_mount instead.

open(path, flags)

Open a file with given path. Access flags should be used to check if the operation is permitted. Flags are expressed in POSIX style (see os module flag constants). Caution: flags such as read-only may be ignored by backends, this is not checked anywhere in base storage backend. For example of testing if flags are respected, see test_backend.test_read_only_flags.

Return type

File

unmount()

Clean up. Called when unmounting.

Backend should implement this method if necessary. External callers should use request_unmount instead.

class wildland.storage_backends.delegate.DelegateProxyStorageBackend(**kwds)

A proxy storage that exposes a subdirectory of another container.

The ‘reference-container’ parameter specifies reference container, either as URL, or as an inline manifest. When creating the object instance:

  1. First, the storage parameters for the reference container will be resolved (see: select_storage()),

  2. Then, the reference storage backend will be instantiated and passed as params[‘storage’] (see: from_params()).

chmod(path, mode)

Set file/directory access mode. Optional.

Return type

None

chown(path, uid, gid)

Set file/directory owner. Optional.

Return type

None

clear_cache()

Clear cache, if any.

create(path, flags, mode=438)

Create and open a file. If the file does not exist, first create it with the specified mode, and then open it. Flags are expressed in POSIX style (see os module flag constants). They might be ignored by backend implementation, depending on backend specifics.

get_hash(path)

Return (and, if get_file_token is implemented, cache) sha256 hash for object at path.

getattr(path)

Get file attributes.

Return type

Attr

mkdir(path, mode=511)

Create a directory with the given name. The directory permissions are encoded in mode.

Return type

None

mount()

Initialize. Called when mounting.

Backend should implement this method if necessary. External callers should use request_mount instead.

open(path, flags)

Open a file with given path. Access flags should be used to check if the operation is permitted. Flags are expressed in POSIX style (see os module flag constants). Caution: flags such as read-only may be ignored by backends, this is not checked anywhere in base storage backend. For example of testing if flags are respected, see test_backend.test_read_only_flags.

Return type

File

open_for_safe_replace(path, flags, original_hash)

This should implement a version of compare-and-swap: open a file, write data as needed, but apply those changes (on release) only if file hash before the changes matched original_hash. Should throw HashMismatchError if hashes are mismatched (both at open and at release).

Return type

File

readdir(path)

Return iterable of files’ names living in the given directory.

Return type

Iterable[str]

rename(move_from, move_to)

Move file/directory. Optional.

retrieve_hash(path)

Get cached hash, if possible; priority is given to local dict, then to permanent storage.

rmdir(path)

Remove a directory.

Return type

None

store_hash(path, hash_cache)

Store provided hash in persistent (if available) storage and in local dict.

truncate(path, length)

Truncate or extend the given file so that it is precisely length bytes long.

Return type

None

Remove a file.

Return type

None

unmount()

Clean up. Called when unmounting.

Backend should implement this method if necessary. External callers should use request_unmount instead.

utimens(path, atime, mtime)

https://github.com/libfuse/python-fuse/blob/6c3990f9e3dce927c693e66dc14138822b42564b/fuse.py#L474

Parameters
  • atime – Timespec access time

  • mtime – Timespec modification time

Return type

None

walk(directory=PurePosixPath('.'))

A simplified variant of os.walk : returns a generator of paths of all objects in the given directory (or the whole storage if none given). Assumes the results will be given depth-first.

Return type

Iterable[Tuple[PurePosixPath, Attr]]

class plugins.encrypted.wildland_encrypted.backend.EncryptedStorageBackend(**kwds)

The ‘reference-container’ parameter specifies inner container, either as URL, or as an inline manifest. When creating the object instance:

1. First, the storage parameters for the inner container will be resolved (see Client.select_storage()),

2. Then, the inner storage backend will be instantiated and passed as params[‘storage’] (see StorageBackend.from_params()).

chmod(path, mode)

Set file/directory access mode. Optional.

Return type

None

chown(path, uid, gid)

Set file/directory owner. Optional.

Return type

None

create(path, flags, mode=438)

Create and open a file. If the file does not exist, first create it with the specified mode, and then open it. Flags are expressed in POSIX style (see os module flag constants). They might be ignored by backend implementation, depending on backend specifics.

Return type

File

getattr(path)

Get file attributes.

mkdir(path, mode=511)

Create a directory with the given name. The directory permissions are encoded in mode.

Return type

None

mount()

Initialize. Called when mounting.

Backend should implement this method if necessary. External callers should use request_mount instead.

Return type

None

open(path, flags)

Open a file with given path. Access flags should be used to check if the operation is permitted. Flags are expressed in POSIX style (see os module flag constants). Caution: flags such as read-only may be ignored by backends, this is not checked anywhere in base storage backend. For example of testing if flags are respected, see test_backend.test_read_only_flags.

Return type

File

readdir(path)

Return iterable of files’ names living in the given directory.

rename(move_from, move_to)

Move file/directory. Optional.

rmdir(path)

Remove a directory.

Return type

None

truncate(path, length)

Truncate or extend the given file so that it is precisely length bytes long.

Return type

None

Remove a file.

unmount()

Clean up. Called when unmounting.

Backend should implement this method if necessary. External callers should use request_unmount instead.

Return type

None

utimens(path, atime, mtime)

https://github.com/libfuse/python-fuse/blob/6c3990f9e3dce927c693e66dc14138822b42564b/fuse.py#L474

Parameters
  • atime – Timespec access time

  • mtime – Timespec modification time

Return type

None

class wildland.storage_backends.dummy.DummyStorageBackend(*args, **kwargs)

Dummy storage

getattr(path)

Get file attributes.

open(path, flags)

Open a file with given path. Access flags should be used to check if the operation is permitted. Flags are expressed in POSIX style (see os module flag constants). Caution: flags such as read-only may be ignored by backends, this is not checked anywhere in base storage backend. For example of testing if flags are respected, see test_backend.test_read_only_flags.

readdir(path)

Return iterable of files’ names living in the given directory.

class wildland.storage_backends.static.StaticStorageBackend(*, params=None, **kwds)

Read-only storage backend containing files listed in the storage manifest directly.

get_root()

Get the directory root.

Return type

DirEntry

class wildland.storage_backends.local_cached.LocalCachedStorageBackend(*args, **kwargs)

A cached storage that uses info_all().

info_all()

Load information about all files and directories.

Return type

Iterable[Tuple[PurePosixPath, Attr]]

class wildland.storage_backends.local_cached.LocalDirectoryCachedStorageBackend(*args, **kwargs)

A cached storage that uses info_dir().

info_dir(path)

Load information about a single directory.

Return type

Iterable[Tuple[PurePosixPath, Attr]]

class wildland.storage_backends.local.LocalStorageBackend(*, relative_to=None, **kwds)

Local, file-based storage

chmod(path, mode)

Set file/directory access mode. Optional.

Return type

None

chown(path, uid, gid)

Set file/directory owner. Optional.

Return type

None

create(path, flags, mode=438)

Create and open a file. If the file does not exist, first create it with the specified mode, and then open it. Flags are expressed in POSIX style (see os module flag constants). They might be ignored by backend implementation, depending on backend specifics.

getattr(path)

Get file attributes.

Return type

Attr

mkdir(path, mode=511)

Create a directory with the given name. The directory permissions are encoded in mode.

Return type

None

mount()

Initialize. Called when mounting.

Backend should implement this method if necessary. External callers should use request_mount instead.

Return type

None

open(path, flags)

Open a file with given path. Access flags should be used to check if the operation is permitted. Flags are expressed in POSIX style (see os module flag constants). Caution: flags such as read-only may be ignored by backends, this is not checked anywhere in base storage backend. For example of testing if flags are respected, see test_backend.test_read_only_flags.

readdir(path)

Return iterable of files’ names living in the given directory.

Return type

List[str]

rename(move_from, move_to)

Move file/directory. Optional.

Return type

None

rmdir(path)

Remove a directory.

Return type

None

truncate(path, length)

Truncate or extend the given file so that it is precisely length bytes long.

Return type

None

Remove a file.

Return type

None

utimens(path, atime, mtime)

https://github.com/libfuse/python-fuse/blob/6c3990f9e3dce927c693e66dc14138822b42564b/fuse.py#L474

Parameters
  • atime – Timespec access time

  • mtime – Timespec modification time

Return type

None

watcher()

If manifest explicitly specifies a watcher-interval, use default implementation. If not, we can use the smarter LocalStorageWatcher.

class wildland.storage_backends.http.HttpStorageBackend(**kwds)

A read-only HTTP storage that gets its information from directory listings.

getattr(path)

Get file attributes.

Return type

Attr

info_dir(path)

Retrieve information about files in a directory (readdir + getattr).

Return type

Iterable[Tuple[PurePosixPath, Attr]]

make_url(path, is_dir=False)

Convert a Path to resource URL.

Return type

str

open(path, _flags)

Open a file with given path. Access flags should be used to check if the operation is permitted. Flags are expressed in POSIX style (see os module flag constants). Caution: flags such as read-only may be ignored by backends, this is not checked anywhere in base storage backend. For example of testing if flags are respected, see test_backend.test_read_only_flags.

Return type

PagedHttpFile

Syncing

class wildland.hashdb.HashCache(hash: str, token: str)

Hash cache

hash: str

Alias for field number 0

token: str

Alias for field number 1

class wildland.storage_backends.watch.FileEvent(type, path)

File change event.

class wildland.storage_sync.base.SyncEvent

Base class for sync events.

static fromJSON(s)

Deserialize from JSON.

Return type

SyncEvent

toJSON()

Serialize to JSON.

Return type

str

class wildland.storage_sync.base.SyncState(value)

Current state of a syncer.

File/dir entries

class wildland.storage_backends.generated.CachedDirEntry(name, get_entries_func, timestamp=0, timeout_seconds=3)

A version of FuncDirEntry that cached its results.

clear_cache()

Invalidate cache.

get_entries()

List all entries in this directory.

Return type

Iterable[Entry]

get_entry(name)

Get a specific entry by name, or raise KeyError.

By default, uses get_entries(), but you can override it for a more efficient implementation.

Return type

Entry

class wildland.storage_backends.generated.DirEntry(name)

Directory entry. Can be listed.

abstract get_entries()

List all entries in this directory.

Return type

Iterable[Entry]

get_entry(name)

Get a specific entry by name, or raise KeyError.

By default, uses get_entries(), but you can override it for a more efficient implementation.

Return type

Entry

class wildland.storage_backends.generated.Entry(name)

File or directory entry.

abstract getattr()

File attributes.

Return type

Attr

class wildland.storage_backends.generated.FileEntry(name)

File entry. Can be open, producing a File.

abstract open(flags)

Open a file.

Return type

File

class wildland.storage_backends.generated.FuncDirEntry(name, get_entries_func, timestamp=0)

Shortcut for creating function-based directories.

>>> def get_entries():
...     yield FuncFileEntry('foo.txt', lambda: b'foo')
...     yield FuncFileEntry('bar.txt', lambda: b'bar')
...
>>> d = FuncDirEntry('dir', get_entries)
get_entries()

List all entries in this directory.

Return type

Iterable[Entry]

getattr()

File attributes.

Return type

Attr

class wildland.storage_backends.generated.FuncFileEntry(name, on_read=None, on_write=None, timestamp=0)

Shortcut for creating function-based files.

>>> def read_foo():
...     return 'hello from foo'
...
>>> def write_bar(data: bytes):
...     logger.info('bar: %s', data)
...
>>> f1 = FileEntry('foo', on_read=read_foo)
>>> f2 = FileEntry('bar', on_write=write_bar)
getattr()

File attributes.

Return type

Attr

open(flags)

Open a file.

Return type

File

Files

class wildland.storage_backends.base.File

Abstract base class for a file. To be returned from open() and create().

Methods are optional to implement, except File.release().

read(length=None, offset=0)

Read data from an open file. This method is a proxy for wildland.storage_backends.base.StorageBackend.read().

Return type

bytes

abstract release(flags)

Releases file-related resources. Should accept flag constants from os module (e.g. os.O_RDWR), but they might be ignored (depending on backend specifics).

Return type

None

write(data, offset)

Write data to an open file. This method is a proxy for wildland.storage_backends.base.StorageBackend.write().

Return type

int

class wildland.storage_backends.generated.CommandFile(on_write, attr)

A write-only file that triggers a callback.

release(flags)

Releases file-related resources. Should accept flag constants from os module (e.g. os.O_RDWR), but they might be ignored (depending on backend specifics).

write(data, _offset)

Write data to an open file. This method is a proxy for wildland.storage_backends.base.StorageBackend.write().

class wildland.storage_backends.buffered.FullBufferedFile(attr, clear_cache_callback=None)

A file class that buffers reads and writes. Stores the full file content in memory.

Requires you to implement read_full() and write_full().

read(length=None, offset=0)

Read data from an open file. This method is a proxy for wildland.storage_backends.base.StorageBackend.read().

Return type

bytes

abstract read_full()

Read the full file.

Return type

bytes

release(_flags)

Save pending changes on release. If overriding, make sure to call super().release() first.

write(data, offset)

Write data to an open file. This method is a proxy for wildland.storage_backends.base.StorageBackend.write().

Return type

int

abstract write_full(data)

Replace the current file content.

Return type

int

class wildland.storage_backends.buffered.PagedFile(attr)

A read-only file class that stores parts of file in memory. Assumes that you are able to read a range of bytes from a file.

TODO: This currently performs only 1 concurrent read. A better implementation would allow parallel reads, but would need to ensure that a given part is read only once.

read(length=None, offset=0)

Read data from an open file. This method is a proxy for wildland.storage_backends.base.StorageBackend.read().

Return type

bytes

abstract read_range(length, start)

Read a range from a file. This is essentially read(), but renamed here so that read() proper can use the buffer.

Return type

bytes

release(flags)

Releases file-related resources. Should accept flag constants from os module (e.g. os.O_RDWR), but they might be ignored (depending on backend specifics).

class wildland.storage_backends.generated.StaticFile(data, attr)

A read-only file with pre-determined content.

read(length=None, offset=0)

Read data from an open file. This method is a proxy for wildland.storage_backends.base.StorageBackend.read().

Return type

bytes

release(flags)

Releases file-related resources. Should accept flag constants from os module (e.g. os.O_RDWR), but they might be ignored (depending on backend specifics).

class wildland.storage_backends.generated.StaticFileEntry(name, data, timestamp=0)

Shortcut for creating static files.

getattr()

File attributes.

Return type

Attr

open(flags)

Open a file.

Return type

File

class wildland.storage_backends.http.PagedHttpFile(session, url, attr)

A read-only fully buffered HTTP file.

read_full()

Read the full file.

Return type

bytes

write_full(data)

Replace the current file content.

Return type

int

File attributes

class wildland.storage_backends.base.Attr(mode, size=0, timestamp=0)

File attributes. A subset of statinfo.

classmethod dir(size=0, timestamp=0)

Simple directory with default access mode.

Return type

Attr

classmethod file(size=0, timestamp=0)

Simple file with default access mode.

Return type

Attr

is_dir()

Convenience method to check if this is a directory.

Return type

bool

class plugins.dropbox.wildland_dropbox.backend.DropboxFileAttr(mode, size=0, timestamp=0, content_hash=None)

Attributes of a Dropbox file/directory.

classmethod from_file_metadata(metadata)

Convert given file’s metadata to attribute object.

Return type

DropboxFileAttr

classmethod from_folder_metadata(_metadata)

Convert given file’s metadata to attribute object.

Return type

DropboxFileAttr

Mixins

class wildland.storage_backends.cached.CachedStorageMixin(*args, **kwargs)

A mixin for caching file information.

You need to implement info_all(), and invalidate cache (by calling clear_cache() or update_cache()) in all operations that might change the result.

clear_cache()

Invalidate cache.

Return type

None

getattr(path)

Cached implementation of getattr().

Return type

Attr

info_all()

Retrieve information about all files in the storage.

Return type

Iterable[Tuple[PurePosixPath, Attr]]

readdir(path)

Cached implementation of readdir().

Return type

Iterable[str]

refresh()

Refresh cache.

Return type

None

update_cache(path, attr)

Update item in the cache instead of invalidating all of the items.

Return type

None

class wildland.storage_backends.cached.DirectoryCachedStorageMixin(*args, **kwargs)

A mixin for caching file information about a specific directory.

You need to implement info_dir(), and invalidate the cache (by calling clear_cache() or update_cache()) in all operations that might change the result.

clear_cache()

Invalidate the cache.

Return type

None

getattr(path)

Cached implementation of getattr().

Return type

Attr

info_dir(path)

Retrieve information about files in a directory (readdir + getattr).

Return type

Iterable[Tuple[PurePosixPath, Attr]]

invalidate_dir_cache(path)

Invalidates directory cache

Return type

None

readdir(path)

Cached implementation of readdir().

Return type

Iterable[str]

update_cache(path, attr)

Update item in the cache instead of invalidating all of the items.

Return type

None

class wildland.storage_backends.generated.GeneratedStorageMixin

A mixin for auto-generated storage.

A simple usage is to implement callbacks:

>>> from wildland.storage_backends.base import StorageBackend
>>> from functools import partial
>>>
>>> class MyStorage(GeneratedStorageMixin, StorageBackend):
...     def get_root(self):
...         return FuncDirEntry('.', self._root)
...
...     def _root(self):
...         for i in range(10):
...             yield FuncDirEntry(f'dir-{i}', partial(self._dir, i))
...
...     def _dir(self, i):
...         yield FuncFileEntry('readme.txt', partial(self._readme, i))
...         yield FuncFileEntry(f'{i}.txt', partial(self._file, i))
...
...     def _readme(self, i):
...         return f'This is readme.txt for {i}'
...
...     def _file(self, i):
...         return f'This is {i}.txt'
...
get_root()

Get the directory root.

Return type

DirEntry

getattr(path)

getattr() for generated storage

Return type

Attr

open(path, flags)

open() for generated storage

Return type

File

readdir(path)

readdir() for generated storage

Return type

Iterable[str]

class wildland.storage_backends.file_children.FileChildrenMixin(*, params=None, read_only=False, **_kwds)

A backend storage mixin providing support for pattern-manifest types of glob and list.

glob type is a UNIX-style expression for listing all manifest files that fit a given pattern. list type is an array that holds a list of relative paths to child objects manifests within the storage itself. The paths are relative to the storage’s root (i.e. path for Local storage backend).

When adding this mixin, you should append the following snippet to the backend’s SCHEMA:: “manifest-pattern”: { “oneOf”: [ {“$ref”: “/schemas/types.json#pattern-glob”}, {“$ref”: “/schemas/types.json#pattern-list”} ], }

Furthermore, classes using this mixin should remember to use super() call in cli_options and cli_create (see LocalStorageBackend as an example).

add_child(client, wl_object)

Add Wildland object manifest to this storage.

If the given object is already a child of this storage, update it.

property can_have_children: bool

Check if storage can have subcontainers.

If False get_children have to return empty collection or raise error. If True get_children can return an empty or non-empty collection.

Return type

bool

get_children(client=None, query_path=PurePosixPath('*'))

List all child objects provided by this storage.

Return type

Iterable[Tuple[PurePosixPath, Link]]

get_subcontainer_watch_pattern(query_path)

This function must return pattern usable by mount-watch and by search’s resolve mechanism.

Return type

PurePosixPath

has_child(wl_object_uuid_path)

Check if the given container is child of this storage.

Return type

bool

remove_child(client, wl_object)

Remove Wildland object manifest from this storage.

If the given object is already a child of this storage, nothing happens.

property supports_publish: bool

Check if storage handles child manifests.

At the moment only simple file-based backends with manifest-pattern: glob are supported.

Return type

bool

Data transfer objects

class wildland.user.User(owner, pubkeys, paths, manifests_catalog, client, manifest=None, members=None)

A data transfer object representing Wildland user. Can be converted from/to a self-signed user manifest.

add_catalog_entry(path)

Add a path to a container or a dict with container/link fields to user’s manifests catalog.

add_user_keys(sig_context, add_primary=True)

Add user keys (primary key only if add_primary is True and any keys listed in “pubkeys” field) to the given sig_context.

get_catalog_descriptions()

Provide a human-readable descriptions of user’s manifests catalog without loading them.

get_primary_publish_path()

Return primary path where the manifest is going to be published, by convention it’s a path that starts with /.uuid/

Return type

PurePosixPath

get_publish_paths()

Return all paths where the manifest is going to be published (that shall always return primary publish path, among others)

Return type

List[PurePosixPath]

get_unique_publish_id()

Return unique id for publishable object. In most cases it’s going to be an uuid of the manifest while User object, for example, may return user’s signature.

Return type

str

property has_catalog: bool

Checks if user has a manifest catalog defined

Return type

bool

has_catalog_entry(path)

Returns true if a catalog entry is present in user’s manifest

load_catalog(warn_about_encrypted_manifests=True)

Load and cache all of user’s manifests catalog.

property primary_pubkey

Primary pubkey for signatures. User manifest needs to be signed with this key

remove_catalog_entry(path)

Remove a catalog entry (provided asn url string or a dict). Returns True if operation was successful.

to_manifest_fields(inline, str_repr_only=False)

Return a dict with fields ready to be put inside a manifest (inline or standalone). The dict can be later modified, so be careful and deepcopy when needed. str_repr_only = True means the fields will only be used for logging purposes, never use it for other purposes

When implementing, take care to verify that returned fields are valid (through Schema.validate or other appropriate methods).

Return type

dict

to_repr_fields(include_sensitive=False)

This function provides filtered sensitive and unneeded fields for representation

Return type

dict

to_str(include_sensitive=False)

Return string representation

class wildland.container.Container(owner, paths, backends, client, title=None, categories=None, manifest=None, access=None)

Wildland container

add_storage_from_obj(storage, inline=True, storage_name=None, new_url=None)

Add a given Storage object to own backends, replace if exists. While replacing, current inline state will override the inline parameter. :param storage: Storage object :type inline: bool :param inline: should the storage be added as inline storage or standalone one :type storage_name: Optional[str] :param storage_name: if inline=False and the storage does not already exist, this name will be used to save it :type new_url: Optional[str] :param new_url: if inline=False and the storage is already within own backends, replaces its url with this url.

clear_storages()

Remove all storages

copy(new_name)

Copy this container to a new object with a new UUID and appropriately edited storages.

Return type

Container

del_storage(backend_id_or_path=None)

Remove a storage of a given backend_id or URL. Removes only first occurrence of a given storage.

property expanded_paths: List[pathlib.PurePosixPath]

Paths expanded by the set of paths generated from title and categories (if provided).

This method MUST NOT change the order of paths so that /.uuid/{container_uuid} path remains first in the list.

Return type

List[PurePosixPath]

fill_storage_fields(storage_dict)

Fill fields of a storage dict with data from this container. Returns the modified dict, or, if the dict was encrypted or not actually a dict, just the object received.

get_backends_description(only_inline=False)

Get a readable description of own storage backends.

get_primary_publish_path()

Return primary path where the manifest is going to be published, by convention it’s a path that starts with /.uuid/

Return type

PurePosixPath

get_publish_paths()

Return all paths where the manifest is going to be published (that shall always return primary publish path, among others)

Return type

List[PurePosixPath]

get_unique_publish_id()

Return unique id for publishable object. In most cases it’s going to be an uuid of the manifest while User object, for example, may return user’s signature.

Return type

str

is_backend_in_use(path_or_id)

Check if a storage of a given id or url is used by this container.

is_manifests_catalog

whether this container is a manifests catalog, loaded by iterating this manifests catalog itself; this property is set externally by the Search class, when loading a container by iterating a manifest catalog

load_raw_backends(include_inline=True, include_url=True)

Load and return raw backends (copied, so that there will not be surprise side effects). :type include_inline: bool :param include_inline: should inline storages be included? :type include_url: bool :param include_url: should url storages be included?

load_storages(include_inline=True, include_url=True)

Load and return container storages. Returns Storage objects. :type include_inline: bool :param include_inline: should inline storages be included? :type include_url: bool :param include_url: should url storages be included?

property sync_id: str

Return unique sync job ID for a container.

Return type

str

to_manifest_fields(inline, str_repr_only=False)

Return a dict with fields ready to be put inside a manifest (inline or standalone). The dict can be later modified, so be careful and deepcopy when needed. str_repr_only = True means the fields will only be used for logging purposes, never use it for other purposes

When implementing, take care to verify that returned fields are valid (through Schema.validate or other appropriate methods).

Return type

dict

to_repr_fields(include_sensitive=False)

This function provides filtered sensitive and unneeded fields for representation

Return type

dict

to_str(include_sensitive=False)

Return string representation

property uuid: str

Container UUID

Return type

str

property uuid_path: pathlib.PurePosixPath

The main UUID path for this container

Return type

PurePosixPath

validate()

Validate container. This is not done automatically because we might want to load a container having wrong fields.

class wildland.container.ContainerStub(fields)

Helper Wildland Object, representing a subcontainer that has not yet been completely filled with data from parent container.

classmethod from_manifest(manifest)

Create ContainerStub from verified manifest.

get_container(parent_container)

Fill container fields with data from parent_container and construct a complete Container.

Return type

Container

class wildland.storage.Storage(owner, storage_type, trusted, params, client, container=None, manifest=None, access=None)

A data transfer object representing Wildland storage.

property backend_id

Returns backend_id param.

copy(old_uuid, new_uuid)

Copy this storage to a new object, replacing its container uuid from old_uuid to new_uuid

get_mount_path(container)

Return unique mount path for this storage. The path is rooted in the container’s owner forest root.

Return type

PurePosixPath

get_primary_publish_path()

Return primary path where the manifest is going to be published, by convention it’s a path that starts with /.uuid/

Return type

PurePosixPath

get_publish_paths()

Return all paths where the manifest is going to be published (that shall always return primary publish path, among others)

Return type

List[PurePosixPath]

get_unique_publish_id()

Return unique id for publishable object. In most cases it’s going to be an uuid of the manifest while User object, for example, may return user’s signature.

Return type

str

property is_primary: bool

Returns primary param.

Return type

bool

property is_writeable: bool

Returns False if read-only param was set to True.

Return type

bool

promote_to_primary()

Sets primary param to True.

to_manifest_fields(inline, str_repr_only=False)

Return a dict with fields ready to be put inside a manifest (inline or standalone). The dict can be later modified, so be careful and deepcopy when needed. str_repr_only = True means the fields will only be used for logging purposes, never use it for other purposes

When implementing, take care to verify that returned fields are valid (through Schema.validate or other appropriate methods).

Return type

dict

to_repr_fields(include_sensitive=False)

This function provides filtered sensitive and unneeded fields for representation

Return type

dict

to_str(include_sensitive=False)

Return string representation

validate()

Validate storage assuming it’s of a known type. This is not done automatically because we might want to load an unrecognized storage.

class wildland.bridge.Bridge(owner, user_location, user_pubkey, user_id, paths, client, manifest=None)

Bridge object: a wrapper for user manifests.

static create_safe_bridge_paths(user_id, paths)

Creates safe (ie. obscure) bridge path which will not conflict with other potentially existing, user-defined paths which could cause mailicious forest to be mounted without user’s awareness.

Return type

List[PurePosixPath]

get_primary_publish_path()

Return primary path where the manifest is going to be published, by convention it’s a path that starts with /.uuid/

Return type

PurePosixPath

get_publish_paths()

Return all paths where the manifest is going to be published (that shall always return primary publish path, among others)

Return type

List[PurePosixPath]

get_unique_publish_id()

Return unique id for publishable object. In most cases it’s going to be an uuid of the manifest while User object, for example, may return user’s signature.

Return type

str

to_manifest_fields(inline, str_repr_only=False)

Return a dict with fields ready to be put inside a manifest (inline or standalone). The dict can be later modified, so be careful and deepcopy when needed. str_repr_only = True means the fields will only be used for logging purposes, never use it for other purposes

When implementing, take care to verify that returned fields are valid (through Schema.validate or other appropriate methods).

Return type

dict

to_placeholder_container()

Create a placeholder container that shows how to mount the target user’s forest.

Return type

Container

to_repr_fields(include_sensitive=False)

This function provides filtered sensitive and unneeded fields for representation

Return type

dict

to_str(include_sensitive=False)

Return string representation

Wildland Link helper object

get_target_file()

Returns a (potentially) cached bytes of the target file object.

Return type

bytes

to_manifest_fields(inline, str_repr_only=False)

Return a dict with fields ready to be put inside a manifest (inline or standalone). The dict can be later modified, so be careful and deepcopy when needed. str_repr_only = True means the fields will only be used for logging purposes, never use it for other purposes

When implementing, take care to verify that returned fields are valid (through Schema.validate or other appropriate methods).

to_repr_fields(include_sensitive=False)

This function provides filtered sensitive and unneeded fields for representation

Return type

dict

to_str(include_sensitive=False)

Return string representation

Wildland path

class wildland.wlpath.WildlandPath(owner, hint, parts, file_path)

A path in Wildland namespace.

The path has the following form:

[wildland:][owner][@hint]:(part:)+:[file_path]

  • owner (optional): owner determining the first container’s namespace, if omitted @default will be used

  • hint (optional, requires owner): hint to the location of first container’s namespace; takes the form of protocol{address} where address is a percent-encoded URL. For example ‘https{demo.wildland.io/demo.user.yaml}’ or with an alternative port ‘https{wildland.lan%3A8081}’

  • parts: intermediate parts, identifying bridges or containers on the path

  • file_path (optional): path to file in the last container

append(s)

Append string to WildlandPath.

Accepts paths both with and without ‘:’ into.

classmethod from_str(s)

Construct a WildlandPath from a string.

Accepts paths both with and without ‘wildland:’ protocol prefix.

Return type

WildlandPath

classmethod get_canonical_form(s)

Return string being canonical form representation of a Wildland path

Return type

str

has_explicit_or_default_owner()

Check if WildlandPath has explicit owner or default, i.e., not alias.

Return type

bool

classmethod match(s)

Check if a string should be recognized as a Wildland path.

To be used when distinguishing Wildland paths from other identifiers (local paths, URLs).

Note that this doesn’t guarantee that the WildlandPath.from_str() will succeed in parsing the path.

Return type

bool

to_str(with_prefix=False)

Return string representation

class wildland.search.Search(client, wlpath, aliases=mappingproxy({}), fs_client=None)

A class for traversing a Wildland path.

Usage:

search = Search(client, wlpath, client.config.aliases)
search.read_file()
classmethod clear_cache()

Clear path resolution cache.

Calling this method is necessary, if it’s necessary to re-download user’s manifests catalog content during lifetime of the same process. This may be the case for example after (re)publishing some new container.

get_watch_params()

Prepare parameters required to watch given WL path for changes (including any container involved in path resolution).

This function returns a tuple of:
  • list of mount parameters (for WildlandFSClient.mount_multiple_containers())

  • set of patterns (relative to the FUSE mount point) to watch

Watching the patterns is legal only if all returned mount commands succeeded.

Usage: >>> client = wildland.client.Client() >>> fs_client = client.fs_client >>> search = Search(…) >>> mount_cmds, patterns = search.get_watch_params() >>> try: >>> # mounting under unique paths only is enough, no need to pollute user’s forest >>> fs_client.mount_multiple_containers(mount_cmds, unique_path_only=True) >>> for events in fs_client.watch(patterns): >>> … >>> except: >>> …

Return type

Tuple[List, Set[PurePosixPath]]

read_bridge()

Yield all bridges matching the given WL path.

Return type

Iterable[Bridge]

read_container()

Yield all containers matching the given WL path.

Return type

Iterable[Container]

read_file()

Read a file under the Wildland path.

Return type

bytes

resolve_raw()

Resolve the non-file part of the wildland path and yield raw resolution results.

Return type

Iterable[Step]

Returns

result of path resolution, may be multiple entries

write_file(data, create_parents=False)

Read a file under the Wildland path.

class wildland.search.Step(owner, client, container, bridge, user, previous, pattern=None)

A single step of a resolved path.

steps_chain()

Iterate over all steps leading to this resolved path

Manifests

class wildland.manifest.manifest.Manifest(header, fields, original_data, local_path=None)

Represents a loaded manifest.

The data (fields) should not be modified, because it needs to match the return signature.

apply_schema(schema)

Validate the manifest using a provided schema.

copy_to_unsigned()

Make an unsigned deep copy of this manifest. Useful for skipping signing, validation and similar potentially destructive changes.

Return type

Manifest

classmethod decrypt(fields, sig)

Decrypt provided dict within provided SigContext. Assumes encrypted (sub) dict contains an ‘encrypted’ fields that contains a dict with two fields: ‘encrypted-data’ and ‘encrypted-keys’. Returns decrypted dict.

Return type

dict

classmethod encrypt(fields, sig, owner=None)

Encrypt provided dict with the SigContext. Encrypts to ‘owner’ keys, unless ‘access’ field specifies otherwise. Inline storages may have their own access fields. Returns encrypted dict.

Return type

dict

encrypt_and_sign(sig_context, only_use_primary_key=False, encrypt=True)

Sign a manifest. If signed, will replace signature. If attach_pubkey is true, attach the public key to the signature. Can force not encrypting, if needed.

property fields

A wrapper for manifest fields that makes sure the manifest is signed.

classmethod from_bytes(data, sig_context, schema=None, trusted_owner=None, allow_only_primary_key=False, decrypt=True, local_path=None)

Load a manifest from YAML content, verifying it.

Parameters
  • data (bytes) – existing manifest content

  • sig_context (SigContext) – a SigContext to use for signature verification

  • schema (Optional[Schema]) – a Schema to validate the fields with

  • trusted_owner (Optional[str]) – accept signature-less manifest from this owner

  • allow_only_primary_key (bool) – can this manifest be signed by any auxiliary keys associated with the given user?

  • decrypt (bool) – should we attempt to decrypt the manifest

Return type

Manifest

classmethod from_fields(fields, sig=None, local_path=None)

Create a manifest based on a dict of fields.

Has to be signed separately, but is assumed to be verified for the purpose of accessing data.

Return type

Manifest

classmethod from_file(path, sig_context, schema=None, trusted_owner=None)

Load a manifest from YAML file, verifying it.

Parameters
  • path – path to YAML file

  • sig_context (SigContext) – a SigContext to use for signature verification

  • schema (Optional[Schema]) – a Schema to validate the fields with

  • trusted_owner (Optional[str]) – accept signature-less manifest from this owner

Return type

Manifest

classmethod from_unsigned_bytes(data, sig=None, local_path=None)

Create a new Manifest based on existing YAML-serialized content. The content can include an existing header, it will be ignored.

Has to be signed separately.

Return type

Manifest

property original_data

Return manifest’s bytes data.

skip_verification()

Explicitly mark the manifest as unsigned, and allow using it.

to_bytes()

Serialize the manifest, including the signature.

classmethod update_manifest_version(fields)

Update manifest version, if an old version is encountered. Currently able to update from un-versioned to version 1.

Return type

dict

classmethod update_obsolete(fields, is_inline=False)

Update fields obsolete in V1.

classmethod update_to_version_1(fields, add_version=True)
Update any obsolete fields, bringing manifest up to v1. Currently handles:
  • signer –> owner

  • inner-container –> reference-container

  • in local storages, path –> location

Return type

dict

classmethod verify_and_load_pubkeys(data, sig_context)

Load pubkeys directly from manifest’s message (body) into signature context without relying on locally stored users/keys. This might be used in a self-signed manifests context.

Return type

None

class wildland.manifest.manifest.Header(signature)

Manifest header (owner and signature).

classmethod from_bytes(data)

Parse the header.

to_bytes()

Serialize the header.

verify_bytes(data)

Internal consistency check: verify that the serialized manifest parses the same.

verify_rest(rest_data, sig_context, trusted_owner, pubkey=None)

Verify the signature against manifest content (without parsing it). Return signer.

pass the optional pubkey only in case of self-signed manifests

Return type

str

class wildland.manifest.schema.Schema(arg)
classmethod load_dict(name, root_key)

Load a dictionary of schemas from a file.

Return type

Dict[str, Schema]

class wildland.wildland_object.wildland_object.WildlandObject(*args, **kwargs)

Abstract class representing Wildland Objects. To implement it, implement parse_fields() and to_manifest_fields() at minimum. If implementing from_manifest(), make sure fields are correctly validated.

WildlandObject should own all objects contained within; if passing dicts (e.g. as manifests-catalog or backends), deepcopy them.

class Type(value)

Possible Wildland object types.

classmethod from_fields(fields, client, object_type=None, manifest=None, **kwargs)

Initialize a WildlandObject from a dict of fields. Will verify if the dict fits a schema, if object_type supports a schema. This method should NOT be reimplemented by inheriting classes, in order to make sure verification was correctly done. :type fields: dict :param fields: dict of fields :param client: Wildland Client object, used to load any needed supplementary objects :type object_type: Optional[Type] :param object_type: WildlandObject.Type; if loads an unexpected type, will raise an error :type manifest: Optional[Manifest] :param manifest: if the object is constructed from manifest, pass this manifest here :param kwargs: any additional keyword arguments, may be used by WL object inits. :return: an instanced WildlandObject

classmethod from_manifest(manifest, client, object_type=None, **kwargs)

Create a Wildland Object from a provided Manifest :type manifest: Manifest :param manifest: source Manifest; must be verified or trusted :param client: Wildland Client object, used to load any needed supplementary objects :type object_type: Optional[Type] :param object_type: WildlandObject.Type; if loads an unexpected type, will raise an error :param kwargs: any additional keyword arguments, may be used by WL object inits. :return: an instanced WildlandObject

property local_path

Local file path of the object.

abstract classmethod parse_fields(fields, client, manifest=None, **kwargs)

Initialize a WildlandObject from a dict of fields. :type fields: dict :param fields: dict of fields :param client: Wildland Client object, used to load any needed supplementary objects :type manifest: Optional[Manifest] :param manifest: if the object is constructed from manifest, pass this manifest here :param kwargs: any additional keyword arguments, may be used by WL object inits. :return: an instanced WildlandObject

abstract to_manifest_fields(inline, str_repr_only=False)

Return a dict with fields ready to be put inside a manifest (inline or standalone). The dict can be later modified, so be careful and deepcopy when needed. str_repr_only = True means the fields will only be used for logging purposes, never use it for other purposes

When implementing, take care to verify that returned fields are valid (through Schema.validate or other appropriate methods).

Return type

dict

property type: wildland.wildland_object.wildland_object.WildlandObject.Type

Returns Wildland Object Type

Return type

Type

class wildland.manifest.template.StorageTemplate(source_data)

Single storage template object for a storage manifest. See TemplateFile class for details on defining a list of such objects.

get_storage(client, cont, local_dir)

Return Storage object corresponding to this storage template.

Return type

Storage

get_storage_fields(cont, local_dir=None)

Fill template fields with container data and return them.

class wildland.manifest.template.TemplateFile(path)

Defines physical file that holds list of StorageTemplate in yaml format using jinja2 template language. The template file should provide a yaml file with array of objects where each object contains fields required for a specific type of storage backend, especially backend type. Owner and container-path fields will be ignored.

The following container fields can be used: uuid, categories, title, paths, local_path, owner. Sample local storage template file:

- type: local
location: /home/user/{{ uuid }}
- type: dropbox
location: /subdir_in_dropbox{{ local_dir if local_dir is defined else "" }}/{{ uuid }}
read-only: true
token: dropbox_secret_token
class wildland.manifest.template.TemplateManager(template_dir)

Helper class to manage TemplateFile and StorageTemplate.

available_templates()

Returns a list of all TemplateFile in template_dir; TemplateFile must be correct yaml/jinja files with TEMPLATE_SUFFIX.

Return type

List[TemplateFile]

create_storage_template(template_name, params)

Create storage template from given, arbitrary params and append to a given template file. If template file doesn’t exist, create it and then append the template.

Return type

Path

get_file_path(name)

Return path to TemplateFile based on given name.

Parameters

name (str) – Can be either an absolute path to a file or a template name inside template_dir

Return type

Path

static get_jinja_yaml(jinja_template)

Mocks all jinja2 expressions so that content can be correctly interpreted as yaml syntax. Returns content as parsed yaml file.

get_storage_template(name)

Get StorageTemplate; can be specified as filename without suffix, complete filename or StorageTemplate’s internal name. If there is more than one StorageTemplate with a given name, returns first one found.

Return type

StorageTemplate

get_template_bytes(input_template)

Return contents of a storage template’s .jinja file.

Parameters

input_template (str) – either path to template’s file or its name

Return type

bytes

get_template_file_by_name(template_name)

Return TemplateFile object for a given template name.

Return type

TemplateFile

static is_valid_template_file_path(path)

Return true if given path has a valid TemplateFile name.

Return type

bool

remove_storage_template(name)

Remove storage template by name.

Return type

None

save_template_content(input_template, content)

Saves content of a storage template’s .jinja file.

Parameters
  • input_template (str) – either path to template’s file or its name

  • content (str) – content to be saved inside template’s file

Signing

class wildland.manifest.sig.SigContext(key_dir)

A class for signing and verifying signatures. Operates on ‘owner’ identifiers, serving as key fingerprints.

add_pubkey(pubkey, owner=None)

Add a public key to recognized keys. Optionally, can associate key with an owner (by default, a key is owned by itself). pubkey should be the actual public key, not its hash. Returns a key signer ID.

Return type

str

copy()

Create and return a copy of the current context.

decrypt(data, encrypted_keys)

Attempt to decrypt data using all available keys.

Return type

bytes

encrypt(data, keys)

Encrypt data to be readable by keys. Returns a tuple: encrypted message, list of encrypted keys.

Return type

Tuple[str, List[str]]

static fingerprint(pubkey)

Turns a public key into owner/key id. :rtype: str :return: key_id

generate()

Generate a new key pair and store it.

Returns a pair of (owner, pubkey).

Return type

Tuple[str, str]

get_all_pubkeys(owner)

Get all pubkeys owned by a given owner.

get_possible_owners(signer)

List key_ids that can be owners of the provided key_id.

Return type

List[str]

get_primary_pubkey(owner)

Get owner’s primary/own pubkey.

Return type

str

is_private_key_available(key_id)
Parameters

key_id (str) – key id for the key to check

Return type

bool

Returns

if private key for the given key is available

is_valid_pubkey(key)

Verify that given string is a valid public key

Return type

bool

load_key(key_id)

Find a canonical form for the key. If the key is to be used in verification and signing, it must be added to SigContext with add_pubkey. By default, looks for <name>.pub file in the specified key directory. Returns a pair of (owner, pubkey).

Return type

Tuple[str, str]

recognize_local_keys()

Recognize local keys (i.e. ones found using find()) while loading.

remove_key(key_id)

Remove a given key (and a matching private key) from disk and from local context.

remove_owner(owner)

Remove a given key from owners’ lists of others keys (in local context). Key is not deleted.

sign(owner, data, only_use_primary_key=False)

Sign data using a given owner’s key. If not only_use_primary_key, will use the first available key, if multiple are available. Returns signature.

Return type

str

verify(signature, data, pubkey=None)

Verify signature for data, returning the recognized owner. If self_signed, pass pubkey to verify the message integrity.

Return type

str

FUSE driver

class wildland.fs.WildlandFS(*args, **kwds)

A FUSE implementation of Wildland

init_logging(args)

Configure logging module.

install_debug_handler()

Decorate all python-fuse entry points

main(args=None)

Enter filesystem service loop.

Storage Driver

class wildland.storage_driver.StorageDriver(storage_backend, storage=None, bulk_writing=False)

A contraption to directly manipulate wildland.storage_backends.base.StorageBackend

file_exists(relpath)

Check if file exists.

Return type

bool

classmethod from_storage(storage, bulk_writing=False)

Create StorageDriver from wildland.storage.Storage

@param storage: wildland.storage.Storage @param bulk_writing: if True, writing can be cached so reading what we are writing/updating is undefined (files can be updated or not). Reading untouched files works as usual.

Return type

StorageDriver

makedirs(relpath, mode=493)

Make directory, and it’s parents if needed. Does not work across containers.

read_file(relpath)

Read a file from StorageBackend, using FUSE commands.

Return type

bytes

remove_empty_dir(relpath)

Remove a directory if it is empty. Returns False if dir was not empty so not deleted, True otherwise

Return type

bool

remove_file(relpath)

Remove a file.

to_str(include_sensitive=False)

Return string representation

write_file(relpath, data)

Write a file to StorageBackend, using FUSE commands. Returns (StorageBackend, relpath).