Subcontainers

A subcontainer is a container or bridge generated by another storage backend. It can represent part of a given container’s data, a complete another container or even a bridge to another user.

Implementation

A storage backend that wants to implement subcontainers should implement a get_children() method that returns an iterable of either ContainerStub or Link to subcontainer objects. If a ContainerStub is returned, it can omit the following fields (they will be filled by Wildland automatically): - owner in the container part - owner in the storage backend part - container-path in the storage backend part

Additionally, get_subcontainer_watch_pattern that provides pattern usable by mount-watch and by search’s resolve mechanism can be provided if support for mount-watch is expected.

Each subcontainer should have an unique but stable UUID path (one in form of /.uuid/<UUID> where <UUID> is an identifier formed in UUID format). It should be the first path.

If the desired content consists of subcontainers only (the content of the parent container ought not to be accessed directly), it is recommended to not add any extra paths to the parent container or use one in a hidden directory (the .uuid path will be added automatically anyway).

This type of subcontainer usually has one storage backend of delegate type, pointing at its parent container and specifying a subdirectory that should be used. Parent container can be conveniently pointed via a Wildland URL wildland:@default:@parent-container:. Note @default in this context always means the owner of the container (not the default user in the Wildland configuration) and @parent-container special path is valid only when used within a subcontainer.

Storage manifests should be included inline.

Example implementation of a get_children() method would look like this:

def get_children(self, query_path: PurePosixPath = PurePosixPath('*')) -> \
        Iterable[Tuple[PurePosixPath, ContainerStub]]:
    ns = uuid.UUID(self.backend_id)
    dates = []
    for year in self.readdir(PurePosixPath('')):
        for month in self.readdir(PurePosixPath(year)):
            for day in self.readdir(PurePosixPath(year + '/' + month)):
                dates.append(f'{year}/{month}/{day}')

    for date in dates:
        yield PurePosixPath('/timeline/' + date), ContainerStub({
            'paths': [
                '/.uuid/{!s}'.format(uuid.uuid3(ns, date)),
                '/timeline/' + date,
            ],
            'backends': {'storage': [{
                'type': 'delegate',
                'reference-container': 'wildland:@default:@parent-container:',
                'subdirectory': '/' + date,
                'backend-id': str(uuid.uuid3(ns, date))
            }]}
        })

File-based subcontainers

Some storage backends (at the moment, those implementing the FileChildrenMixin, such as local storage or s3 storage) can also provide a manifest-pattern parameter to specify the location of files pointing to subcontainer objects (container or bridge manifest .yaml files).

Two types of manifest-pattern are supported by FileChildrenMixin: glob ( a simplified pattern-matching expression) and list (a list of files).

manifest-pattern:
  type: glob
  path: /manifests/{path}/*.{object-type}.yaml

The path is an absolute path that can contain * and {path}. {path} is expanded to the container path we are looking for.

manifest-pattern:
  type: list
  paths:
  - /container.yaml
  - /bridges/bridge.yaml

The paths is a list of absolute (in the context of the storage) paths to container or bridge files.

timeline storage

timeline is an example storage with subcontainers. It’s a storage that sorts files according to the modification time. For example, if a file foo.txt has a modification time of 2020-05-01, it will be available under 2020/05/01/foo.txt. At the same time, each date-based directory is exposed via a subcontainer with a path /timeline/<YEAR>/<MONTH>/<DAY>.

timeline example (using CLI)

Create a user, if you haven’t done that yet:

$ wl user create User

Create the “reference” container, and directory with files:

$ wl container create Inner --path /reference

$ wl storage create local Inner --location $HOME/proxy-data \
    --container Inner
$ mkdir ~/proxy-data
$ touch ~/proxy-data/file1.txt -t 202005010000
$ touch ~/proxy-data/file2.txt -t 201905010000

Create the proxy container storage:

$ wl container create Proxy --path /.proxy

$ wl storage create timeline Proxy \
    --reference-container-url file://$HOME/.config/wildland/containers/Inner.container.yaml \
    --container Proxy

Mount:

$ wl start
$ wl container mount --with-subcontainers Proxy

You should be able to see the files:

$ find ~/wildland/timeline
/home/user/wildland/timeline
/home/user/wildland/timeline/2019
/home/user/wildland/timeline/2019/05
/home/user/wildland/timeline/2019/05/01
/home/user/wildland/timeline/2019/05/01/file2.txt
/home/user/wildland/timeline/2020
/home/user/wildland/timeline/2020/05
/home/user/wildland/timeline/2020/05/01
/home/user/wildland/timeline/2020/05/01/file1.txt

timeline example (self-contained manifest)

All manifests can be inlined. You can create a container.yaml file (or edit existing one using wl container edit)

owner: <OWNER>
paths:
  - /.uuid/11e69833-0152-4563-92fc-b1540fc54a69
  - /.proxy

backends:
  storage:
    - type: timeline
      container-path: /.uuid/11e69833-0152-4563-92fc-b1540fc54a69
      owner: <OWNER>
      reference-container:
        owner: <OWNER>
        paths:
          - /reference
        backends:
          storage:
            - type: local
              container-path: /.uuid/11e69833-0152-4563-92fc-b1540fc54a69
              owner: <OWNER>
              path: /home/user/proxy-data

This file can be signed with wl container sign (the edit command will do that automatically), then mounted using wl container mount.

--only-subcontainers option example

There are use-cases where you want treat a parent container only as a wrapper for the subcontainers. This means that you want to point at the parent container, mount all of its subcontainers but skip mounting the parent container’s storage itself.

This option is going to work only if --with-subcontainers is set to true.