Documents and stored objects¶

Chill has a whole system to manage documents, provided by the ChillDocStoreBundle. Other bundles rely on it whenever a file must be attached to something: documents linked to a person or an accompanying period, documents generated by the document generator, exports, the user manual, etc.

This page describes the architecture of this system, and the tools available for developers: entities, storage, encryption, upload from the browser, forms and widgets.

Overview¶

documents metadata are stored in StoredObject entities;
a document may have multiple versions, stored in StoredObjectVersion entities. Old versions are cleaned regularly (see deletion and cleanup);
the content of the documents is read and written through Chill\DocStoreBundle\Service\StoredObjectManagerInterface. Two built-in implementations exist, located in src/Bundle/ChillDocStoreBundle/AsyncUpload/Driver: LocalStorage and OpenstackObjectStore;
documents are stored encrypted on the storage. The key and initialization vector to encrypt / decrypt each version are stored in the StoredObjectVersion;
documents are uploaded directly from the browser to the storage, using signed URLs, and also downloaded and decrypted directly in the browser;
Chill offers the possibility to edit documents online (as a WOPI server, using Collabora) or with the local software (LibreOffice, using WebDAV). See Editing documents.

Entities¶

StoredObject¶

Chill\DocStoreBundle\Entity\StoredObject represents a document, and holds its metadata:

the title of the document, if it has one, should be stored in the StoredObject entity (title property);
a uuid, used in api endpoints and by the WOPI server;
a status: empty (just created, no content yet), ready, pending / failure (used by the document generator while a generation is in progress or has failed);
a deleteAt date, to schedule the deletion of the document (see below);
a prefix: each version's filename on the storage starts with this prefix.

A StoredObject does not know which domain entity it belongs to: entities like PersonDocument, AccompanyingCourseDocument, or any entity of your own, hold a relation to the StoredObject.

StoredObjectVersion¶

Each version of a document's content is a Chill\DocStoreBundle\Entity\StoredObjectVersion, which stores:

the filename of the version on the storage (generated from the stored object's prefix and a random suffix);
an incremental version number;
the encryption material for this version: the initialization vector (iv) and the key (keyInfos, a JSON Web Key);
the content type (mime type) of this version.

Versions should not be created manually: use StoredObject::registerVersion(), or better, let the StoredObjectManagerInterface::write() method or the upload flow register it for you.

A version may be pinned by a StoredObjectPointInTime, which prevents its deletion by the cleanup jobs (this is used, for instance, to keep the state of a document at the time of a signature or of a workflow transition).

Reading and writing content: the StoredObjectManagerInterface¶

The content of documents must always be read and written through Chill\DocStoreBundle\Service\StoredObjectManagerInterface:

use Chill\DocStoreBundle\Entity\StoredObject;
use Chill\DocStoreBundle\Service\StoredObjectManagerInterface;

final readonly class MyService
{
    public function __construct(
        private StoredObjectManagerInterface $storedObjectManager,
    ) {}

    public function doSomething(StoredObject $storedObject): void
    {
        // read the current (last) version's content, decrypted
        $content = $this->storedObjectManager->read($storedObject);

        // write a new version (the manager encrypts the content and registers
        // a new StoredObjectVersion on the stored object)
        $version = $this->storedObjectManager->write($storedObject, $newContent, 'application/pdf');
    }
}

Most methods accept either a StoredObject (the last version is used) or a specific StoredObjectVersion.

There are two built-in implementations, located in src/Bundle/ChillDocStoreBundle/AsyncUpload/Driver:

LocalStorage: stores the files on the local filesystem of the server;
OpenstackObjectStore: stores the files in an OpenStack Swift object store, using temporary URLs.

The driver is selected through the chill_doc_store configuration. See Document storage for the configuration of both drivers.

Encryption¶

Documents are stored encrypted (AES-256-CBC, see StoredObjectManagerInterface::ALGORITHM): the storage backend never sees the clear content. Each StoredObjectVersion has its own key and initialization vector, stored in the database (keyInfos and iv properties).

Who encrypts / decrypts depends on where the content is handled:

server side, the driver (the StoredObjectManagerInterface implementation) is responsible for encrypting on write() and decrypting on read();
in the browser, the javascript widgets encrypt the file before uploading it to the signed URL, and decrypt the downloaded content, using the WebCrypto API (see Resources/public/vuejs/StoredObjectButton/helpers.ts). The key and iv travel through the Chill backend api, never through the storage.

Uploading from the browser¶

Documents are uploaded directly from the browser to the storage backend, without transiting through the PHP application. This is the "async upload" flow:

the browser asks the backend for a signed upload URL: GET /api/1.0/doc-store/async-upload/temp_url/{uuid}/generate/post (the user must be allowed to edit this stored object). The TempUrlGeneratorInterface implementation of the driver computes the signed URL;
the browser encrypts the file and uploads it, with the signature, directly to the storage (the OpenStack object store, or the local storage endpoints under /public/stored-object/ when using the LocalStorage driver);
when the surrounding form is submitted, the new version (filename, iv, key, type) is registered on the StoredObject.

The same mechanism, with GET signed URLs, is used to download the (encrypted) content in the browser.

The StoredObjectType form type¶

To attach a document to an entity through a form, use Chill\DocStoreBundle\Form\StoredObjectType. It renders the upload / replace widget, and maps the uploaded version onto the StoredObject:

use Chill\DocStoreBundle\Form\StoredObjectType;

$builder
    ->add('object', StoredObjectType::class, [
        'label' => 'Document',
        // set to true to also edit the StoredObject's title:
        'has_title' => false,
    ]);

The pages using this form type must load the mod_async_upload module, which handles the upload:

{% block js %}
    {{ parent() }}
    {{ encore_entry_script_tags('mod_async_upload') }}
{% endblock %}

{% block css %}
    {{ parent() }}
    {{ encore_entry_link_tags('mod_async_upload') }}
{% endblock %}

Collections of documents¶

To manage a collection of stored objects (add / remove multiple documents), use Chill\DocStoreBundle\Form\CollectionStoredObjectType (src/Bundle/ChillDocStoreBundle/Form/CollectionStoredObjectType.php). It is a ChillCollectionType pre-configured with StoredObjectType entries (with title), and add / remove buttons.

Displaying documents: buttons and widgets¶

Twig filters¶

The easiest way to render actions for a document in a twig template is the chill_document_button_group filter, which renders a button group with all the actions available for the document and the current user (download, edit online with Collabora, edit with the desktop software through WebDAV, see versions history, ...):

{{ document.storedObject|chill_document_button_group(document.title, is_granted('SOME_EDIT_ROLE', document)) }}

{# smaller variant #}
{{ document.storedObject|chill_document_button_group(document.title, true, {'small': true}) }}

A download-only button is also available:

{{ document.storedObject|chill_document_download_only_button }}

Those filters render a Vue component; the pages using them must load the corresponding module (mod_document_action_buttons_group for the button group, mod_document_download_button for the download button), with encore_entry_script_tags / encore_entry_link_tags as shown above.

Vue components¶

When building a Vue application, use the components directly:

src/Bundle/ChillDocStoreBundle/Resources/public/vuejs/DocumentActionButtonsGroup.vue: displays all the actions for a document;
src/Bundle/ChillDocStoreBundle/Resources/public/vuejs/StoredObjectButton/DownloadButton.vue: a single button to download the document (fetches the encrypted content through a signed URL, and decrypts it in the browser);
src/Bundle/ChillDocStoreBundle/Resources/public/vuejs/DropFileWidget: the drop-file upload widget used by StoredObjectType.

Online and desktop edition, locking¶

Chill offers the possibility to edit documents:

online, in the browser: Chill acts as a WOPI server, and an office suite like Collabora Online renders and saves the document;
on the local software (e.g. LibreOffice), through WebDAV.

See Editing documents for the setup and details.

To prevent simultaneous edition, documents are locked while being edited: a StoredObjectLock entity is attached to the StoredObject. The logic is located in src/Bundle/ChillDocStoreBundle/Service/Lock:

the StoredObjectLockManager service creates, checks and releases locks. A lock records the edition method (WOPI or WebDAV, see StoredObjectLockMethodEnum), a token, the users holding the lock, and an expiration date (60 minutes by default, refreshed while the edition goes on);
expired locks are removed regularly by the CleanOldLockCronJob.

Deletion and cleanup¶

Do not delete the content of stored objects yourself. Instead:

to delete a whole document, set its deleteAt date ($storedObject->setDeleteAt($clock->now())) and remove the domain entities that reference it. The RemoveExpiredStoredObjectCronJob collects expired stored objects regularly, and dispatches one message per version on the message bus; the handler removes the content from the storage, then the version and stored object rows;
old versions of documents are cleaned automatically by the RemoveOldVersionCronJob: versions older than 90 days are removed, except the current version of each document and the versions pinned by a StoredObjectPointInTime.

This logic is located in src/Bundle/ChillDocStoreBundle/Service/StoredObjectCleaner. See also Cron jobs for how these jobs are scheduled.