@afp/toolkit-decisive-moment

About

This is a WIP / POC. This project aims at interfacing some functionalities of FFMPEG.wasm (only ffmpeg core available at time of writing) with a nice UI for the AFP journalists.

For now the only use-case is simple: a journalist needs to extract a few seconds -the "decisive moment"- from a (possibly quite long) video, to promote it on diverse (social) platforms. Hence it should also be cropped/resized to match the technical implications of those platforms.

It involves a step-by-step process, which should aim to be as modular as possible (hopefully, for future developement of similar tools).

Client-side app

This app is using the ffmpeg.wasm library in order to leverage the computing power on the client-side, instead of server-side. This aims to both unload DDV server(s), and avoid long upload/download durations. For now the video the user wants to edit is assumed to be their computer already.
Later-on we could imagine posting a request for each ffmpegOperation to a server with a native FFMPEG installed (not using WebAssembly), if this becomes problematic (see SharedArrayBuffer issue below, for example).

---

Store

This is where the main logic is.
We created a custom reactive store store.js with the Vue.observable() function (instead of using Vuex). We could not use Vuex in the project because of the way it is integrated in Toolkit (which is an other Vue app). The store should be extended very carefully or rewritten to avoid mutability or anti-patterns.
It is responsible for:

• Initializing FFMPEG
• Set the current/prev/next Step
• Handling FFMPEG Queue (async logics), which queues up the different FFMPEG commands created within the app. See "Operations Queue" below.
• Reduce the params created inside the Steps, make Operations from them, and feed these to the Queue.

FFMPEG.wasm

The ffmpeg WebAssembly code is initialized by the createFFmpeg() function in the store.js. It can take a few seconds to load (async, so doesn't block the main thread).

doFfmpegOperation.js

This is the file where we transform javascript objects into a string (see FFMPEGparamsList) which is the command passed to FFMPEG.wasm. The function doFfmpegOperation is not called directly, but instead passed as a callback to the ffmpegQueue (see Queue below).

Operations Queue

See Queue.js. This is a concurrent queue. It allows to execute the FFMPEG command lines asyncronously. By associating an id to every Operation, we can freely add tasks to a queue/worker, and get their result back when they are finished.

Components

Batcher (FfmpegBatcher.vue)

This is the parent component. It is responsible for:

• Listing all the Vue components for each Step.
• Apply forward/backward animations between those Steps.

The Steps

There are 3 types of Steps for now:

• One initial step (StepImport here)
• Several intermediary steps, which are the operations that the user cares about: StepMoment (previously StepCut) and StepCrop for now.
• One final step (StepSave) The aim is to be able to re-organize or create additional intermediary steps, in order to create different workflows (and so different use-cases, with an identical or similar code-base).

Import Step (StepImport.vue)

This is the step where the user manually imports the video(s) he wants to modify.
It's only one video for now, but we could imagine a multiple-imports possibility in the case we would want to combine videos. Also we only manage a few file formats for now.
To support other file formats we would need to convert them up-front, to be able to display them in the browser (this is problematic for long videos).

Save Step (StepSaver.vue)

This is the last step. The user is able to:

• See the items queued from the previous step (loaders),
• Vaguely inspect the output files,
• Download them seperately or in a single zip file.

Intermediary Steps (StepMoment.vue, StepCrop.vue, ...)

These are the steps where the user actually modify/edit the video.
These should have as much feedback as possible on the immediate result(s) of the editing.

---

Some known issues

SharedArrayBuffer

This is used by FFMPEG.wasm (like most WebAssembly code), and is subject to big changes/restrictions. As this project is hosted with other tools, this could become problematic if we cannot apply the new security rules (see https://web.dev/cross-origin-isolation-guide/).
If we are unable to use the FFMPEG.wasm library, the most realistic alternative seems to be using the same workflow as DDV (using FFMPEG on a server, instead of client-side). But this would imply huge video transfers (or new restrictions) and an even bigger load on AFP servers.
An other way would be to find an equivalent of FFMPEG.wasm that does not use SharedArrayBuffer, but it seems unlikely at time of writing (mostly for performance reasons).

Mutable store

The current reactive store (using Vue.observable()) is a compromise. Maybe a rewrite using a dedicated library would be necessary some time later.
Vuex could not be used because of Toolkit-related issues (integration) at time of writing. Simply put: currently Vuex must be on the App level, while this code is supposed to be imported in an other, bigger Vue App.

---

Project setup

npm install

Compiles and hot-reloads for development

npm run serve

Compiles and minifies for production

npm run build

To publish to npm (for toolkit integration)

npm run build && npm version patch && npm publish

Lints and fixes files

npm run lint

npm run lint --fix

Customize configuration

See Configuration Reference.