[web-animations-2][css-animations-2] Allow controlling video playback via an animation timeline

[flackr]

Similar to #9110, authors often want the ability to drive videos by animation timelines. See #11587 and #6861 (comment).

We'll need to figure out the details, but this could be something like

document.querySelector('#my-video').timeline = new ViewTimeline(...);

Possibly also with css support, e.g.:

#my-video {
  video-timeline: view();
}

[birtles]

For what it's worth, when we designed the Web Animations API, the intention was that driving video playback would happen via different types of effects and you'd synchronize it with other animations via group effects.

[ydaniv]

For what it's worth, when we designed the Web Animations API, the intention was that driving video playback would happen via different types of effects and you'd synchronize it with other animations via group effects.

Yeah, that would be a more coherent method.
If we consider a new VideoEffect that can take a <video> as target and put in side an Animation that takes a timeline and ranges. Then we can reason about synching that animation with others using group effects.

I suppose if we have video-timeline we also need to consider video-range.
But OTOH, we don't really need all the other animation-* properties.

So, how would a declarative API look like? It seems to me that all we need here is -timeline and -range. So that could be the declarative syntax, while the WAAPI side could be a new type of VideoEffect.

WDYT?

[SebastianZ]

I suppose if we have video-timeline we also need to consider video-range. But OTOH, we don't really need all the other animation-* properties.

I can see possible use cases for some other animation-* properties. E.g. @flackr mentioned video-play-state which could possibly also be covered using animation-play-state. For seconds-long videos used in chats you often see them repeated a few times before they stop, which would be covered by animation-iteration-count. animation-duration could influence the video duration. And animation-direction could influence whether the video is played forwards or backwards.

Sebastian

[bramus]

(#) Possibly also with css support, e.g.:

#my-video {
video-timeline: view();
}

Bikeshedding: maybe playback-timeline to keep it open to other media elements?

---

(#) For what it's worth, when we designed the Web Animations API, the intention was that driving video playback would happen via different types of effects and you'd synchronize it with other animations via group effects.

I feel like that’s the type of thing authors would be able to make themselves using an onProgress event, something that’s been part of the discussion for CustomEffect (#6861). I also saw a similar request in #9974 which would be solved by onProgress.

(Also see https://brm.us/scroll-driven-video#the-demos where I built this type of thing – using a rAF lacking any onProgress event)

My feeling here is that instead of trying to determine all sorts of new *Effect interfaces and requiring authors to overhaul their code to use a GroupEffect, this can all be unified in onProgress.

If the GroupEffect is considered to be more viable than onProgress, I think we should offer authors a BaseEffect class that they can extend in order to roll their own VideoEffect to drive a video progressing based on an animation. This way they would not need to wait for all vendors to implement that specific type of Effect, or any other new type of Effect.

To make these effects available in CSS, they could register it using something like CSS.registerAnimationEffect(new VideoEffect(document.querySelector('#my-video')), '--video'); … which smells a lot like Houdini at that point :)

(Personally I believe an onProgress better fits the mental model authors have about all this, in which they want to “take an animation (effect) and map its progress to something else”. Yes, GroupEffect allows more composability, but I believe that’s only for “advanced” users.)

---

Winging back to Rob’s proposal: What I like about it is that it allows an author to take a ViewTimeline/ScrollTimeline and use that as the timeline for the media file, without requiring some animation to run. This “an animation must be present” requirement has been a recurring point of friction for me, as I sometimes only need the timeline in order to drive things. In those cases I end up setting a dummy animation to goes from opacity 1 to opacity 1 on some element, which seems a bit stupid to be able to achieve my goal.

[johannesodland]

I can see possible use cases for some other animation-* properties. E.g. @flackr mentioned video-play-state which could possibly also be covered using animation-play-state. For seconds-long videos used in chats you often see them repeated a few times before they stop, which would be covered by animation-iteration-count. animation-duration could influence the video duration. And animation-direction could influence whether the video is played forwards or backwards.

I agree with @SebastianZ .

Having delay and end-delay would be useful to stagger videos, or to control the start and end of a common pre-rendered video depending on the context. We often use pre-rendered videos as illustrations or decorations in our articles where this would be useful.

Having the rest of the options/properties could be useful with a regular document timeline as well. I.e. one could use end-delay to play a video to predefined stop. Direction would allow us to do that in reverse, effectively allowing us to play a video step-by-step. (I would love a .playTo() as well :) )

[flackr]

So what I'm hearing is that we should have an easy way to make an effect that drives the video. I wonder if this would be something like

CSS:

video {
  animation-name: video();
}

Or, in JS:

new Animation(new VideoEffect(video), { /* animation options */ });

To the concern from @bramus about just wanting to drive a video, this would be as simple as:

video {
  animation-name: video();
  animation-timeline: view();
}

[bramus]

So what I'm hearing is that we should have an easy way to make an effect that drives the video. I wonder if this would be something like

CSS:

video {
animation-name: video();
}

Isn’t that inverted? I would imagine the code to look more like this:

#elem {
  animation: anim 5s linear infinite;
  animation-timeline-name: --timeline-from-the-elem;
}

video {
  playback-timeline: --timeline-from-the-elem; /* Link my playback to the progress of the timeline of the animation */
}

I know, animation-timeline-name as a name for the property is a bit weird here. It should export a timeline that exposes the overallProgress of the animation.

[ydaniv]

Isn’t that inverted? I would imagine the code to look more like this:

#elem {
  animation: anim 5s linear infinite;
  animation-timeline-name: --timeline-from-the-elem;
}

video {
  playback-timeline: --timeline-from-the-elem; /* Link my playback to the progress of the timeline of the animation */
}

I know, animation-timeline-name as a name for the property is a bit weird here. It should export a timeline that exposes the overallProgress of the animation.

These are two different features:

1. Allow an animation use a video's playback as timeline.
2. Allow a video's playback use a progress-based timeline, e.g. ViewTimeline.

Example for the first can be playing a canvas-based effect on a video.
And the second is simply video scrubbing via scroll.

Both are valid, but in this case we're referring to the second.

---

I like the video() approach as a declarative way of binding the progress of the animation to the playback, and then we can reuse all the props that animations bring with them. i think also quite coherent with the VideoEffect in WAAPI.

Just wondering as possible future extensions, other things we may want to control on a video could be:

1. playbackrate
2. volume - though this is probably better handled via separate API for audio

Just to keep in the background.

[bramus]

Both are valid, but in this case we're referring to the second.

So do I.

With playback-timeline I mean to “consume” a timeline, similar to animation-timeline. I guess it’s pretty confusing because view-timeline and scroll-timeline allow you to create a timeline. And in my example animation-timeline-line would also be used to create a timeline. Maybe progress-timeline-name is a better word for it.

(In hindsight maybe we should have called animation-timeline animation-driver or something similar?)

[flackr]

The reason to use a video effect, is that you get to re-use the rest of the animation ecosystem. With playback-timeline you then either don't have, or need to add other ways to specify the following:

• Specify named ranges, offsets within those ranges
• Have non-linear playback of the video
• Looping the video some number of times over the given timeline

Building this on top of an animation effect gets you all of these things implicitly. I think of it as that you are animating the video progress, which is similar to animating some internal css property that sets the video time.

The only part I'm uncertain of, which playback-timeline doesn't handle any better, is how to differentiate multiple videos on the same element - e.g. supposing we have support for video as a css image #6732 it would be nice to be able to associate the animation with a particular video.

[ydaniv]

With playback-timeline I mean to “consume” a timeline, similar to animation-timeline. I guess it’s pretty confusing because view-timeline and scroll-timeline allow you to create a timeline.

Like you said, playback-timeline should be declaring a timeline for using it in animations, just like view-timeline and scroll-timeline. This is probably more consistent with our current syntax for SDA.

---

Building this on top of an animation effect gets you all of these things implicitly. I think of it as that you are animating the video progress, which is similar to animating some internal css property that sets the video time.

Yes, agreed.

The only part I'm uncertain of, which playback-timeline doesn't handle any better, is how to differentiate multiple videos on the same element - e.g. supposing we have support for video as a css image #6732 it would be nice to be able to associate the animation with a particular video.

I think that video() is the anonymous form which works fine with the current source of the element.
I guess if we'll need to consider a single element with multiple videos we'll need to either have some sort of a coordinated list, or we'll use a new property to define idents for each source.

But another thing that comes to mind, we need to think whether video() is the correct syntax for referring to the playback as an effect, or whether we'll want to use that for referring to a playback timeline? Kind of like view() and scroll()? Or maybe it could be both, depending on context?

[SebastianZ]

While we were focusing on video playback so far, I want to note that any solution we come up here should also apply to audio playback. So the functionality and naming of the new feature should be general enough to cover both.

---

Allow an animation use a video's playback as timeline.

This was proposed by @bramus in #9110.

Sebastian

[ydaniv]

want to note that any solution we come up here should also apply to audio playback

Audio is a bit more problematic because of autoplay policy, but that aside, I'm not sure how desired, or even viable, it is to scrub an audio file using scroll or pointer. Audio behaves differently from video in these cases, e.g. pitch will change based on playback rate.
So I would suggest deferring control of audio playback to Custom Effects et al.