[web-animations-1] Make updating the playback rate asynchronous

[birtles]

From @alancutter on May 6, 2016 3:52

Playing and pausing Animations are asynchronous operations due to the existence of off main thread animations. This detail is captured with pending play and pause tasks.

Changing the playback rate of an Animation is similarly asynchronous and should have its own pending task.

Copied from original issue: w3c/web-animations#151

[birtles]

What about setting the start time, current time, iteration count etc.? Those also will take time to be reflected by off main thread animations.

I guess the difference here is that when we update the playbackRate we want to use the currentTime as seen on the compositor as the currentTime to maintain?

The reason play and pause set the ready promise is that we don't know in advance what startTime / currentTime we'll end up using for those. So the idea here would be that the currentTime will shift in between calling playbackRate = 2 and the ready promise resolving in a way that is not necessarily equivalent to what would happen if we just used the currentTime on the main thread when the playbackRate was set.

This could be quite tricky to spec since we need to define all the interactions with other things that affect ready promises etc. (e.g. if you set the playbackRate, then call pause() do we re-use the ready promise etc. etc.). Also, we'll need to work out what happens with reverse().

[birtles]

From @shans on November 7, 2016 2:38

When changing the playback rate, you're effectively defining a new startTime. This is again unknown until the compositor sees the change.

[birtles]

I started working on this and I wanted to make a few notes.

Firstly, I think there are probably three different models for how this could behave.

A. Setting the playback rate does the following three things:

1. Makes the start time unresolved
2. Sets the hold time (just until the async task runs)
3. Updates the playback rate

Note that the above three are a set.

If we decide, for example, not to make the start time unresolved, then the relationship between start time, current time, and playback rate will be broken. Furthermore, we'd violate the invariant that only one of the start time and hold time can be set at any given moment.

Likewise, if we do clear the start time, but don't set the hold time, we won't be able to calculate a current time until the async task runs which would be weird.

B. Setting the playback rate does nothing until the async task runs

This is pretty attractive because then everything happens as a batch. It means storing extra state but is otherwise possibly simpler.

However, it's a bit weird from a JS point of view. e.g.

anim.playbackRate = 2;
console.log(anim.playbackRate); // --> '1'? Huh?

C. Setting the playback rate updates the values reported to JS but internally we don't update the state until the async task runs

This, I think, is actually more complex still.

It seems (A) is the right way to do this and I believe it might match what Chrome does.

In terms of spec changes, I think most of the procedures just need updating to say to cancel the pending playback rate change task in places where we cancel pending play/pause tasks.

As for the procedure to update the playback rate, it seems quite complex.

Considering the desired behavior by playback state I think we have something like:

Playing

1. Set the hold time to the current time
2. Make start time unresolved
3. Update playback rate to the new value
4. Create new ready Promise
5. Wait until we are ready to play with the new playback rate
   1. Calculate the previous time as the current time we'd get from using ready time as the timeline time, and the old start time and playback rate.
   2. Do necessary parts of the silently set the current time procedure to update the start time based on the updated playback rate so that the current time matches previous time (and using ready time as the timeline time).
   3. Resolve the promise

Paused/Finished/Idle
No change needed: these can happen synchronously.
(Ideally we can just do the same as for 'playing' but run the task synchronously and not create new Promise objects.)

Play pending
I'm pretty sure we can make the async task from updating the playback rate work in this case. However, there are some odd cases too.

Like if we do:

// Animation is running
anim.playbackRate = 2;
anim.play();

When we finish syncing with the compositor, we should calculate the start time from the running time, not the (temporary) hold time.

There might be some way to make the pending play task handle this and re-use it. I'm not sure.

Pause pending
In this case, we want to update not the start time, but the hold time using the previous time we calculate above.

I'm not really sure yet how we should factor this out. e.g.

• Should we have a playback rate pending change task which has context "pausing" or "playing" so it knows whether to update the start time or hold time?
• Or should we re-use the play pending task and pause pending task and pass in extra context that tell us:
  • How to calculate the equivalent hold time
  • Something that tell us that when we are doing the pause not to preserve the existing hold time

I suspect a variant on the second approach where we just some sort of previous-playback-rate-to-start-time ratio is enough to both:

• allow us to calculate the correct "previous time"
• indicate that we should update the hold time

[birtles]

This is proving messy. If we update the playback rate and clear the start time immediately, then when the async task runs, we want to calculate where the animation would be if continued running with the original start time and playback rate, and then use that as the current time when calculating the updated start time / hold time. i.e. it's not a single ratio but two values that need to be stored. Perhaps I can just use some hand-wavy language for this.

[birtles]

I think the complexity I'm facing here comes down to the fact that we're triggering an async operation from simply setting a member.

If we made playbackRate update synchronously and not do a compensatory seek to maintain the currentTime, and then had a separate method setPlaybackRate() that updated the playbackRate and did the compensatory seek, this would be a lot easier and there would be a strong parallel with calling play(), pause() vs setting currentTime, startTime. I think that ship has probably sailed, however and we're stuck with this behavior.

[birtles]

The more I work on this, the more I think it's not a good idea for setting a member to trigger an async operation like this.

There are a few reasons but perhaps the most concrete issue is that by doing that we would provide no obvious way to achieve synchronous behavior for applications that require it.

Consider the asynchronous operations.

1. Playing

Async play:

anim.play();

Sync play (i.e. start on this main thread frame):

anim.startTime = document.timeline.currentTime;

2. Pausing

Async pause:

anim.pause();

Sync pause (i.e. pause at this main thread frame):

anim.startTime = null;

3. Changing the playback rate

Currently we only have:

anim.playbackRate = 2.0;

which we are trying to make asynchronously update timing.

If you want to synchronously update the playback rate with a compensatory seek you would need to do something like:

// Record current time before we update startTime
const previousCurrentTime = anim.currentTime;
// Pause first since updating the playbackRate can be done synchronously when paused
anim.startTime = null;
anim.playbackRate = 2.0;
// Do compensatory seek
anim.startTime = (document.timeline.currentTime - previousCurrentTime) / 2.0;

At a glance it seems like if we just define setting the currentTime to cancel the async compensatory seek, then the following would work:

const previousCurrentTime = anim.currentTime;
anim.playbackRate = 2.0;
// Define the following to cancel the async playback rate change
anim.currentTime = previousCurrentTime;

However, that's not the case because in the above example anim would be paused since setting the playbackRate causes the startTime to become unresolved (and there would be no async operation to resolve the startTime, unless we make setting the currentTime here also resolve the startTime in this case, which seems at least a little odd).

I've checked what Chrome does in this case, and it doesn't appear to cancel the async update. That is, there's no obvious way to update the playbackRate in a synchronous way (except perhaps the pause-first approach, which I haven't tried).

Updating the playbackRate without doing a seek is not too bad though:

const previousStartTime = anim.startTime;
anim.playbackRate = 2.0;
// Define the following to cancel the async playback rate change
anim.startTime = previousStartTime;

Preferred solution

I think the way this interface should work is:

partial interface Animation {
   // setting synchronously updates the playbackRate with NO compensatory seek
   attribute double playbackRate;
   // asynchronously updates the playbackRate and does a compensatory seek
   void updatePlaybackRate(double playbackRate);
}

Async playback rate change:

anim.updatePlaybackRate(2.0);

Sync playback rate change:

anim.playbackRate = 2.0;

Or, sync playback rate change with compensatory seek:

const previousCurrentTime = anim.currentTime;
anim.playbackRate = 2.0;
anim.currentTime = previousCurrentTime;

Doing this would mean you have a clear distinction where:

• Setting members is always synchronous. The model is updated immediately based on main-thread state even if that means it causes the animation to jump.
• Methods form an asynchronous layer that update members at different times with appropriate values.

If we introduce the API changes suggested in #196, it would mean that isPending is only ever set to true by calling methods; never by setting members.

I'm dubious, however, that we can drop the compensatory seek arising from setting playbackRate without breaking content. I'd be very very glad to be proven wrong, however.

Realistic solution

Assuming we can't drop the compensatory seek behavior from playbackRate, then I think we should make this interface as follows:

partial interface Animation {
   // setting synchronously updates the playbackRate with compensatory seek based
   // on main thread state (currently specified behavior)
   attribute double playbackRate;
   // asynchronously updates the playbackRate and does a compensatory seek
   void updatePlaybackRate(double playbackRate);
}

Async playback rate change:

anim.updatePlaybackRate(2.0);

Sync playback rate change with compensatory seek:

anim.playbackRate = 2.0;

Sync playback rate change without compensatory seek:

const previousStartTime = anim.startTime;
anim.playbackRate = 2.0;
anim.startTime = previousStartTime;

The downside of this is that some authors will never notice updatePlaybackRate and just set playbackRate even when they probably want the async behavior.

@alancutter what do you think?

[birtles]

Discussed with @shans and @dstockwell at BlinkOn on Sep 20. Adding updatePlaybackRate() and making playbackRate synchronous seems straightforward.

The question that remains is whether we can drop the compensatory seek from setting playbackRate. The strongest motivation for doing that would be that if both updatePlaybackRate() and playbackRate do a compensatory seek, then some authors will erroneously use playbackRate when they want updatePlaybackRate() and the user may get a sub-optimal experience on a resource-constrained device. If playbackRate doesn't do the seek there's no potential for confusion.

We discussed adding a use counter for content that sets playbackRate such that the startTime actually changes (so that we ignore, for example, setting the playbackRate before playback has begun, or setting playbackRate to 1, etc.).

[birtles]

I'm going to try to make progress on this today. A few updates first:

• We have a use counter in Chrome for seeing if anyone is relying on setting playbackRate to maintain the current time: https://www.chromestatus.com/metrics/feature/timeline/popularity/2162
  • So far it seems rarely used so we could probably drop this behavior.
• I'm less certain that updatePlaybackRate is the right method name. Perhaps just setPlaybackRate is ok. Or even changeSpeed? That might better communicate that the current time stays the same?
  • I could use help with this naming.

[birtles]

I started working on this in commit 76e57a6.

It's a pretty massive change (as predicted in the second comment in this issue). I think the next step will be to try implementing it in Firefox and see what I discover. It's going to need a lot of tests too but hopefully many of them will just come from updating existing tests.

[birtles]

I've updated the branch with a few changes I noticed while implementing. I'm currently thinking:

• Would changePlaybackRate be better (than setPlaybackRate/updatePlaybackRate)?
• I think using a method for this is the right choice. Just fiddling with this it occurs to me that in future we might pass options to this method to do a smooth transition from one playback rate to another.