[web-animations] Allow adjacent filling animations to be coalesced

[birtles]

Closely related:

• Additive transform animations easily invoke undesirable matrix interpolation ([web-animations-1] Additive transform animations easily invoke undesirable matrix interpolation #2204)
• Define requirements for retaining animations to be returned by getAnimations() ([web-animations-1] Define requirements for retaining animations to be returned by getAnimations #2054)

Using the Web Animations API authors can easily generate animations in an unbounded fashion. Unless a means is provided to UAs to discard or compact these animations, this will result in a memory leak.

For example:

elem.addEventListener('mousemove', evt => {
  circle.animate(
    { transform: `translate(${evt.clientX}, ${evt.clientY})` },
    { duration: 500, fill: 'forwards' }
  );
});

Comparing to other animation techniques:

• CSS transitions do not suffer this problem since there can only be one animation per property at any time.
• SMIL does not suffer this problem since each <animate>-type element only has one active interval at a time so that number of simultaneous animations is bounded by the number of <animate>-type elements (and if an author is producing <animate> elements in an unbounded fashion they hopefully won't be surprised to learn that they're leaking memory!).
• CSS animations technically suffers this problem except that it is more obvious to authors that they are producing a memory leak since they must explicitly append to the list of animations each time. By contrast, there is nothing in the above code to suggest to an author that they are creating a memory leak.

One reason past animations must be retained is due to the way the getAnimations() API is currently defined. Issue #2054 will define a means to avoid having getAnimations() return all such animations.

The other reason past animations must be retained is to produce the correct result when additive animations or animations with implicit keyframes are involved. As a result, even once getAnimations() is fixed, there are many circumstances where UAs cannot discard or compact past animations, thereby producing a memory leak.

[birtles]

Analysis of the problem

Issue 1: Specified values can depend on context

Animations can be defined with property values such as 10em, var(--yellowish), or 50vh. The corresponding computed values will depend on context that can change over time.

Consider an animation that moves an element to left: 90vw. The expectation is that even after this animation has finished and is filling, if the user rotates their screen then the element's left position represents 90vw of the updated viewport dimensions (and not, for example, the computed value of 90vw at the point when the animation finished).

When several animations affecting the same property of the same element have finished and are filling, the final animated value of the property might be the combination of a number of such context-sensitive values (e.g. 1rem + 2em + var(--active-nudge-width)). If the context changes, the computed value of any one of these values might update affecting the end value. As a result UAs must maintain all the specified values of filling animations where additive animations are used.

Furthermore, this problem does not merely apply to additive animations (i.e. animations with composite operations add or accumulate as part of their fill value) but also to animations that use implicit 0% or 100% keyframes. That is because such animations might fill mid-way through an interval, e.g.:

elem.animate({ left: '5em' }, { duration: 1000, iterations: 0.5, fill: 'forwards' });
elem.animate({ left: '50vw' }, { duration: 1000, fill: 'forwards' });

In the above example the fill result will be 50vw + 2.5em.

Issue 2: Operations cannot be re-ordered to create an intermediate result independent of the underlying value

To calculate the animated value of a property, a stack of animation effects is generated. In order to address the memory leak identified in this issue we would like to coalesce adjacent filling effects within this stack.

Even supposing issue 1 can be resolved such that we do not need to maintain the specified values of such effects, we face another problem: we cannot always calculate the result of an effect independent of its underlying value. As a result, we cannot coalesce effects.

In general, the result of a filling keyframe effect for a single property is as follows:

result = (L acc(n) (U comp_A A)) interp(p) (L acc(n) (U comp_B B))

where:

• L = value of the last keyframe at offset 1.0 (if any)
• acc(n) = accumulate operation applied n times to LHS, with RHS applied after that
• U = underlying value
• comp_A,_B = composite operator for value A / B
• A = the ‘to value’ being interpolated from at progress p
• interp(p) = interpolate operation at progress p
• B = the ‘from value’ being interpolated to at progress p

In order to be able to calculate this result independent of U so that we can combine it with adjacent effects we would need to factor out U such that it is an independent term in the above expression.

For example, if we could somehow re-write the above along the lines of:

result = (U interp(p) ∅) comp_A (U interp(p-1) ∅) comp_B ((L acc(n) A) interp(p) (L acc(n) B))

(where ∅ represents a no-op/neutral value as described in #2204)

we could most likely represent a section of the stack as a component that is independent of the underlying value (the ((L acc(n) A) interp(p) (L acc(n) B)) part) plus some portions of the underlying value.

For many properties this is possible. For example, since lengths are combined using addition and multiplication, we can trivially factor out the underlying component and add it later.

However, consider transform lists. Interpolation and accumulation produce different results based on the shape of the inputs and are therefore not only not commutative (neither is addition) but nor are they associative.

For example, suppose we have an animation that fills forwards at the midpoint of an interval between ‘rotate(30deg)’ and ‘rotate(390deg)’. Furthermore, suppose both values have composite operation ‘accumulate’. The animated fill value is calculated as:

result = (U accumulate ‘rotate(30deg)’)) interp(0.5) (U accumulate ‘rotate(390deg)’)

Now suppose U, the underlying value, is ‘translate(10px)’. Since ‘rotate(x)’ and ‘translate(x)’ have a different shape, we will convert both to matrices first, and then accumulate them. However since ‘rotate(30deg)’ and ‘rotate(390deg)’ have the same matrix representation, when we interpolate the result we will end up with just the matrix-equivalent of ‘translate(10px) rotate(30deg)’.

If we try to factor out a component that is independent of the underlying value (e.g. by substituting 'none' for U) and interpolate just ‘rotate(30deg)’ and ‘rotate(390deg)’, we will get ‘rotate(210deg)’. If we subsequently accumulate that with U of 'translate(10px)' will end up producing ‘translate(10px) rotate(210deg)’ instead.

You can observe the difference in output here: https://codepen.io/birtles/pen/GYWxww
(requires Firefox Nightly)

Because we have interpolation and accumulation procedures that are not associative it seems we cannot collapse the adjacent filling animations in the general case without altering the output.

[birtles]

Proposed solution

First a few prerequisites:

• It would be preferable the solution doesn't produce a lot of special-case code that gets rarely exercised and hence poorly tested (e.g. behavior that only applies when filling multiple additive animations and only when the shape of the arguments doesn't match)
• It would be preferable to avoid any sharp discontinuities when an animation finishes (e.g. making specified values responsive to context while running but not when the animation finishes)
• It would be preferable to make animations do sensible things when using context-sensitive values (i.e. just saying "use computed values all the time" is suboptimal).

And a few observations:

• This problem not only affects additive animations but also animations with implicit keyframes which are very convenient. They are used in CSS transitions (implicit from), in CSS animations (implicit 0%/100% keyframes), in SMIL animation (to-animations) and are popular in techniques such as FLIP animation (which are best written with an implicit to value). Simply saying "Let's abandon composite operations" doesn't solve the problem unless we are also willing to abandon implicit keyframes.
• For the same reason, simply saying "Let's drop per-keyframe composite operations" doesn't solve the problem. Implicit keyframes are essentially keyframes with a zero value and a per-keyframe 'add' composite operation.

I've discussed this with @heycam a little and while I still don't have any really good solutions, my current thinking is:

• Define a concept fill-able value that is a representation of a specified value that:
  • Is not context-sensitive
  • Is of such a form that when combining with another fill-able value all operations are associative (I suspect we need more than just "associative" but this math is not my strong point)
• Require that any keyframe set through the API with a composite operation other than 'replace' has its values converted to corresponding fill-able values at the point where there keyframe is set.
• Re-introduce a neutral value / no-op concept.
• Possibly drop 'accumulate' as a composite mode and keep it only as an iteration composite mode. This might not be necessary but if we are to keep it, I will likely need help with the math to prove we can keep it.

As an example, for a transform list value, the fill-able value would be the matrix form of the value. This would only apply for keyframes with a composite mode of 'add' or 'accumulate' (so that most common use cases still work as expected) and the conversion would happen on setting so there is no change in behavior between running and filling. This conversion should ensure we have means of representing a adjacent filling effects that is independent of context and the underlying value. Calculating this value will likely require the use of the neutral / no-op value concept.

I've tried and abandoned a number of other approaches and this is the best I have so far. It's not ideal in that it means you basically can't use 6em and composite: add in a responsive manner but I don't have any better ideas yet (and I actually expect that for some use cases you want the 6em to not be responsive).

[birtles]

One further issue to clarify is that there is currently discussion of making interpolation dependent on context with regards to color-interpolation (see #366). This too would inhibit collapsing stacks of animations. Presumably we would need to clarify at what point "interpolation context" is frozen.

[birtles]

In light of the requirement to freeze interpolation context, I'm starting to consider converting these additive keyframes at the point when the animation begins filling indefinitely. It would mean a jump in behavior but not a jump in output and I wonder if it might be more natural for an author to understand ("Oh, that animation's not updating because it's finished").

SMIL takes this to the extreme where, per spec, a frozen to-animation doesn't even respond to the base value anymore. At least in Gecko we chose to deviate from the spec here and make it responsive.

[birtles]

I've updated the patch for this so that it should be mostly complete now.