[css-testing] Normalize linking to CSS properties

[gregwhitworth]

In reviewing the flex test suite I found that there are numerous tests that are testing the same thing but linking to different aspects of the spec. I would like to normalize these, and recommend that tests in the future ensure that they link to the same location based on our decision here. Let me provide an example:

propdef-flex-basis: 9 tests
flex-basis-property: 10 tests

In talking with @gsnedders there is a desire to improve understanding test -> spec coverage, which my node module is trying to achieve. It would be really nice if most tests are linking to the sections linked in the TOC, and only going into sub anchors where necessary.

I say all that to say that my recommendation is to link to flex-basis-property rather than the propdef. That said I don't have a really strong opinion on this, as since we do have a desire to move to more test driven spec writing, we'll need to look at the spec for tests that link to should & must statements within the spec. But I need us to agree on something so that we can make the tests match and thus the counts accurate.

[plinss]

In the past we've preferred linking tests to the most specific anchor in the spec. The spec anchor database (available via json api from shepherd) allows tooling to identify which sections those anchors are in (as used in shepherd and the older test harness).

Also note that anchors tend to move into different sections, often with section # differences between the draft and /TR, some tests also link to the draft vs the /TR version of a spec.

I favor keeping the spec links more specific and having the tooling map the anchors to sections, rather than shift the tests to more coarse-grained linking.

[gsnedders]

So, AFAICT, and @tabatkins should probably clarify, the *-property anchors are hardcoded IDs for sections whereas the propdef-* anchors are generated from .propdef blocks.

I don't think we can reasonably check for this; there's an unbounded set of possible hardcoded IDs for sections, and without being able to reasonably check for this we can't stop people from linking to wrong ID.

[css-meeting-bot]

The CSS Working Group just discussed Testing - Normalize linking to CSS properties.

The full IRC log of that discussion

<Rossen_> Topic: Testing - Normalize linking to CSS properties
<myles> gregwhitworth: a lot of these tests go to ... sections .... so i need to know .... <static static>
<Rossen_> Github issue: https://github.com//issues/1395
<myles> gregwhitworth: i want to clean up flex tests.
<tantek> jensimmons++

[fantasai]

Earlier versions of the testing guidelines said to use only the ToC anchors. If we needed more fine-grained anchors, that generally meant we needed subsections, which we then added to the spec.

[css-meeting-bot]

The CSS Working Group just discussed Testing - Normalize linking to CSS properties.

The full IRC log of that discussion

<Rossen_> Topic: Testing - Normalize linking to CSS properties
<Rossen_> Github issue: https://github.com//issues/1395
<myles> gregwhitworth: a lot of tests are duplicative in the flex tests. They link to the flex bases or ... somewhere else
<myles> gregwhitworth: i want to make them uniform
<myles> gregwhitworth: someone commented
<myles> fantasai: we used to have the rule that you would link only to the table of contents
<myles> fantasai: i'm okay to revise that rule, but that rule was simple
<myles> gregwhitworth: that doesn't work because we need deeper links
<myles> gregwhitworth: except it's okay for now
<myles> fantasai: the main cause where it's tricky is that sections are usually what you want, except for flexbox and grid layout algorithms, each bullet point has an anchor, which allows us to be more fine-grained, and it makes more sense
<myles> gregwhitworth: gsnedders and i were talking about how to understand test coverage. If anyone has any suggestions, please mention them
<myles> dbaron: there are cases where the header is more stable across spec versions
<myles> fantasai: usually we have one section per property
<myles> dbaron: not always, and there are often multiple properties in the same section. you might want to link to one individually
<myles> fantasai: we do that less now.
<myles> dbaron: css-align has a lot of these
<myles> Rossen: we are out of time, folks
<myles> Rossen: we have a solution which will unblock gregwhitworth and gsnedders
<myles> Rossen: need to continue investing in improving
<myles> Rossen: please help if you have ideas
<plinss> s/where the header/where the property link/
<myles> Rossen: do we need resolution?
<myles> gregwhitworth: no
<myles> Rossen: we will follow fantasai off a clifff
<fantasai> ~_~
<myles> gregwhitworth: i'll link to sections for now, and if someone argues, we can revisit
<myles> plinss: let's not make a rule, instead, let's just do this for a single spec
<myles> Rossen: ok
<myles> plinss: ok