[css-cascade][css-cascade-3] Test suite appears orphaned, not in web-platform-tests, nor any other discoverable repo

[tantek]

https://drafts.csswg.org/css-cascade-3/ has a Test Suite: link to http://test.csswg.org/suites/css-cascade-3_dev/nightly-unstable/ which as far as I can tell is not in any respository anywhere (e.g. not in web-platform-tests), so there’s no way to report issues on the tests, nor submit patches for the tests.

There is https://github.com/w3c/web-platform-tests/tree/master/css/css-cascade which appears to have a few of the test files, e.g.:

https://github.com/w3c/web-platform-tests/blob/master/css/css-cascade/all-prop-001.html appears to be a port of http://test.csswg.org/suites/css-cascade-3_dev/nightly-unstable/html/all-prop-001.htm

However there are only 8 such test files in web-platform-tests/tree/master/css/css-cascade whereas there appear to be over 100 in http://test.csswg.org/suites/css-cascade-3_dev/nightly-unstable/html/toc.htm

This should be considered a CR-exit blocker (PR blocker), since there is no obvious way for implementers to file issues on (and submit corrections for) the tests which implementations must pass in order to demonstrate interop to exit CR.

E.g. I would file an issue for this test https://test.csswg.org/suites/css-cascade-3_dev/nightly-unstable/html/cascade-import-002.htm, specifically, tests g,h,q,r, are wrong because they depend on implementation of meta http-equiv=link which was officially removed from HTML in 2010! (See: https://www.w3.org/Bugs/Public/show_bug.cgi?id=9797)

Labels: css-cascade, css-cascade-3, Needs review of Test Case(s), PR blocker

(Originally published at: http://tantek.com/2018/061/b1/css-cascade-3-test-suite-not-in-web-platform-tests)

[plinss]

All the test suites on test.csswg.org/suites are generated by the build system (most are done nightly, including css-cascade-3). The source for all the tests are in fact in the WPT repo, as that's the only place the build system gets its tests from.

The build doesn't care which directories the test files are it as it looks for rel='help' links and includes each test file that has a link to the relevant spec(s).

Issues are on tests are filed in the wpt repo. A quick search in the wpt repo finds the source file you're referring to at: https://github.com/w3c/web-platform-tests/blob/master/css/CSS2/cascade-import/cascade-import-002.xht
(note the file extensions may change between the source and the built test suite as the build does xhtml->html conversion (and vice-versa as needed)).

Ba

[tantek]

Appreciate the explanation and link to the source file; makes sense.

However there is still a fundamental usability problem of the discoverability of how to file issues and suggested improvements for CSS module test suites.

I would like to suggest improving the generated test suite home pages themselves (e.g. http://test.csswg.org/suites/css-cascade-3_dev/nightly-unstable/) to link directly to https://github.com/w3c/web-platform-tests/ and suggest searching it for any source files one might want to file issues (or contribute patches) for, as you demonstrated in your comment (which I will do shortly for the cascade-import-002.htm source file specifically, thanks for the pointer. Update, done: web-platform-tests/wpt#9910).

Note: the existing text of "More information on the contribution process and test guidelines is available on the wiki page." is not really useful, as the "wiki page" that is linked to (http://wiki.csswg.org/test) has A TON of links (a maze of passages that all appear alike if you will), none of which contain the precisely useful advice that you gave in your comment! Nor is it readily obvious how to fix http://wiki.csswg.org/test as it seems to serve many purposes, and the two likely links "How to Contribute" and "Reviewing Tests" both say on their pages: "This page has been deprecated and is no longer being maintained." with a top-level link to http://web-platform-tests.org/ which is also not useful, and that's already three clicks deep (if you guessed right which links to click) with still no answer as to how to contribute to this specific module's test suite.

Where should I file an issue and/or patch for the template or generation of the home pages of CSS module test suites like the specific page http://test.csswg.org/suites/css-cascade-3_dev/nightly-unstable/?

Thanks!

(Originally published at: http://tantek.com/2018/061/t2/improving-test-suite-home-pages)

[plinss]

Agreed, the templates are out of date.

The bulk of the code doing the build is in w3ctestlib, the templates specifically are here:
https://github.com/w3c/csswg-w3ctestlib/tree/master/templates

PRs are welcome, but I'll have to merge into the upstream mercurial repo for it to work.

The template engine used is: http://template-toolkit.org/python/ (for syntax reference)

[tantek]

Thanks. Submitted a minimal suggested improvement here:

w3c/csswg-w3ctestlib#1

So at least folks looking to provide feedback on specific test files will have a chance of finding the place to do so.

(Originally published at: http://tantek.com/2018/066/t1/)