Skip to content

Conversation

@khiga8
Copy link
Contributor

@khiga8 khiga8 commented Apr 5, 2022

Fixes https://github.com/github/primer/issues/728
Looking into this as part of accessibility passion week ❤️
cc: @primer/accessibility

What are you trying to accomplish?

This PR introduces a GitHub Action workflow that conducts axe check on examples on the documentation site.
The script uses @axe-core/cli.

We run the documentation site and run axe checks on every iframe on the components and utilities pages.
We exclude rules that are dependent on full-page context as those don't make sense to check.

What approach did you choose and why?

  • Created a needs_to_be_fixed list which represents starting point violations. We should fix this over time.
  • Decided to go with @axe-core/cli because we don't have any browser tests that run doc examples. Introducing those seem high-effort. I'm also not familiar with pa11y, which is a similar, alternate option.

NOTE: the axe CLI will fail when called on a page without the specified element in --include. This is why we limit it to /components/ and /utilities/ path which are guaranteed (as of now) to have iframe elements we want to check.

What should reviewers focus on?

This workflow takes around 5 minutes to complete. Is that acceptable? Thoughts appreciated.

Are additional changes needed?

  • No, this PR should be ok to ship as is. 🚢

@changeset-bot
Copy link

changeset-bot bot commented Apr 5, 2022

⚠️ No Changeset found

Latest commit: ca7717c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch 2 times, most recently from b50e32d to 97afe83 Compare April 5, 2022 02:45
@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch from 97afe83 to b192a17 Compare April 5, 2022 05:04
@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch from ba6516b to e06d043 Compare April 5, 2022 14:07
@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch 4 times, most recently from 190aefc to 4a27fed Compare April 5, 2022 14:35
@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch from 4a27fed to e283ad7 Compare April 5, 2022 14:45
@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch from a1da4fe to 8e3b4e1 Compare April 5, 2022 14:54
@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch from ecae8ed to dc23b54 Compare April 5, 2022 15:49
@khiga8
Copy link
Contributor Author

khiga8 commented Apr 5, 2022

I'm not familiar with pa11y so I went with axe and a custom bash script. I'm not sure which is "better" but it might be worth considering too.

@khiga8
Copy link
Contributor Author

khiga8 commented Apr 5, 2022

@khiga8 khiga8 marked this pull request as ready for review April 5, 2022 16:41
@khiga8 khiga8 requested a review from a team as a code owner April 5, 2022 16:41
@khiga8 khiga8 requested a review from langermank April 5, 2022 16:41
@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch from 3c9ae5c to 76965e5 Compare April 8, 2022 01:14
@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch from 76965e5 to 191b02d Compare April 8, 2022 01:18
@khiga8 khiga8 force-pushed the kh-add-axe-workflow branch from 2ad98c9 to 4a24aaf Compare April 8, 2022 01:46
Copy link
Contributor Author

@khiga8 khiga8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonrohan, @simurai This is ready for a hopefully final review :)

I updated things a bit after thinking through @simurai feedback!

Unlike PRC or PVC docs, Primer CSS doc examples don't have dependencies on external code since it's just HTML and CSS. CSS changes shouldn't affect HTML semantics so I think we only really need to run axe on URLs that correspond to doc files that were modified rather than on every doc page for every update.

Changes:

  • Added this handy action which allows us to skip steps in the axe workflow based on pattern matching of changed files. This allowed us to get rid of some unpleasant parsing/conditional logic from our bash script.
  • We only run axe checks on urls that correspond to doc paths that were modified which gets passed in from the Action in the former step.

Example run of when there are multiple modified files. We only run axe on 2 urls that correspond to paths that were changed. Time: 3 min 25s

Example run of axe action early returning because there were no changes to doc paths we cared about. Time: 9s

If we decide it is useful to run full axe checks for every doc page on every PR run, we can always revert back to: bd9eb4b.

Comment on lines +72 to +83
echo "==========================================================================================="
echo "WARNING"
echo ""
echo "Oh no! We have to skip accessibility checks on these doc pages you updated:"
echo ""
printf '%s\n' "${skipped[@]}"
echo ""
echo "because they are part of the starting point violations list."
echo ""
echo "Help us get one closer to getting rid of this list!"
echo "Consider addressing the accessibility issues on the examples of these pages as part of your PR ❤️"
echo "==========================================================================================="
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would love if we added this message as a comment to the PR as a future enhancement.
This will bring greater attention to the fact that axe check is being skipped on a doc page someone has updated.

fetch-depth: 0
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@v18.6
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# This workflow runs axe checks on modified doc/content/components/* or doc/content/components/* pages because those include code examples.
# Developers frequently copy and paste examples directly so it's important to ensure the examples are accessible.
# Learn about @axe-core/cli: https://github.com/dequelabs/axe-core-npm/tree/develop/packages/cli

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This STRING_OF_PATHS_WE_CARE_ABOUT will be passed in by the previous step.

@khiga8 khiga8 marked this pull request as ready for review April 8, 2022 02:07
Copy link
Contributor

@simurai simurai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the speed improvements. They sound great. 🏇

@khiga8
Copy link
Contributor Author

khiga8 commented Apr 11, 2022

Anything else I should do to get this merged? 🙏 @primer/css-reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants