Essential accessibility testing
BS 8878 describes a combination of methods it regards as a
- Validation testing of code (8.4.2)
- Manual testing against WCAG 2.0 (and/or ATAG if applicable)
- Assistive technology testing and adjusting browser/OS settings
that people with disabilities may use. (8.4.4)
Note that the above testing method does not necessarily involve
people with disabilities, so if this is the full extent of the
testing you must ensure that your site features a simple feedback
mechanism that is easy to find and use and that allows users with
disabilities to involve real users in your web product
Note also that this type of testing is limited by the tester's
knowledge of the WCAG Guidelines, and of the needs of disabled web
BS 8878 notes that the best way for a product to comply with the
Equality Act (link ext http://www.equalities.gov.uk/equality_act_2010.aspx
(this link will open in a new browser
window)), and the DDA (link ext
http://www.dwp.gov.uk/employer/disability-discrimination-act/ (this link will open in a new browser
window)) is to carry out user testing with people
Higher levels of testing (expert review and user testing)
BS 8878 strongly recommends using the
following methods to test the usability and technical accessibility
of the web product more extensively. Testing should be carried out
on early designs, prototypes and finished code.
- Expert reviews(link expert reviews 8.4.5)
should be carried out by agencies familiar with the
technology needs of people with disabilities and web product best
practices to assess and identify potential issues with the
accessibility of early designs and prototypes.
- User testing (link user testing 8.4.6) with
disabled and older users who are representative of the web
product (link 6.7 user goals). To asses the usability of the
As part of the web product's post-launch accessibility
maintenance, organizations should conduct testing using a
combination of automated tools (link int) and
markup validation (link int). The purpose of this
testing is to make sure that the accessibility of the product has
not been compromised by work carried out on the site
This type of testing is low-cost and intended to provide a type
of alert to issues that may have been introduced at some point, but
should not be considered as a replacement for more comprehensive
Manual testing for conformance to WCAG 2.0 and ATAG
Claims of WCAG 2.0 conformance require each page of a web
product to be tested against a set of guidelines.
BS 8878 notes that It is important that this type of testing is
carried out by testers who can demonstrate a good understanding of
the success criteria for WCAG 2.0 and ATAG guidelines (where
Access 8878 would add that an understanding of specific user
needs and the operation of assistive technologies is also very
important in order to properly understand the guidelines.
We would also add that claims of conformance to WCAG 2.0
guidelines based on testing a number of representative sample pages
should be duly noted in any accessibility statement, and a list of
pages that formed the sample should also be provided.
Testing using assistive technologies and browser/OS settings
commonly used by disabled users (8.4.4)
Testing should be carried out by testers who are trained /
competent in the use of these technologies so that they are using
them in a similar way to people who uses those technologies all of
The level of testing and the technologies used (i.e. assistive
technology, browser or operating system) is documented and
available in the product accessibility statement.
Where testers feel that problems are due to issues with the
browser (e.g. non-conformance with UAAG), then work-arounds should
be considered and information about these should be documented in
the accessibility statement.
Minimum level of testing
The choice of technologies and settings that organizations use
to test products should be documented. As a minimum, testing should
- Screen reader
- Screen magnifier
- Commonly used browser/OS settings for text resizing.
- Keyboard-only testing
Desired level of testing
Testing with every combination of browser/OS, and assistive
technology defined in the product's accessibility policy and the
keyboard alone. This helps to ensure that the web product is
accessible to people who have multiple impairments, or have a
particular browsing preference.
The extent of testing needs to be documented in the products
Expert review (8.4.5)
The expert review provides an extensive and reliable testing
method for discovering and advising on the resolution of
accessibility issues. It is more reliable than WCAG 2.0 testing and
should be carried out by an accessibility expert.
Expert reviews are particularly useful in early design stages
where prototypes may present difficulties for disabled users and
users of assistive technologies (e.g. where a product is not fully
developed and not supporting the features that a user may need to
We note that a combination of expert review and user testing
makes for a very comprehensive testing approach.
BS 8878 identifies different levels of expert review:
- Heuristic testing using WCAG 2.0 as a base plus known issues
that the expert has become aware of.
- Cognitive walkthrough where goals and tasks are attempted using
assistive technologies and alternative browser settings and access
methods such as keyboard-only.
The choice and level of testing should be documented in the
product accessibility policy. Again, testers should be
trained/competent in the use of the technology they use for
User testing with people with disabilities (8.4.6)
The main purpose of user testing with people with disabilities
is to assess their ability to complete goals and tasks outlined in
the product accessibility policy (link int).
If an organization chooses to use user testing, the test group
should represent a range of users with disabilities and older
people in accordance with the product's accessibility test
plan (link int)
BS 8878 notes that user testing should be used when the product
is nearing launch; ideally through a number of iterations as faults
will need to be corrected and then re-tested.
The organization needs to justify and documented in the
product's accessibility statement:
- Information about each user's requirements and their chosen
- The choice and number of users involved in testing along with
justification for how many people were chosen to represent each
user group (sample size).
- Ideally the group should include people of mixed abilities
(e.g. beginner, Intermediate, experienced with their
- The choice of whether to involve a specially trained evaluator
to gather the information from users
If a specially trained evaluator is not involved then
organizations need to be mindful that:
- Evaluators/testers who gather the data from end users do not
"lead" or influence the users. For example, if a user is stuck
while trying to complete a task then this difficulty in completing
the task needs to be documented along with the level of help or
hints that were given to the user before they could continue.
- BS 8878 states that organizations should "ensure that all user
testing conforms to BS EN ISO 9241-210 and all testers are aware of
best practice Codes of Conduct for testing".
Automated testing (8.4.7)
Automated testing tools can be extremely effective at finding
technical issues that may cause a website to be inaccessible. Their
advantage is in their ability to quickly detect potential issues on
a page or set of pages. These are powerful tools that can be used
to assist a developer or professional technical auditor in finding
issues with a website. These tools can greatly reduce the work
involved in pinpointing specific issues with websites.
Organizations need to be aware that test results from automated
tools are limited in that they are based on algorithms and code
analysis, they are not a substitute for manual testing or expert
review. For example, a tool can detect whether an image in a HTML
document has an alt attribute (text equivalent), but not whether it
is actually an accurate or functional attribute for the image. In
addition, automated tools frequently report false positive results
i.e. reporting issues that aren't there, and developer time may be
wasted fixing issues that have been falsely reported.
Automated tools are most useful for detecting issues with WCAG
2.0 criteria that can be consistently identified programmatically
(such as whether attributes are present in code), especially for
periodically analysing whole sites, and supplementing manual
BS 8878 notes the limitation of automated testing which it
states should be used to test the products page templates near
completion before they are used in the product, testing post-launch
(periodic maintenance). It also notes that organizations should be
"only a minority of WCAG criteria can be programmatically
verified, so automated testing on its own is not sufficient to
check conformity against WCAG or be assured that their products are
List of accessibility testing tools on GAWDS (this link will open in a new browser
Post launch testing (8.5)
Following launch, organizations need to be confident that their
product remains accessible in accordance with the product's
accessibility policy. Events that determine the timing and extent
of testing include:
- any update to the product where the level of testing will
depend on the extent of the update
- small changes (such as adding an image or paragraph) should be
tested against WCAG 2.0
- Large changes that may have an impact on the way that the
product is used (e.g. adding a new home page template or form)
should be tested by people with disabilities
- the emergence of new assistive technologies, or updates or new
versions of existing technologies.
- review of feedback that has been submitted by users and
including any relevant assistive technologies.
- annual testing of the product which should be as extensive as
BS 8878 notes that organizations may want to periodically review
any new technologies such as a new Operating System that may impact
the use of the product.
8.5 Post-launch programme of accessibility testing
Organizations should develop a regular programme of
accessibility testing after the web product is launched to maintain
the degree of user-experience specified in the product's
This programme of testing should include:
- testing the accessibility of all updates to the product
(whether as minor as an update to a page, or as major as a new
release of the product);
- small changes, such as adding a new graphic, writing a new
paragraph or changing a form should be tested, at a minimum, for
conformity to WCAG;
- large changes that affect important tasks within the interface
(e.g. how a user logs onto a site or buys a product) should ideally
undergo user testing with disabled and older people;
- testing the product with any new assistive technologies, or new
versions of existing assistive technologies, which are launched
after the web product is launched;
- reviewing feedback provided by the product's users; and
- annually benchmarking the site against the accessibility policy
by running user evaluation or conformity inspections to identify
any new accessibility problems.
Organizations might also periodically review any new
technologies, devices, user behaviours or expectations that would
change disabled and older people's accessibility requirements of
the web product.