Access 8878: Web Accessibility - Code of Practice

Get BS 8878BSI Logo (this link will open in a new browser window)

Methods for testing accessibility

To have confidence in the accessibility of your web product, you'll need to use a number of different testing methods at key stages throughout the product lifecycle.

Essential accessibility testing

BS 8878 describes a combination of methods it regards as a "minimum".

  • Validation testing of code (8.4.2)
  • Manual testing against WCAG 2.0 (and/or ATAG if applicable) (8.4.3)
  • Assistive technology testing and adjusting browser/OS settings that people with disabilities may use. (8.4.4)

Note that the above testing method does not necessarily involve people with disabilities, so if this is the full extent of the testing you must ensure that your site features a simple feedback mechanism that is easy to find and use and that allows users with disabilities to involve real users in your web product lifecycle.

Note also that this type of testing is limited by the tester's knowledge of the WCAG Guidelines, and of the needs of disabled web users.

BS 8878 notes that the best way for a product to comply with the Equality Act (link ext http://www.equalities.gov.uk/equality_act_2010.aspx (this link will open in a new browser window)), and the DDA (link ext http://www.dwp.gov.uk/employer/disability-discrimination-act/ (this link will open in a new browser window)) is to carry out user testing with people with disabilities.

Access 8878 Services

Higher levels of testing (expert review and user testing)

BS 8878 strongly recommends using the following methods to test the usability and technical accessibility of the web product more extensively. Testing should be carried out on early designs, prototypes and finished code.

  • Expert reviews(link expert reviews 8.4.5) should be carried out by agencies familiar with the technology needs of people with disabilities and web product best practices to assess and identify potential issues with the accessibility of early designs and prototypes.
  • User testing (link user testing 8.4.6) with disabled and older users who are representative of the web product (link 6.7 user goals). To asses the usability of the product.

Access 8878 Services

Periodic testing

As part of the web product's post-launch accessibility maintenance, organizations should conduct testing using a combination of automated tools (link int) and markup validation (link int). The purpose of this testing is to make sure that the accessibility of the product has not been compromised by work carried out on the site post-launch.

This type of testing is low-cost and intended to provide a type of alert to issues that may have been introduced at some point, but should not be considered as a replacement for more comprehensive testing techniques.

Manual testing for conformance to WCAG 2.0 and ATAG (8.4.3)

Claims of WCAG 2.0 conformance require each page of a web product to be tested against a set of guidelines.

BS 8878 notes that It is important that this type of testing is carried out by testers who can demonstrate a good understanding of the success criteria for WCAG 2.0 and ATAG guidelines (where applicable).

Access 8878 would add that an understanding of specific user needs and the operation of assistive technologies is also very important in order to properly understand the guidelines.

We would also add that claims of conformance to WCAG 2.0 guidelines based on testing a number of representative sample pages should be duly noted in any accessibility statement, and a list of pages that formed the sample should also be provided.

Access 8878 Services

Testing using assistive technologies and browser/OS settings commonly used by disabled users (8.4.4)

Testing should be carried out by testers who are trained / competent in the use of these technologies so that they are using them in a similar way to people who uses those technologies all of the time.

The level of testing and the technologies used (i.e. assistive technology, browser or operating system) is documented and available in the product accessibility statement.

Where testers feel that problems are due to issues with the browser (e.g. non-conformance with UAAG), then work-arounds should be considered and information about these should be documented in the accessibility statement.

Access 8878 Services

Minimum level of testing

The choice of technologies and settings that organizations use to test products should be documented. As a minimum, testing should include:

  • Screen reader
  • Screen magnifier
  • Commonly used browser/OS settings for text resizing.
  • Keyboard-only testing

Desired level of testing

Testing with every combination of browser/OS, and assistive technology defined in the product's accessibility policy and the keyboard alone. This helps to ensure that the web product is accessible to people who have multiple impairments, or have a particular browsing preference.

The extent of testing needs to be documented in the products accessibility policy

Expert review (8.4.5)

The expert review provides an extensive and reliable testing method for discovering and advising on the resolution of accessibility issues. It is more reliable than WCAG 2.0 testing and should be carried out by an accessibility expert.

Expert reviews are particularly useful in early design stages where prototypes may present difficulties for disabled users and users of assistive technologies (e.g. where a product is not fully developed and not supporting the features that a user may need to test accurately).

We note that a combination of expert review and user testing makes for a very comprehensive testing approach.

BS 8878 identifies different levels of expert review:

  • Heuristic testing using WCAG 2.0 as a base plus known issues that the expert has become aware of.
  • Cognitive walkthrough where goals and tasks are attempted using assistive technologies and alternative browser settings and access methods such as keyboard-only.

The choice and level of testing should be documented in the product accessibility policy. Again, testers should be trained/competent in the use of the technology they use for testing.

Access 8878 Services

User testing with people with disabilities (8.4.6)

The main purpose of user testing with people with disabilities is to assess their ability to complete goals and tasks outlined in the product accessibility policy (link int).

If an organization chooses to use user testing, the test group should represent a range of users with disabilities and older people in accordance with the product's accessibility test plan (link int)

BS 8878 notes that user testing should be used when the product is nearing launch; ideally through a number of iterations as faults will need to be corrected and then re-tested.

The organization needs to justify and documented in the product's accessibility statement:

  • Information about each user's requirements and their chosen technologies
  • The choice and number of users involved in testing along with justification for how many people were chosen to represent each user group (sample size).
  • Ideally the group should include people of mixed abilities (e.g. beginner, Intermediate, experienced with their browser/OS/assistive technology).
  • The choice of whether to involve a specially trained evaluator to gather the information from users

If a specially trained evaluator is not involved then organizations need to be mindful that:

  • Evaluators/testers who gather the data from end users do not "lead" or influence the users. For example, if a user is stuck while trying to complete a task then this difficulty in completing the task needs to be documented along with the level of help or hints that were given to the user before they could continue.
  • BS 8878 states that organizations should "ensure that all user testing conforms to BS EN ISO 9241-210 and all testers are aware of best practice Codes of Conduct for testing".

Access 8878 Services

Automated testing (8.4.7)

Automated testing tools can be extremely effective at finding technical issues that may cause a website to be inaccessible. Their advantage is in their ability to quickly detect potential issues on a page or set of pages. These are powerful tools that can be used to assist a developer or professional technical auditor in finding issues with a website. These tools can greatly reduce the work involved in pinpointing specific issues with websites.

Organizations need to be aware that test results from automated tools are limited in that they are based on algorithms and code analysis, they are not a substitute for manual testing or expert review. For example, a tool can detect whether an image in a HTML document has an alt attribute (text equivalent), but not whether it is actually an accurate or functional attribute for the image. In addition, automated tools frequently report false positive results i.e. reporting issues that aren't there, and developer time may be wasted fixing issues that have been falsely reported.

Automated tools are most useful for detecting issues with WCAG 2.0 criteria that can be consistently identified programmatically (such as whether attributes are present in code), especially for periodically analysing whole sites, and supplementing manual testing.

BS 8878 notes the limitation of automated testing which it states should be used to test the products page templates near completion before they are used in the product, testing post-launch (periodic maintenance). It also notes that organizations should be aware that

"only a minority of WCAG criteria can be programmatically verified, so automated testing on its own is not sufficient to check conformity against WCAG or be assured that their products are accessible."

List of accessibility testing tools on GAWDS (this link will open in a new browser window)

Post launch testing (8.5)

Following launch, organizations need to be confident that their product remains accessible in accordance with the product's accessibility policy. Events that determine the timing and extent of testing include:

  • any update to the product where the level of testing will depend on the extent of the update
  • small changes (such as adding an image or paragraph) should be tested against WCAG 2.0
  • Large changes that may have an impact on the way that the product is used (e.g. adding a new home page template or form) should be tested by people with disabilities
  • the emergence of new assistive technologies, or updates or new versions of existing technologies.
  • review of feedback that has been submitted by users and including any relevant assistive technologies.
  • annual testing of the product which should be as extensive as possible.

BS 8878 notes that organizations may want to periodically review any new technologies such as a new Operating System that may impact the use of the product.

8.5 Post-launch programme of accessibility testing

Organizations should develop a regular programme of accessibility testing after the web product is launched to maintain the degree of user-experience specified in the product's accessibility policy.

This programme of testing should include:

  • testing the accessibility of all updates to the product (whether as minor as an update to a page, or as major as a new release of the product);
  • small changes, such as adding a new graphic, writing a new paragraph or changing a form should be tested, at a minimum, for conformity to WCAG;
  • large changes that affect important tasks within the interface (e.g. how a user logs onto a site or buys a product) should ideally undergo user testing with disabled and older people;
  • testing the product with any new assistive technologies, or new versions of existing assistive technologies, which are launched after the web product is launched;
  • reviewing feedback provided by the product's users; and
  • annually benchmarking the site against the accessibility policy by running user evaluation or conformity inspections to identify any new accessibility problems.

Organizations might also periodically review any new technologies, devices, user behaviours or expectations that would change disabled and older people's accessibility requirements of the web product.

Access 8878 Services

Access 8878 is the leading resource for BS 8878

Web Accessibility – Code of Practice Access 8878 is the leading resource for BS 8878 (this link will open in a new browser window)Cookies Policy