Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wire up accessibility bridge in Windows #77838

Closed
chunhtai opened this issue Mar 10, 2021 · 4 comments
Closed

Wire up accessibility bridge in Windows #77838

chunhtai opened this issue Mar 10, 2021 · 4 comments
Assignees
Labels
a: accessibility Accessibility, e.g. VoiceOver or TalkBack. (aka a11y) engine flutter/engine repository. See also e: labels. P2 Important issues not at the top of the work list platform-windows Building on or for Windows specifically

Comments

@chunhtai
Copy link
Contributor

chunhtai commented Mar 10, 2021

Make windows use the accessibility library to support the screen reader.

This issue is blocked on #77836

Updating to UI Automation APIs is covered in #94782

The Windows UWP analogue of this issue is #93928.

@chunhtai chunhtai added engine flutter/engine repository. See also e: labels. a: accessibility Accessibility, e.g. VoiceOver or TalkBack. (aka a11y) platform-windows Building on or for Windows specifically labels Mar 10, 2021
@chinmaygarde chinmaygarde added the P2 Important issues not at the top of the work list label Mar 15, 2021
@cbracken cbracken changed the title Wires up accessibility bridge in Windows Wire up accessibility bridge in Windows Oct 18, 2021
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 20, 2021
Windows does not provide a system notification the embedder can
subscribe to to determine when a screen reader or other assistive
technology that requires the semantics tree has been enabled. [1]

In the absence of such a notification, we watch for queries for the
IAccessible COM object representing the root node of the semantics tree,
and on receipt of such queries we enable the semantics tree. For the
time being, we assume that if accessiblity is enabled, it will be
enabled for the duration of the application lifetime. In a future patch
we can optimise this by adopting the approach taken by Chromium, which
is to disable semantics if we haven't received a query within some
reasonable timeout.

This patch enables Flutter's semantics tree; a follow-up patch wires in
the AccessibilityBridge which maintains the AXTree of COM objects that
is the Windows representation of the Flutter SemanticsTree.

Issue: flutter/flutter#77838

[1]: https://docs.microsoft.com/en-us/windows/win32/winauto/screen-reader-parameter
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 21, 2021
Adds method for dispatching a semantics action received from the OS to
the matching node in Flutter's semantics tree.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 21, 2021
Adds method for dispatching a semantics action received from the OS to
the matching node in Flutter's semantics tree.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 21, 2021
Adds method for dispatching a semantics action received from the OS to
the matching node in Flutter's semantics tree.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 21, 2021
Adds method for dispatching a semantics action received from the OS to
the matching node in Flutter's semantics tree.

Issue: flutter/flutter#77838
cbracken added a commit to flutter/engine that referenced this issue Oct 21, 2021
Adds method for dispatching a semantics action received from the OS to
the matching node in Flutter's semantics tree.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 21, 2021
Adds method for dispatching a semantics action received from the OS to
the matching node in Flutter's semantics tree.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 21, 2021
Windows does not provide a system notification the embedder can
subscribe to to determine when a screen reader or other assistive
technology that requires the semantics tree has been enabled. [1]

In the absence of such a notification, we watch for queries for the
IAccessible COM object representing the root node of the semantics tree,
and on receipt of such queries we enable the semantics tree. For the
time being, we assume that if accessiblity is enabled, it will be
enabled for the duration of the application lifetime. In a future patch
we can optimise this by adopting the approach taken by Chromium, which
is to disable semantics if we haven't received a query within some
reasonable timeout.

This patch enables Flutter's semantics tree; a follow-up patch wires in
the AccessibilityBridge which maintains the AXTree of COM objects that
is the Windows representation of the Flutter SemanticsTree.

Issue: flutter/flutter#77838

[1]: https://docs.microsoft.com/en-us/windows/win32/winauto/screen-reader-parameter
cbracken added a commit to flutter/engine that referenced this issue Oct 21, 2021
Windows does not provide a system notification the embedder can
subscribe to to determine when a screen reader or other assistive
technology that requires the semantics tree has been enabled. [1]

In the absence of such a notification, we watch for queries for the
IAccessible COM object representing the root node of the semantics tree,
and on receipt of such queries we enable the semantics tree. For the
time being, we assume that if accessiblity is enabled, it will be
enabled for the duration of the application lifetime. In a future patch
we can optimise this by adopting the approach taken by Chromium, which
is to disable semantics if we haven't received a query within some
reasonable timeout.

This patch enables Flutter's semantics tree; a follow-up patch wires in
the AccessibilityBridge which maintains the AXTree of COM objects that
is the Windows representation of the Flutter SemanticsTree.

Issue: flutter/flutter#77838

[1]: https://docs.microsoft.com/en-us/windows/win32/winauto/screen-reader-parameter
kylinchen pushed a commit to XianyuTech/engine that referenced this issue Oct 22, 2021
Adds method for dispatching a semantics action received from the OS to
the matching node in Flutter's semantics tree.

Issue: flutter/flutter#77838
kylinchen pushed a commit to XianyuTech/engine that referenced this issue Oct 22, 2021
Windows does not provide a system notification the embedder can
subscribe to to determine when a screen reader or other assistive
technology that requires the semantics tree has been enabled. [1]

In the absence of such a notification, we watch for queries for the
IAccessible COM object representing the root node of the semantics tree,
and on receipt of such queries we enable the semantics tree. For the
time being, we assume that if accessiblity is enabled, it will be
enabled for the duration of the application lifetime. In a future patch
we can optimise this by adopting the approach taken by Chromium, which
is to disable semantics if we haven't received a query within some
reasonable timeout.

This patch enables Flutter's semantics tree; a follow-up patch wires in
the AccessibilityBridge which maintains the AXTree of COM objects that
is the Windows representation of the Flutter SemanticsTree.

Issue: flutter/flutter#77838

[1]: https://docs.microsoft.com/en-us/windows/win32/winauto/screen-reader-parameter
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 19, 2021
This is the third in a series of patches adding accessibility support
for the Windows embedder. This patch wires in the Accessibility bridge,
and lands the core structure of the Windows FlutterPlatformNodeDelegate
and AccessibilityBridgeDelegate classes, including:

* Instantiating the AccessibilityBridge when the semantics tree is
  enabled.
* Creating FlutterPlatformNodeDelegate wrappers for semantics tree
  nodes.
* Building and updating the accessibility tree on semantics updates.
* Returning the native IAccessible objects when queried.

Breaking this out so that the follow-up patches can be reviewed and
landed in smaller, independent chunks.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 19, 2021
This is the third in a series of patches adding accessibility support
for the Windows embedder. This patch wires in the Accessibility bridge,
and lands the core structure of the Windows FlutterPlatformNodeDelegate
and AccessibilityBridgeDelegate classes, including:

* Instantiating the AccessibilityBridge when the semantics tree is
  enabled.
* Creating FlutterPlatformNodeDelegate wrappers for semantics tree
  nodes.
* Building and updating the accessibility tree on semantics updates.
* Returning the native IAccessible objects when queried.

Breaking this out so that the follow-up patches can be reviewed and
landed in smaller, independent chunks.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 19, 2021
This is the third in a series of patches adding accessibility support
for the Windows embedder. This patch wires in the Accessibility bridge,
and lands the core structure of the Windows FlutterPlatformNodeDelegate
and AccessibilityBridgeDelegate classes, including:

* Instantiating the AccessibilityBridge when the semantics tree is
  enabled.
* Creating FlutterPlatformNodeDelegate wrappers for semantics tree
  nodes.
* Building and updating the accessibility tree on semantics updates.
* Returning the native IAccessible objects when queried.

Breaking this out so that the follow-up patches can be reviewed and
landed in smaller, independent chunks.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 19, 2021
This is the third in a series of patches adding accessibility support
for the Windows embedder. This patch wires in the Accessibility bridge,
and lands the core structure of the Windows FlutterPlatformNodeDelegate
and AccessibilityBridgeDelegate classes, including:

* Instantiating the AccessibilityBridge when the semantics tree is
  enabled.
* Creating FlutterPlatformNodeDelegate wrappers for semantics tree
  nodes.
* Building and updating the accessibility tree on semantics updates.
* Returning the native IAccessible objects when queried.

Breaking this out so that the follow-up patches can be reviewed and
landed in smaller, independent chunks.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Dec 10, 2021
This enables unittests for the accessibility bridge in
common_cpp_unittests, when running on Windows. Previously, we only
tested on macOS.

Issue: flutter/flutter#77838
cbracken added a commit to flutter/engine that referenced this issue Dec 10, 2021
In the Win32 accessibility tree, each AXTree node has an associated
IAccessible object. In WindowWin32, our WM_GETOBJECT handler returns the
IAccessible associated with the root node of the tree (node 0). On other
platforms, we often add our root accessibility object as a subnode of
some existing accessibility object associated with the view. On Windows,
the root IAccessible _is_ the accessibility object associated with the
view (on Windows, and HWND).

In the previous implementation, AccessibleObjectFromWindow actually just
returns the root IAccessible object, which is equivalent to just
returning GetNativeViewAccessible. Instead, we just return null once we
hit the root of the tree.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Dec 10, 2021
The default implementation of GetUniqueId on ui::AXPlatformNodeDelegate
always returns ID 1. We had previously implemented this on the windows
platform node delegate, but for consistency's sake, and because the
default implementation is surprising, we're promoting this to the
FlutterPlatformNodeDelegate base class.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Dec 10, 2021
The default implementation of GetUniqueId on ui::AXPlatformNodeDelegate
always returns ID 1. We had previously implemented this on the windows
platform node delegate, but for consistency's sake, and because the
default implementation is surprising, we're promoting this to the
FlutterPlatformNodeDelegate base class.

Issue: flutter/flutter#77838
cbracken added a commit to flutter/engine that referenced this issue Dec 10, 2021
TestAccessibilityBridgeDelegate::accessibility_events previously held
values of type ui::AXEventGenerator::TargetedEvent. TargetedEvent
contains an AXNode pointer and a const reference to a
ui::AXEventGenerator::EventParams object, and as such it's unsafe to
make or read copies of TargetedEvent values outside the scope of the
AccessibilityBridgeDelegate::OnAccessibilityEvent callback.

In this patch, we update the accessibility_events vector to simply hold
EventType values since this is the only part of the value we use in our
existing tests. If in future we need the full TargetedEvent, we'll need
to properly copy these values.

This patch also fixes a typo in the accessibility_events identifier and
converts an EXPECT_EQ to an ASSERT_EQ in a case where the following
test expectations are meaningless/could crash if the
accessibility_events size isn't as expected.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Dec 10, 2021
This enables unittests for the accessibility bridge in
common_cpp_unittests, when running on Windows. Previously, we only
tested on macOS.

Issue: flutter/flutter#77838
cbracken added a commit to flutter/engine that referenced this issue Dec 10, 2021
This enables unittests for the accessibility bridge in
common_cpp_unittests, when running on Windows. Previously, we only
tested on macOS.

Issue: flutter/flutter#77838
cbracken added a commit to flutter/engine that referenced this issue Dec 10, 2021
The default implementation of GetUniqueId on ui::AXPlatformNodeDelegate
always returns ID 1. We had previously implemented this on the windows
platform node delegate, but for consistency's sake, and because the
default implementation is surprising, we're promoting this to the
FlutterPlatformNodeDelegate base class.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Dec 13, 2021
This adds support for toggle switches such as the material Switch and
CupertinoSwitch widgets. Previously, we added support for checkboxes on
desktop platforms, but not for toggle switches.

Issue: flutter/flutter#77838
@voipworld
Copy link

hello,
is there anything to do from my side to support windows A11I?
building the default flutter project for win wouldn't be accessible.

cbracken added a commit to cbracken/flutter_engine that referenced this issue Dec 14, 2021
Adds support to the common desktop accessibility bridge for widgets that
include a semantics node with the `isSlider` flag set.

Issue: flutter/flutter#77838
cbracken added a commit to flutter/engine that referenced this issue Dec 14, 2021
Adds support to the common desktop accessibility bridge for widgets that
include a semantics node with the `isSlider` flag set.

Issue: flutter/flutter#77838
@cbracken
Copy link
Member

@voipworld you'll need to be building off the Flutter master branch to pick up all the recent changes that have landed to support Windows accessibility.

@cbracken
Copy link
Member

Since this issue is a very generic umbrella issue that focuses on getting the basic wiring in for Windows accessibility support, I'm going to close this as resolved. The cross-platform accessibility bridge in //flutter/shell/platform/common/accessibility_bridge.h is wired into the Windows embedder and dozens of smaller accessibility-related patches have landed to enable toggle switches, sliders, etc.

All future accessibility work should be filed as individual bugs covering each specific issue and include repro steps for actual vs expected behaviour. I've linked a couple above; specifically migrating to UI Automation (UIA) APIs in #94782.

cbracken added a commit to cbracken/flutter that referenced this issue Dec 15, 2021
Microsoft Active Accessibility (MSAA) does not include
increment/decrement keyboard shortcuts for manipulating sliders and
other similar controls. To make up for this, we give the slider keyboard
focus when it gains accessibility focus so that the user can use the
arrow keys to manipulate the slider.

Issue: flutter#77838
cbracken added a commit that referenced this issue Dec 15, 2021
Microsoft Active Accessibility (MSAA) does not include
increment/decrement keyboard shortcuts for manipulating sliders and
other similar controls. To make up for this, we give the slider keyboard
focus when it gains accessibility focus so that the user can use the
arrow keys to manipulate the slider.

Issue: #77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Dec 16, 2021
IAccessible objects should implement accHitTest. Our implementation, in
AXPlatformNodeWin, delegates hit testing to
AXPlatformNodeDelegate::HitTestSync. Here, we do a quick recursive
depth-first walk of the accessibility tree to return the best match.

We define the best match as the deepest node in the tree that contains
the point under test whose children do not.

This also adds a ToScreenPoint() method to FlutterWindowWin32 which
avoids a raw Win32 API call, which facilitates mocking out the behaviour
of this method in tests. Since we don't have a valid HWND in unittests,
ClientToScreen() always returns the point 0,0.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Dec 16, 2021
IAccessible objects should implement accHitTest. Our implementation, in
AXPlatformNodeWin, delegates hit testing to
AXPlatformNodeDelegate::HitTestSync. Here, we do a quick recursive
depth-first walk of the accessibility tree to return the best match.

We define the best match as the deepest node in the tree that contains
the point under test whose children do not.

Issue: flutter/flutter#77838
cbracken added a commit to cbracken/flutter_engine that referenced this issue Dec 16, 2021
IAccessible objects should implement accHitTest. Our implementation, in
AXPlatformNodeWin, delegates hit testing to
AXPlatformNodeDelegate::HitTestSync. Here, we do a quick recursive
depth-first walk of the accessibility tree to return the best match.

We define the best match as the deepest node in the tree that contains
the point under test whose children do not.

Issue: flutter/flutter#77838
cbracken added a commit to flutter/engine that referenced this issue Dec 16, 2021
IAccessible objects should implement accHitTest. Our implementation, in
AXPlatformNodeWin, delegates hit testing to
AXPlatformNodeDelegate::HitTestSync. Here, we do a quick recursive
depth-first walk of the accessibility tree to return the best match.

We define the best match as the deepest node in the tree that contains
the point under test whose children do not.

Issue: flutter/flutter#77838
@github-actions
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 28, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
a: accessibility Accessibility, e.g. VoiceOver or TalkBack. (aka a11y) engine flutter/engine repository. See also e: labels. P2 Important issues not at the top of the work list platform-windows Building on or for Windows specifically
Projects
None yet
Development

No branches or pull requests

5 participants