weeks[9].toString(); // More Accessibility Info

Ok, let’s see what I did this week. 🙂

Beside of what I presented in the post from yesterday I also made the status bar work properly for the next type of elements:

– Time signature

– Markers

– Jumps

– Measures

– Accidentals

– Articulations

– Dynamics

– Rest

– Repeat Measures

I also implemented a system that tells the ScoreAccessibility class that the the current information has changed. For those of you who don’t want to read the long version in the previous post, here is the short one. 🙂

Bassically is an Observer pattern implemented with signals and slots. So the ScoreAccessibilityInfo has a slot that updates the status bar and the ElementAccessibilityInfo and Element each have a signal for telling the the information has changed. The signal from Element is connected to the signal from ElementAccessibilityInfo which in the end is connected to the slot from ScoreAccessibilityInfo for the object the ocupies the status bar. The reason why I like very much the signal/slot system in combination with the Observer pattern is that you don’t have to check if there is something else connected to the other end. You just emit the signal and if there is a ElementAccessibilityInfo instantiated, it will propagate the signal and if it’s also connected to the slot, it updates the status bar, but like I said there is no checking required. I think that this is a very nice way of applying the “Tell, don’t ask” principle.

Also, I got in touch with one of the main developers for Accessibility from Qt, Frederik Gladhorn, and he was able to help me with most of the problems that I had. Now the code works in my testing project so I can start moving it in MuseScore. I wasn’t yet able to give “fake” focus to a QObject so that the screen-reader can view it, but once I do that I will unpload the project so that anyone can use it. There are two things that I would like to mention here, that are not specifically written in the Qt documentation:

– Most screen-readers ignore any update that is not about the widget that has focus

– Child(0) is not the object itself, it’s the first(if it exists) child widget

This week I’ll probably have working the screen-reader and the basic information collected about all the Element objects so I can start collecting extra info. 

As I discussed with Marc, the status bar will only have the basic information, but the screen-reader will actually tell more, like for example for a note: what annotations, articulations it has, if it has explicit accidentals, etc. We will provide all this info, but the user doesn’t need to listen to it every time, since moving to the next element makes the screen-reader tell the info of that one.

Thank you for reading!

See you next week! 🙂

Advertisements

Design of the score accessibility system in MuseScore

In this post I’m going to present in detail my idea for the system that is going to provide feedback for the score. Almost all of the following design is already implemented in the score_accessibility branch of my repository.

For this I’ve created three classes:

1. ScoreAccessibility

2. ElementAccessibilityInfo

3. AccessibleScoreView (not implemented yet)

A. Class description

1. ScoreAcessibility

Since there is only one status bar that needs to be updated and only one screen-reader at a time it makes sense that this class will have at most one instance. I chose to use the Singleton pattern instead of static functions because it makes more sense to me to have an object that represents a curent state than static members. Other advantages of using this pattern are the fact that the object can be accessed from any part of the code by calling ScoreAccessibility::instance and that object can be used as a parameter in functions later on if that is the case.

This class agregates the ElemenetAccessibilityInfo class and the AccessibleScoreView class.

This class exposes two methods and one slot:

– updateAccessibility(ElementAccessibilityInfo* e);

this method updates the status bar using the informations received from parameter “e” and when the the AccessibleScoreView will be implemented it will notify the screen-reader that the accessibility informations have changed.

– clearAccessibilityInfo()

for the moment this method only cleans the status bar.

– slot currentInfoChanged()

this slot updates the information of the current ElementAccessibilityInfo (the one that currently ocupies the status bar)

2. ElementAccessibilityInfo

Here there were two options:

i ) add a virtual method to Element class that will return a string with the accessible information

ii) create a new class that takes care of constructing the accessibility information for an Element

I chose the second one for the following reasons (advantages):

R1) (Single responsability principle) I didn’t want to overcrowd the Element hierarchy with methods that compute and update its accessibility information. By going with the second solution, I’m defering this responsability to the ElementAccessbilityInfo class and the Element class gets (idealy) only:

    – one member accInfo (a pointer to it’s ElementAccessibilityInfo object)

    – a virtual method ElementAccessibilityInfo* accessibilityInfo() that instantiates and returns the object

    – a signal that is emited when the internal information changes and the status bar might need to be updated

R2) Reduce the duplicated code. The initial idea was to not have duplicated code at all, the informations would be added using inheritance.

R3) The visual/conceptual way of looking at score elements doesn’t match always 1:1 to how the inheritance hierarchy is implemented for Elemenets. By creating a new class I’m able to create a new hierarchy.

(this example might debatable and I won’t inherit myself also NoteAccessibilityInfo from RestAccessibilityInfo, but it goes to show that, musically, things are a bit different from how they are programmed)

For example: a note can be viewed musically as a rest with a pitch – both have a duration, a measure, a beat, the rest has a reduced set of annotations that can be added to it, where as the note can have annotations, accidentals, articulations, grace notes, etc. If you look at things this way, the note can inherit from the rest class. But every note has a duration only musically, programatically, notes are grouped in chords and the chord has the duration. It makes sense to implement things the way they are now, but it doesn’t make sense to provide info about a note to the user without giving its duration.

R4) Saving memory (not implemented yet completly). It makes sense to me (at least now, maybe things will change as I implement further) that at every point, only one accessibility object/info needs to be instantiated, the one that curently occupies the status bar. At this point, the object that contains all the information is not created until the respective element is not selecte. ( but it is not destroyed once created, at the moment )

R5) At one point I might want to implement feedback for commands (and maybe other things) not just for elements. Then idealy I need a base class and I don’t see any way (conceptually speaking) to connect elements with commands, but I can see a base class AccessibilityInfo that is inherited by ElementAccessibilityInfo and CommandAccessibilityInfo

R6) Self mentaining code – As long as the getters in the interfaces of Element hierarchy classes remain the same, there is no reason to change any code in the ElemenetAccessibilityInfo hierarchy, even if the internal structure changes.

Disadvantages:

D1) R3) unfortunatelly goes both ways, so I can’t always do things as efficient as I want and without duplicating code, or finding a compromise solution.

D2) I might not always have getters for the information that I need

The ElementAccessibilityInfo class has only two public functions and a signal:

QList<QString> info() that returns a list with all the informations as QStrings

updateInfo() which updates the strings in the list returned by info

– (signal) infoChanged() this signal annouces that the strings have changed and you might want to call info() again.

ElementAccessibilityInfo can be see from the point of view of design patterns as a Template, because info calls updateInfo, which is virtual, since every derived class needs to update in its own way.

AccessibleScoreView

This class will inherit QAccessibleWidget (implementing in this way QAccessibleInterface) so that the screen-reader will see the informations that I will provide. By talking to Frederik Gladhorn, one of the main developers for accessibility from Qt, I understood that the idea with the status bar was not enough, because I can’t make the screen-reader tell those informations without giving focus to the status bar. Most screen-readers ignore any changes that happen to widgets that don’t have focus. Since the ScoreView is the widget that has focus when selecting different score elements, I chose to implement this class.

B. Workflow

1) the client (in my case the selectSingle function) calls the Element::accessibilityInfo method

2) the Element::accessibilityInfo method checks if Element has an instance of ElementAccessibilityInfo, or subclass, if not it creates one and it returns it.

3) The ElementAccessibilityInfo class constructs the necesary info in the constructor and connects its infoChanged signal with the parent’s accessibilityInfoChanged signal

4) the client gets the instance of ScoreAccessibility, by calling ScoreAccessibility::instance()

5) the client calls using instance from 4) the ScoreAccessibility::updateAccessibility method using the ElementAccessibilityInfo object that he obtained at 2) as parameter

6) ScoreAccessibility::updateAccessibility looks if the current given parameter is the object that it already had, if not, it dissconects its infoChanged slot from the signal of the former object, it connects the signal to the the new object, and it replaces the former object with the parameter. After that it calls ElementAccesibilityInfo::info and updates the status bar and the screen-reader.

7) Everytime the ElementAccessibilityInfo::infoChanged is emited, the ScoreAccessibility class calls updateAccessbility giving as a parameter the object that it already has.

Other advantage:

– Loose-coupling – as can be seen in the workflow the ScoreAccessibility class has no idea about the Element or ElementAccessibility hierarchy, or about what are those informations that it receives. This is a big plus if as I said earlier I want to implement feedback for commands. The only change that I would need to do is modify everywhere is ElementAccessibilityInfo to AccessibilityInfo (2 places 🙂 )

At first it seems like if I would’ve implemented everything directly in the Element I would’ve not needed the signals, but I would still need a system to tell the ScoreAccessibility that the information has changed and it needs to update itself. At most, the accessibilityInfoChanged signal would be directly connected to the currentInfoChanged slot. This way there is another signal involved, but it is connected and disconected in the constructor, respectively in the destructor of the ElementAccessibilityInfo, so it was taken care of and I see no reason in the future for someone to manually connect these 2 signals in any other way.

One of the reasons I wrote this post is so Marc Sabatella can see my full thinking process behind the code, otherwise it would have probably been included in the tommorow’s weekly update. I’m going to discuss these ideas with him and I will let you know if anything changes. Mostly we are going to talk about the disadvantages that I haven’t seen, because naturaly if I had seen them, I wouldn’t have chosen these solutions. 😀 Of course, if anybody else has a comment, or advice I’m happy to hear them all. 🙂

weeks[8].toString(); // Screen Reader and Accessibility Info

Hello everyone! 🙂

This week I continued my reasearch on how Qt intereacts with the screen-reader and the implementation of classes that update the status bar with the necesary informations.

1. Screen-reader

Unfortunately, here I’ve hit a dead end. I’ve succesfully implemented the QAccessibleInterface in order to provide custom accessibility info, by subclassing the QAccessibleWidget class and the QAccessibleInterface class, but I couldn’t make the screen-reader to update the informations using the QAccessible::updateAccessibility method.  I send an email on the Qt Accessibility mailing list, and asked on two forums, but no luck… In the end I’ve sent a bug report to Qt about this. For the moment I’m goind to put this aside until I here back from Qt Accessibility developers and I’m focusing on gathering all the necesary informations about the score elements so when I find out what needs to be done, everything will be in place. You can find the forum posts here[0][1] and the bug report here[2]. Also in the bug report I’ve sent my testing project so you can view there how to implement the QAccessibleInterface and if by any chance you manage to update the screen-reader feedback, please let me know. 🙂 I’ve set some debugging messages so you can see what informations Qt is querying, but it seems that either it doesn’t send them to the screen-reader, or the screen-reader ignores them.

 

2. Accessibility info

2.1 What I’ve done

I’ve implemented the apropriate subclasses of the ElementAccessibilityInfo that I’ve talked about for the next elements and they provide the respective info (for now):

– Note:

Note type, Pitch, Duration, Bar, Beat

– Cleff:

Cleff type, Bar

– Key signature (for classic key signatures)

Key signature, Key Signature type, Bar

– Barline

Barline, Barline type

Also, the information updates for consecutive selections.

2.2 Problems

No problems so far. I didn’t even had to break the encapsulation, by making the ElementAccessibilityInfo a friend class of Element as I was worried.

2.3 What else needs to be done

– Write the implementation for other element types. For now, the default behavior is to write in the status bar the element type.

– Update the status info when changing the element and the selection remains on it. For this I think I will use signals.

– Save memory space. Another advantage that I didn’t see right away is that I can keep only one ElementAccessibilityInfo object “alive”. The one that is currently in the status bar, the other ones can be deleted and reinstantiated when it is needed.

I’m very dissapointed that I’ve spent almost the whole week trying to make the screen-reader work with no luck, but hopefully the developers will be able to help me. Also, I think that by end of next week I will finish gathering the info about all the element types (at least the basic info).

Thank you! See you next week! 🙂

weeks[7].toString(); // Screen Reader feedback for score

First of all I want to appologise for being late with this weekly blog post. I just got back from camping last night and I didn’t have my laptop with me.

This week did a lot of reasearch on how to provide accessibility using the  screen reader for the score and I’ve started implementing the design for the it.

1. Reasearch

1.1 What I’ve done

– I’ve read the documentantion for the following classes: QAccessible, QAccessibleObject, QAccessibleInteface

– I’ve read some of the code from the Qt project regarding accessibility, especially for the QAccessible::updateAccessibility function

– I’ve created small programs in order to figure out what can be done and how the API works

1.2 Problems

– Unfortunatelly there is one major flaw in the idea with the status bar. The initial idea was to populate the status bar with a QLabel (or more QLabels) and make the screen reader tell the informations that are presented there. From what I can tell both by running tests and reading Qt’s code the updateAccessibility function only works when the widget for which is called has focus. Also, it is called internally by Qt when the QWidget::setFocus method is called. This means that in order for the screen reader to give the user all the informations, the status bar needs to keep the focus in that period of time. This is not an option, because the user could not would have to wait until all the information is provided and the score gets the focus again.

In order to solve this problem, I’m thinking about using the scoreview, or scoretab to place the information in their accessibleName/accessibleDescription. I need to do more tests in order to be sure.

– One thing that I don’t yet understand is how the QAccessibleObject works since a QObject that is not a QWidget cannot gain focus.

2. Design

2.1 What I’ve done

– I’ve created the ScoreAccessibility class that will manage and update all the informations using the method updateAccessibilityInfo. In the implementation of this class I used the Singleton pattern. In this way there is only one global instance for the aplication and I can access it from any object using the static method ScoreAccessibility::instance.

– As I said the ScoreAccessibility class will only manage the informations. These informations will be provided using another class, ElementAccessibilityInfo. The ElementAccessibilityInfo class will be inherited in order to get the appropriate informations for all the element types. ElementAccessibilityInfo keeps a private list with QStrings(or QLabels, or another custom class, I haven’t decided yet) and it will be populated by subclasses with the apropriate informations and this list is what the ScoreAccessibilityInfo will use.

I could’ve just added the QList to the Element class, but I think that these solution is better from a design point of view, by deferring responsability through agregation and by keeping the entropy.

2.2 Problems

– I might need to break the encapsulation by making the ElementAccessibilityInfo a friend class of Element if at some point I need to access protected, or private members

 

I’ve made this design work for the Note element and for now it works well, but I have to implement more in order to see if it holds for every element type, so things might change along the road. Also there is still that major problem of finding a place to put the informations so that the screen-reader can access them.

Because I was away for a few days I will work more this week in order to recover the lost time. 🙂

That’s all for this week!

Thank you!

weeks[6].toString(); // Navigation and Screen Reader feedback

Ok, lets see what I’ve done this week. 🙂

1. Navigation
First of all I finished every aspect of the next/prev element command, so the only thing that remains to be done for navigation is the next annotation/articulation command, but as discussed in the previous post I will implement this after the screen reader feedback.

1.1 What I’ve done

– Fixed all the problems with the barlines. I know I said last week that I only have to test the navigation for barlines, but when I did, I discovered that it didn’t cover all the use cases, so I had to find a more general implementation. Now everything works as expected

– Refactored the functions. There was a lot of duplicated code in two huge functions, so I moved some of it in the segment object, creating three smaller functions and now the code is much more easier to read and understand.

– Made the canvas adjust its position during the navigation

– Fixed the coding style

1.2 What else needs to be done

– As I said, the only thing that needs to be implemented is the next annotation/articulation command.

 

2. Screen reader feedback

As I mentioned in the proposal a very important part of my project is making the screen-reader tell the score to visual impaired musicians. After talking to Marc I think I found a good solution in order to do this. First of all I will implement an accessibility interface for the status bar using the accessibility API provided by Qt like this example [0] and after that I will make the select command populate the status bar with all the informations that the user needs.

For example:

“Select note 1 of 3 Pitch: C4 Duration: 8th note Bar: 4 Beat: 1”

I’ve started doing the reasearch for how to implement this, I’m playing with the above example and reading the documentations provided by Qt for the accessibility API for QAccessibleInterface, QAccessibleObject and QAccessibleWidget, but I don’t have yet anything functional. Hopefully, if everything works as planned I should have the status bar working by the end of the week.

3.  Bug fix

– I also fixed this bug: http://musescore.org/en/node/24243

-See Pull Request here:  https://github.com/musescore/MuseScore/pull/1027

See you next week! 🙂

[0] http://www.ranorex.com/blog/enabling-automation-for-custom-qt-widgets-by-adding-accessibility