User Evaluation Part 1- University of Sri Jayawardhanapura

The main evaluation process of our concept, initiated at University of Sri Jayawardhanapura, yesterday. According to the plan the system would be introduced to 5 students , completely and partially blind per day, for two days. The evaluation happens in qualitative and quantitative manner. The quantitative evaluation is handled by introducing two systems with similar functionality, one with positional layout information and the second one without it. The process is expected to show some distinct time, error rates and request for help rates. The non positional system is completely linear: uses the top and down arrow keys, and has no positional information involved. The other system is composed of the qualities we have described earlier.

The evaluation began at 9 45 A.M at the blind personal's computer lab in the university. There was a computer booth with a simple computer, which served our purpose very well; and least disturbed other students. This was converted into our experiment machine

figure 1

The setup is shown in figure 1, with the original pc on left and the environment setup by us in the left. Two machine towers, speaker boxes and another box were used to elevate the top speakers approximately giving similar distance to the top and the bottom rows. The right side speaker tended to fall due to the weight of the hanging wire, so two stones were dropped into the speaker box, so that it would be balanced. The bottom speakers were kept on PC's original system tower and the computer table. We took a photograph of the location before changing and the booth was prapared exactly as it was, according to the image taken. This is important because blind people find their stuff imagining that they had a exact location, or that they were in the same place they left them. Ex the head set should be always kept on top of the system unit.
Exact information regarding the evaluation is unavailable with me as documents, so I will note down an overall idea of the process. We introduced the system to three male and two female students. One male student was completely blind while the other two were partially blind. One female student was blind and other was partially blind. The second female student mentioned has lost here sight recently. With out evaluating the documented data I feel that there is no real advantage regarding speed and efficiency in introducing the positional system. The experiment had so many limitations in selecting the sample. Some students had extraordinary capabilities. They showed lower time in using the positional or the non positional system. Since the sample size available is limited, there issues are unavoidable.
Most students identified the positioning of the objects in space. Every one exept one said that this was an important addition. (It was raining out side when he used the system, so we are not sure he clearly felt the positioning and layout information). Almost every one under stood the table layout structure and the comparison structure of data. Most of them used the term table or "වගුව" to describe what they heard (a table exactly). Overall in my openion the concept is worth enough to be investigated further. There are limitations in sound; but similarly there are limitations in vision as well. It is impossible to read a book or a slide when the line are cramped, just as the sounds. The evaluation ended at around 3 30 in the evening. And we hope to complete our sample of 10 tomorrow at the same place at the same time.

User Evaluation - Session 1

Now it's time to test our system in a proper manner. We have practices to conduct our evaluation in previous pilots. And we have developed our application according to the prior feedback as well.

We have managed to gather 10 participants from University of Sri Jayawardenapura (USJP) with the help of Mr. Chanaka and Mr. Kasun Nayanajith. We have divided the evaluation in to two days (27th and 29th April 2011), thus five participants per day. We were lucky to acquire a sound proof room which is used for sound recordings.

There were 2 female and one 3 male participants. Among the female participants 1 was totally blind and the other partially blind. Among the male ones 1 was totally blind and 2 were partially blind.




We managed to set-up the evaluation environment by 9.30 a.m. on 27th of April and managed to start the evaluation at 9.40 a.m. As planned I was assigned to note down personal information of the users. After that Kanishka demonstrated the system and then the user is prompted to perform the given task. I have recorded the time, errors and memory aids provided while Malshika handled the recording and making notes. Afterwards the user is asked to describe the layout of the home page as he/she remember and then the home page page was replayed to assist the user to describe. Finally there was a Q&A session which had 14 ranked (1-5) questions, which was handled by my self.

According to our observations and discussions we got various types of feedback on the system; positive, negative and medium.

Positive feedback were generally on the;

• Identifying different objects on the screen
• having uniform positions for particular functions
  
• Error recovery
• Table structures
• Data comparisons

Among these; ididentification of the table structures gained high amount of positive responses. They said it is very difficult to distinguish among different rows and columns on a table when they are using a screen reader like JAWS even though it says that this is a "Table".

We have observed that users had very negative responses on the speed and clarity of the speech and said that it would be better to increase the speed and clarity.

We got medium responses for the points like;

• Navigational layout
• Interactivity level

majority of the users identified the locations of the objects on the home page very accurately though they were not much successful in recalling those location s from their memory. One of the participants even turned his head towards the sound directions as we noticed.

Typing speed was another important factor in the evaluation. Some of the users showed significantly low completion times since they are well trained in typing. One of the users have used computer before he/she become blind and he/she doesn't use screen readers to a fine level as she/he can see the screen to some extent. He/she found it somewhat difficult to identify the proper sound positioning.


Another important finding is that the difficulty of distinguishing different locations in the vertical direction. Most of the users said that it is not that easy to identify differences in the vertical direction (up & down) compared to the horizontal direction (left & right). But some of the users stated that when they get used with the system it become more familiar.


To sum-up, it was a successful user evaluation. This was our first experience in this kind of a evaluation - a human evaluation. It was a wonderful experience for us as well. We are planning to continue this session for another five participants on tomorrow (29th April 2011). Hope it will be a successful one as well!

Success at J'Pura...

Date: 21.04.2011

Time: 1.30 p.m.
Venue: Sri Sumangala Building, University of Sr Jayawardanapura (USJP)

We thought of getting more feedback for our system from an experienced computer user. Thus we decided to demonstrate our system to Mr. Chanaka of USJP. He also is involved in some programming activities like Mr. Asoka.

In this demonstration we have omitted the horizontal set-up according to previous feedback. It was his office cubical which we had to set up our layout. However it was sufficient enough for the purpose.

There was a noticeable improvement in conducting our experiment with compared to the previous ones. This one was more standard and focused in nature. Additionally we have decided to include 15 questions in the end which covered our research hypothesis and usability aspects. Purpose of inclusion of those questions was to address the qualitative and subjective nature of the user experience.

User feedback were amazingly positive. He also was able to identify the table structure and said it is important to have it for a better comprehension of the information. Another fact which arisen was having uniform location for particular functions on the screen (e.g. "Logout" in top-right). He stated that he wasn't aware of such standard positions of objects on the screen before this experience. But he stressed that it is better to have such positions if they can get spatial cues for object positions.

He clearly stated that he can identify the differences among the objects very clearly. He described the layout of the homepage very accurately.

Mr. Chanaka suggested that it will be great if we can extend this particular concept to be used with popular screen readers and web pages.

After interviewing Mr. Chanaka we met with Mr. Kasun who is a partially blind student in 2nd year at USJP. We witnessed the location to be used for our future experiments and agreed to take 5 students per day as time permits. We have planned to to those experiments on 27th and 29th of April. Before that we will be doing a pilot study on our colleagues on 25th of April to make sure we won't make serious mistakes in our experiments.

Yes, a Diversion!

We have faced a fundamental design issue in the previous session - whether the layout of the interface should be horizontal or vertical. Dr. Wimalaratne suggested that the horizontal one would be more familiar to the user. And that suggestion has seemed valid according to the observations done on 08th of April at ADMTC lab with Mr. Asoka Siriwardena of University of Colombo, who is an ICT instructor in Faculty of Arts. Its the first time we have demonstrated our system to a actula blind person.

There we have tried both the set ups - horizontal and vertical. Even though Mr Asoka was satisfied with both the systems with respect to the functionality, he suggested that it is better to work in vertical plane. He stated that though they can't see anything they still "think" that there is a vertical screen in front of them when they work with the computer.

In my opinion this was the main point in this particular observation. Apart from that he pointed out an important aspect of our system which can distinguish it from the others - identification of the table structures. In their routine activities they couldn't identify and use a table effectively since the sequential nature of the conventional screen readers. He pointed out that tables are very useful tools of data representation from which they couldn't be benefited currently.

He also suggested that our concept can be extended towards a web developing platform where they can identify the locations of the objects on the screen. It should be mentioned that Mr. Asoka is a web developer himself. He stated that urrently they have no mean of verifying their design decisions since they don't have the spatial information.

The other feedback were about the navigational issues such as speed, clarity of voice, key configuration etc. which had been noted down to be corrected in future developments.

In the evening we have visited Mr. Chanaka of University of Sri Jayawardenapura, who is also an ICT instructor and he himself is a totally blind person too. He also seconded the fact which the screen should be vertical than horizontal, though we haven't demonstrated the system to him.

A diversion?
Yes, a Diversion!

His Comments!

It was our second day at Sri Jayawardenapura University. As discussed on the very first day we were planned to do our second pilot test with Chanaka Sir, who is conducting computer training for visually impaired persons in the University. In our 1st discussion he asked us to show him the system, before start the actual evaluation. If he experience it first it is easier to explain to his students.

The system was setup in a cubicle inside Channa Sir's office. The speaker positioning was vertical as decided after Asoka Sir's experiment.

First gave him a brief description about our interface with positioned audio.

Secondly some tasks were demonstrated by Kanishka according to our preprepared evaluation plan. Then he was asked to do the same tasks on his own. This pilot test was well performed as desired, than all the other pilot sessions with both blind and non blind persons.

During the evaluation it is observed that Chanaka Sir is the most fastest learner of the system so far. He grabbed the concept quickly and was able to traverse using arrow keys to the correct positions. In the second task he was managed to traverse to the exact position without any confusions or aids.

Following are some of the HCI related comments gathered during the pilot test;

• Notification for num lock ON or OFF for use of numerical key pad, since it is used as shortcut keys when num lock OFF
• Concerned about positions of the words which has typed, in our applications user inputs are not positioned, they just readout from all 4 speakers
• Notification messages such as "Successfully Registered! Your information...." can be demonstrated after each input event
• Enter button can be used to focus and come out from a text box
• Add a background audio clip during page loading, to indicate the user has to be wait until the next page appears, or else at the beginning can play audio clips such as "Please wait" and after page loads "Welcome ....! (page name)"

Finally he gave some great advancements for audio based interfaces; this can be an extension for existing screen readers (JAWS) and generalize to all applications. He is really glad to use this new technology in teaching his students (both blind and non blind), since he can point out the place of a particular element. i.e. in MS word he can say "goto 'Home' on the top left corner.

This can be considered as the best and the most successful pilot test we have done so far!

Sri Jayawardhanapura- Initialization Test

An initialization evaluation session was held at University of Sri Jayawardhanapura today. The test subject is the Head of blind computing at the university: Mr. Chanka. Mr Chanka is an experienced, advanced computer user mainly working with JAWS. We were enlightened on few other open-source alternatives to JAWS which he suggested, we should try and integrate our system/concept into. The testing program started at around 2 PM, after lunch and was held for about an hour.

The evaluation process was more controlled and well defined than the other sessions. The subject was assisted during the registration process, in which he would familiarize with the text box management. After registering the subject was asked to login to the system using the predefined, username and password. The reason for the predefined login was that it had all the necessary relationships already defined (eg. The new user has no friends or messages. ).
The user was guided to search a friend using the term 'ni' and see the results, then select the first name (there was only one suggestion) and view the friends profile. The profile holds a table, something they do not experience JAWS, the table had the terms Name,Contact Number, Address etc and the respective values in the right column. The structure was correctly identified and was explained to us as a table (surprisingly the word table was used by the subject, which was more than we expected. :)). Then the table structures and how they are built in HTML was discussed with the subject (subject initiated by the user). Mr. Chanaka is a PHP developer. The tables are identified by the blind using the arrow keys in a word document, but it is impossible to actually experience in an web page as the subject explained.
Then the discussion was held to identify the observations by the user on the grid. The grid held name,address,contact number and user-name as the column headings, which were read prior to the data. Then the respective data were read according to the grid. The subject correctly identified the layout. He explained that the column names were in the top and the date is spread in respect to the column names. The explanation was initiated by the user, without any indication about the column names and data from us.
As suggestions the user said that identifying the location is important in teaching process. Expecailly if teaching to a non blind student. (The example he used is, if he could askthe student to find gmail registration link in top left corner, in the page, they would actually look, and the explanation would be complete ).
Mr. Chanka expressed his idea about using the concept on MS word 2007, which has the ribbon at the top of the page as a strip. He said that if the placement of the ribbon was known by the blind user it would increase the experience to a maximum level.
Finally he said that this is a very important feature that must be included in JAWS.
We came into believing that qualitative data may be more relevant to our project than qualitative data.

First pilot test with an actual user - Asoka Sir

Asoka sir, a lecturer, computer programmer and a web designer is invited for our first pilot test. Thursday April 21, 2011 around 10am, we managed to setup our system in ADMTC lab.

Initially the speaker setup was horizontal as in the picture. Thus the test object is seated in the center of the horizontal space.

Myself and Kanishka went to Art faculty and take Asoka Sir to the ADMTC lab. He was ready to come with us as discussed before. He asked Kanishka to take him to the UCSC Software Development Unit, because he got some problems with a php application he developed which includes Sinhala Unicode. Now you can guess his level of technical knowledge!

Then we started the experiment. At the very first time when traversing on the home page, he got scared when the sound comes from the behind. There we found another reason to accept the fact which is pointed by Prasad sir "the vertical layout is more suitable than the horizontal layout in a 2D sound environment. They already have an idea of the current layout of the monitor. Then we changed the speaker layout to the horizontal plane.

Following are some of the changes has to be done according to his comments;

• Use 'enter' instead of 'esc' to get out from a text box after typing the text.
• Before focus a text box use arrow keys to traverse on the page while listening, after focus use arrow keys to traverse inside the entry.
• Increase the speed of screen reading.
• Use 'ctrl' key to stop reading the page whenever user wants to.
• Add a sound clip or a tactical thing like "please wait" when processing and end of the process "Welcome". Otherwise they might be confused and could do unnecessary stuff which may lead to crash the system.
• Change the screen reader to a better one like "e-speak".
• Can introduce shortcut keys for complex navigations (e.g.: move to top left entry -> home key).

He also suggested us to do further researches on non-visual desktop access.

His final comment on our approach is;

For a typical non-technical computer user, the content of the web page is more important than the shape.

But for a web designer this concept is really helpful. Now they have to ask from somebody else if the position of an element is right or wrong. With this kind of environment they can work their own!