Evaluation second session- University of Sri Jayawardhanapura

The second evaluation session was held at University of Sri Jayawardhanapura. The process was identical to the session one: similar time and venue. The system setup was quick, since we were familiar with the environment. Our concept was introduced to five students. Three being female and 2 being male. This time we met higher percentage of totally bind students. The process was balanced so there ware 5 students who experienced the linear model of our system and 5 with the 2 dimensional environment.
During the evaluation process, even without analyzing the data, one thing was sure.What we thought of about performance and the layout information was absolutely wrong. It was noticed that they performed quite well with the linear system than the dimensional system when considering time, error rates and help required. This evaluation had many issues which we are unable to measure.

• When measuring time, some people were very fast in typing (there was less typing, other than in registration page)
• Some students were very quick in understanding the new system (concept) while others were unable to grab the concept that quick. Specially the concept of moving in two directions using the arrow keys was unfamiliar to them.

These things are not measurable, but affects the final output significantly. Anyway at a glance I would say they did better with the liner system than ours in time, error rates and time of learning. But they said it would have been easier to learn our system faster if they were introduced to the concept earlier (they are already familiar with the linear model, thanks to JAWS).
Most people got the idea of location and the layout of the information, but they had an issue with mapping the arrow keys with the layout they had in there mind. My guess would be that they were unfamiliar with the whole idea of moving in two directions using a computer. At times they pointed the right direction telling that the object was there and the location they currently reside, but were unable to navigate towards it in the first try. Totally blind, male students showed best results in identifying the direction and moving towards it.
Another observation was that, they correctly identified the movement of sound from left to right. It was even viable from their body language that they perceived the movement. But most found it difficult to identify the sound coming from up and bellow. The reason I would say is that they never expected to hear sound from all four directions. They know of stereo systems, but very rerely experience 4 speakers, specially placed top and bottom. Other reason I would suggest is that human ears are placed in left and right sides of the head. So it would be not an issue to identify left and right channels, but the top and bottom would at times give the same idea to the ears. But most of the students born blind, coped extraordinarily well in this issue. When they got the idea of two directions the perception improved.
After completing the evaluation with 10 students we spoke with the president of the blind society and thanked him. They were very helpful and without their organizing skills.(They had allocated time slots to participate in the experiment, without our intervention). Overall I think we made some really good findings which were not investigated before. The results of the evaluation must be compiled and conclusions made.

User Evaluation - Session 2

It's Friday the 29th of April 2011 on which we conducted the second part of our evaluation. It was the same place and no of participants was five (5). This time all the participants were blind among which 3 were female and 2 were male. There was one participant who had a Facebook account and hoping to learn programming stuff. However typing speed of all the participants were low compared to the other day.Evaluation procedure was same as the other day. We were on the schedule as expected.

ajor feedback we got was the difficulty of identifying differences in vertical direction. Only one user stated that there is no difference in the vertical and horizontal directions (these are not horizontal and vertical planes. They are the directions on the vertical plane).

Another novel finding was the fact that they are not familiar with the arrow keys. They don't use arrow keys much in their day-to-day work. Instead they heavily depend on Tab, Shift and Enter keys. Users haven't had any issues with the introduction of the Function keys (F1, F2, F3) in this system. They got used to it as they work on the system and no one complained about it as well.


One user pointed out that it is better to read out the password text box at least like "***" when it loses the focus. he suggested that he got confused when he didn't hear the entered password and he thought it as his fault.

In the end we were able to conduct our use evaluation for 10 users. We divided them in to two groups A and B - 5 users per group.

For group A we presented the positioned system firstly and recorded the time, error memory aids provided. Then we asked them to describe the direction of the layout of the home page from memory and then while playing home page. Finally we presented them the questionnaire.

For group B we presented the non-positional system to gather quantitative data (i.e. time, error memory aids provided ) and then demonstrated the positioned system to gather qualitative data (i.e. layout of the home page and questionnaire).

Results were more or less the same from the previous session. Now we have to analyse these results and document them in our thesis. But I can tell one thing this experience was worthy of more than thousand documents!!!

1st day: 27th April, 2011

A significant day of our research project, as well as our entire university life. Three of us were looking forward to have a tremendous experience!

8.00am:

Entering to the J'pura University. Had breakfast from the Art faculty canteen, with a lot of uncertainty and a curious mind. Although there was a possibility of getting a negative results for the hypothesis, it was a pleasure to do a such kind of experiment.

8.45am:

There was a room, which uses for sound recording, therefore sound proof, and was ideal to carryout our experiments. It was booked early by Kasun with Chanaka Sir's approval. We managed to setup the room by 9.10am. Laptop computer with the prototype to be evaluated, USB keyboards, external sound card and 4 speakers are main components of the setup. Distance between 2 speakers were approximately 1 meter. In addition some boxes and dumb CPUs were used to keep the speakers. Furnished setup is shown in the picture.

There were two systems; with and without positional information, to be evaluated and compared. Thus 10 students are evaluated in 2 days. Approximate time for one student is 1 hour. Therefore only 5 students are evaluated in a day.

Evaluation procedure is explained in the User Evaluation Part 1- University of Sri Jayawar... by යාත්‍රිකයා.

Participants of the day.

B - blind PB - partially blind F - female M - male

	Positioned	Non Positioned
Nilmini (B) (F)	√
Harshani (PB) (F)	√
Kasun (PB) (M)		√
Chaminda (B) (M)		√
Aruna (PB) (M)	√

9.40am:

Nilmini  was ready to be our first test subject. She was born blind. Her computer literacy is moderate and has done a computer course sometime back.

Kanishka started to demonstrate the system with positional information. The session is being recorded and got some pictures with their approval.

At the very first time she identified that the page begins from the top left corner, before Kanishka explained it.

Another important factor observed from Nilmini is she turned her head according to the sound. She unreservedly looked at the exact position where the sound comes from. She identified the Personal info structure, as a table with 2 columns. Even though they can draw table structures in Braille, they never experienced structures such as tables and graphs in computer applications.As we observed even in the pilot studies, there is a slight difference of identifying up-down and left-right positions. Left-right can be understood more clearly compared to up-down dimension. Nilmini too indicated that there is a difference, but up-down also can be identified without any effort.

She was very excited about the application and asked us if this is imported from another country!

10.45am:

Harshani , a girl who is partially blind. she was also given the system with positional information and asked to perform several tasks. One of them were to find the 'Friend Summary' element on the home page. Below picture shows how she attempted to find that element.

She started from the top left corner of the page and navigate to right side. She didn't realize that there are only 3 elements, thus tried to go to the 4th element. Since notification sound played she realized that its the end of the page. Then she went down and found Friend Summary.

When she is asked to explain positions of the home page elements as she remember, her answer was she doesn't remember since this was used only one time. but it is useful if the logout button is always in a same place.

Harshani identified the table structures correctly and commented as extremely useful. Also she said it is clear and easy to understand. For the question on identifying up-down and left-right positions, she said it is difficult to identify up-down positions compared to left-right.

11.45am:

Kasun , he was the one who organized everything for us at J'pura University. Kasun is the president of the association for students with disabilities.

He was first given the non-positioned system and noted down his performs. Then gave the system with positional information. Even though he has identified the correct position, he had some problems when navigating to that position. He always tried to go to left side first (from the left corner element).

As I observed there was another issue, he tried to start navigation from the position where sound stopped. But auto read functionality just read the page without focusing on any element. At the beginning focus is on top left corner element, until the user move it using arrow keys. He is so much used to current system. Although he has identified the table structure, he has proposed us to label it as a table and read it out. He is not so good in listening. Reason can be he is partially blind.

However he is such a good organizer!

1.00pm-1.45pm: Lunch hour.

1.45pm:

Chaminda  is the preformed subject as we observed. He was also given both non-positioned and positioned systems. At very first time when home page was playing he is understood that the page has 9 elements in both x/y directions. In the Friend Search task he navigated correctly to the exact position straightaway, without any mistakes.

Table structure is identified correctly. He said it has two columns. Even though he has heard about tables structures in computer applications, he has never experienced it like this. He wanted to have this feature when drawing new tables, which they cannot do with current applications. 'Friend Summary', a comparison kind of table is also understood as 4 column table which has headings and data under those headings.

His comment for x/y dimensional identification was up-down not so difficult but left-right easier.

2.50pm:

Aruna, the last test subject of the day. He is also turned his according to the sound, as the 1st user. Even though they can identify the position by sound, it is difficult to navigate using arrow keys to that position. It's not because they cannot find the place, but they have forgotten what is the current focused element.
He has identified the table correctly and navigate to the Friend Summary, such as a non-blind person. He forgot the location first, was in the wrong place after listening for second time, resumed from the exact location. They also didn't know, but we will investigate.

Issue of identifying up-down and left-right is same. Up-down is difficult and left-right is easy.

His final comment was to develop this concept further and develop an application with sound positioning.

3.45pm:

Done for the day and getting ready to go home. Hope to conduct other 5 participants on Friday, 29th :)