Curt Acredolo, RTA Program Evaluator

During the Spring Quarter of 1998, RTA was piloted in two intermediate-level foreign language courses, one in Spanish and one in Japanese. Students were administered pre- and post-class questionnaires and a pre- and post-class achievement test (a one-paragraph composition scored on organization, content and accuracy). They were also asked to complete open-ended course evaluations. Matching sections of the Spanish and Japanese courses were offered in regular classroom formats, and the students enrolled in the regular classes also completed the pre/post questionnaires and achievement test.

Evaluation of the UCD Spring Quarter RTA language courses thus included three components:

• RTA students’ subjective evaluations of the RTA program and the lab curricula
• Experimental and Control Group pre- and post-course questionnaires
• Experimental and Control Group pre- and post-course language proficiency evaluations

The first component provided qualitative data; the second and third provided quantitative data and permitted statistical analysis of changes in scores across pre- and posttest as well as Experimental versus Control Group comparisons. However, since the number of participants in these courses was small, and moreover, since not everyone recruited to supply data was able to do so across both pre- and posttest evaluations, statistical power was severely limited. For this reason, the present evaluation of the program has a decidedly subjective tone throughout. At this juncture, the analyses are considered primarily formative rather than summative -- that is, they are treated as a tool by which to identify existing strengths and weakness and promising amendments.

Praises.

In general, student’s comments at the end of the quarter indicated a very strong endorsement of the RTA program and lab curricula. Representative responses follow:

• RTA gives us an opportunity to play with Spanish, to try to communicate as we would in another country.
• It helps develop impromptu writing skills.
• You get to do something new and different. The activities were interesting.
• It helps students communicate with one another in Spanish, being able to think about what to say.
• It's a good program to practice Spanish. The Chat Window is definitely a strength.
• It improved my Spanish writing/conversation skills.
• RTA gets people speaking Spanish with others through the computer.
• It allows you to use your Spanish writing skills.
• It is a good way for students to communicate and practice the actual applications of Spanish.
• One really has to try and put into words whatever we want to express without the possibility to use signs and gestures.
• RTA definitely made me feel more comfortable using Spanish. Since we were not allowed to use English as a crutch, we had to figure out how to communicate with each other. I've never really had the opportunity to do that before this program.
• It's increased my interest. It's a lot better than sitting in a classroom all the time. It was a nice change and gave me something to look forward to.

• It was unique, and I enjoyed it
• It was an interesting way to learn the language, and it was fun and informative.
• RTA was a nice break from doing regular class work. I enjoyed it, and it was helpful in that I had to prepare for class so that I wouldn’t hold my partner up when using RTA.
• The RTA experience was fun and helpful. It made you think quickly and keep communicating. It took the place of having conversations with a partner in class, which I think is better because most people don’t seem to converse very long or enthusiastically in class.
• I liked using RTA. It was a good experience, especially in the context of a global economy. Being able to use a foreign language fluently also means using it well electronically, and that’s what RTA helps with.
• RTA was fun this quarter. I had the opportunity to communicate with my classmates in Japanese and at the same time practice typing skills in Japanese. I liked this program and I think it was helpful in learning Japanese.
• I think RTA made going to the computer lab more fun.
• I really enjoyed RTA. I think it was helpful.
• RTA was a fun and interesting program. It was effective in helping us with our Japanese.
• I think it was helpful because it forced me to use Japanese at times when I didn’t think I could. Overall it was a good experience.
• I liked it a lot because it improved my Japanese writing skills and helped me with recognizing Kanji.
• I really liked RTA. It was helpful and fun at the same time. It forced us to communicate in Japanese. Speaking Japanese to someone is helpful as well, but it is far too easy to slip in a couple of English phrases while speaking. RTA also allows us to look back on what we said and how we said it.
• I thought the RTA program really helped my writing in Japanese. Before doing RTA, I didn’t know that I would be capable of writing in Japanese without the help of a dictionary or textbook. I think it helped me with constructing Japanese sentences faster.
• It was fun. I had a lot of trouble conversing in Japanese. It helped me articulate my sentences.

Students applauded the structure of the RTA program. They loved the integration of the Text Pad, upon which they could jointly author their papers, and the Chat Window, which provided space for a separate dialog thereby not disrupting the composition. The following quotes reflect responses to the question regarding RTA’s best feature(s):

• The Chat Window is definitely a strength.
• Chat and Text Pad
• The Chat Window: it gave me an informal environment in which to talk in Spanish.
• Chat Window, Text Pad, joint Internet access!
• The Chat Window and Text Pad were neat.
• The Chat Window was very useful while using the Text Pad because we were able to communicate back and forth while writing text.

• I enjoyed being able to go back and forth between the Text Pad and the Chat Window. I thought this was neat because if something was incorrect in Text Pad, you could communicate with your partner without messing up the actual document.
• I liked the Chat Window best because it called for fairly quick responses. I felt that what we were doing in the Chat Window became more conversation-like.
• Text Pad and Chat Window: You and your partner could correct each other’s writing.
• The Chat Window was the best part as it helped practice speed of both composition and comprehension.

The students in the Japanese course also enjoyed the integration of the world wide web into RTA:

• I liked sending stuff from the web because it lets your partner see what you are working with.
• (I liked) looking at the Japanese web sites. (I) learned more about Japan.
• I liked the fact that you could add websites to your messages.

Complaints.

In this particular instantiation of an RTA language learning curriculum, the RTA labs provided considerable opportunity to practice reading and writing in a foreign language, and students were unanimous in their opinion that RTA facilitated that aspect of their foreign language learning. Many students also volunteered the opinion, however, that speaking and listening development were not similarly facilitated, and this deficiency needs to be amended in subsequent versions of the RTA language learning curriculum.

• (RTA provided) no improvement in terms of listening and speaking, but perhaps some in writing.
• It doesn't help me speak better Spanish, which is my big problem.
• (My)writing improved a little. I am more concerned with understanding orally.
• RTA develops reading and writing skills; speaking and listening are not as effected in current version.
• It did not improve my speaking skills.
• Speaking and listening wasn’t really incorporated.
• Speaking and listening is not as effected in the current version.
• Provide more oral exercises.
• Perhaps the use of the microphone so we could talk to each other would help improve speaking skills.
• [We need] more oral and listening exercises.
• [We need more} direct voice interaction.
• If possible, have RTA speak to you in Spanish and maybe answer questions on what it has said.

Solicited complaints focused principally on correctable technological (hardware and software) deficiencies: computer crashes and problems associated with who has control over the Text Pad.

• When using the Text Pad, what was being written on one computer was often a bit different on the other. Also, the Chat Window is very slow.
• Frequently the system would freeze or go down.
• Sometimes the computers didn't work very well, but that was because so many of us were trying to get to NetScape.
• Sometimes it breaks down and the joint essay goes bonkers.
• In Text Pad, it would switch on and off control, and if one person wasn't finished, their work was lost.
• The program shuts down and loses work.
• At times it would jam and just kick you off.
• I would be writing a lot to explain myself to my partner and all of a sudden RTA would shut down.
• Sometimes the screens froze and work was lost.
• Since it involves computers, problems relating to computers can occur, like one screen in a pair doing something different than the other screen.
• Text Pad was odd because you had to take control away from your partner.

• I experienced not a few crashes while using it, and it took a few sessions to get the hang of the program.
• The Text Pad was difficult. It tended to end up with one person or the other taking control and writing with the other sitting around or kibitzing from the Chat Window, and Text Pad kept getting yanked off, replaced by the Chat Window screen, or lost.
• It was annoying that someone could take control of Text Pad while you are writing.

A few students also offered complaints about the curriculum, expressing discontent with some assignments and expressing the feeling that there was insufficient time to complete assignments during lab periods. Here are some representative comments from students in the Japanese course regarding the need for more time.

• I think we just need more time.
• I wish we did it more.

Some students also noted that some RTA resources were introduced but barely used. Following are some representative comments from the students in the Spanish course:

• We didn't really use some of the features, like the White Board, and I was kind of confused about their purpose.
• We never found a real need to use the White Board in any of the assignments.
• I didn't care too much for the microphone thing. I didn't see the reason to use it because it must be used to say things quickly and just does not allow for enough time.

Suggestions.

When asked for suggestions, the primary response concerned the desire to interact with people outside of the lab, perhaps even with people in other countries.

• I would like to chat with people outside of the room.
• Maybe some interaction with people from outside the class, and if at all possible, people from other cities.
• Interacting with classes outside the university.
• I would like to interact with students from Spanish speaking countries.
• Communicating with students from other countries would be fun and interesting.

• I think it would be very interesting if the student can actually talk to people from Japan.
• I want to talk to someone in Japan, and I want to send pictures and stuff.
• I would like to communicate with other Japanese speakers and Japanese people.
• I would like to connect with someone in Japan and use the Chat Window to talk to each other.
• I hope to talk with Japanese people in Japan. So it makes it possible to learn Japanese culture.
• I would like to talk to people from Japan. That would be interesting.

These comments relate to another very important concern, and one expressed principally by the students in the Japanese course, but also implied in some comments by the Spanish students. The concern focuses on the fact that with RTA students are principally exposed to and receive feedback from other similarly skilled students rather than a more experienced and able student or the teacher.

• It just doesn’t work when both people aren’t very good at Japanese.
• I thought we would be e-mailing real Japanese people and chatting with them. That would seem more effective because sometimes you are paired with someone who doesn’t know anything either.
• Maybe it would be more helpful to work with those who are more knowledgeable in Japanese, like some kind of tutorial program.

And it appears that students were not always provided corrective feedback. The end result is that they may have learned incorrect forms from one another rather than correct forms from their instructors. This particular problem and suggestions deserves special attention.

• I wish we could have received a corrected version of our chat and the story we wrote.

Other suggestions were diverse. Some students would like to see groups composed of three or four individuals.

• Chat with more than one other person. Combine groups and have four people together.
• To be able to work with more than one person on a project.
• I think it will be a lot better if I could talk to more than one person.
• Perhaps communicating with other classes would be possible.

Other students noted the need for better instructions on the use of RTA and the incorporation of cut-and-paste capacities.

• Provide clearer, step-by-step instructions on paper.
• Add cut and paste capacity.

RTA’s Influence On Familiarity And Comfort With Information Technology.

In terms of whether or not participation in RTA promoted development of information technology skills, there were two types of responses: Most students admitted to having a fair amount of expertise already, so RTA did not do much to increase their skills, and others said that they began with little information technology familiarity or skill, and RTA thus helped considerably:

• Before this class I only knew how to check my e-mail. Now I can search the web as well as do other things such as chat.
• I had zero familiarity. I do feel more comfortable now.
• I was computer illiterate and now I am much more comfortable.
• Before taking this class I knew how to search the Internet for information but now I seem more comfortable and computer literate.

Students’ Satisfaction With The Cooperative Learning Aspect Of RTA.

Most students expressed enjoyment in getting the opportunity to work in a cooperative learning setting.

• I really liked working in pairs.
• I enjoy it. You get different points of view.
• It was nice having someone to work with collaboratively.
• I think that's great. IT was fun. You get to know other students.
• I highly recommend it.
• It was good to be able to work with someone.
• It was a good experience although it depends on the partner you get.
• I think it worked out very well.
• I enjoy it. This way I can compare my understanding of Spanish with others'.
• I thought it was a good idea. It made it easier to write the papers because we were able to share our Spanish knowledge with each other, and two heads are better than one.
• It felt good to interact with other students.
• I enjoy group work; it makes learning more fun and holds my attention.
• The fact that you could communicate while writing a joint project was a great feature.
• I really liked the idea of writing with others. That way, I can learn from them and help them at the same time

Conclusion.

In conclusion, the students gave RTA a strong endorsement and provided valuable information for improving both RTA and the RTA language course curricula.

• I liked it very much. Once the technical bugs are worked out, it will be an absolute winner!

Students enrolled in both the RTA and standard versions of a Spanish and a Japanese language courses were administered the following six questionnaires via the web at the beginning and end of the quarter:

• Level of familiarity with Spanish/Japanese
  (a series of open-ended questions, some of which were subsequently quantifiable)
• Interest in Spanish/Japanese Immersion
  (four questions responded to on a 5-point scale: 1=Never; 5=Daily)
• Proficiency in Spanish/Japanese
  (22/21 questions responded to on a 5-point scale: 1=Poor; 5=Perfect)
• Comfort with limited Spanish/Japanese
  (9 questions responded to on a 5-point scale: 1=Never; 5=All the time)
• Cooperative learning style
  (18 questions responded to on a 5-point scale: 1=Never; 5=All the time)
• Use of information technology
  (7 questions responded to on a 5-point scale: 1=Never; 5=Daily)

For each of the five 5-point scales, "scale scores" were computed by averaging the responses to the questions comprising that scale.

In the following analyses, data from the Spanish and Japanese courses are considered separately.

Comparisons Of The Control And Experimental Groups Taking The Spanish Language Course.

The following table summarizes not only the questions comprising the individual questionnaires but also average scores on each question (and on the scale scores) for both the Control and Experimental Groups and the p-values associated with the comparison of Control and Experimental Group scores. These data were analyzed using independent t-tests, and for informational purposes, all p-values less than .10 are presented and those less than .05 (of which there are none) are bold-faced. As can be seen by looking down the right-most column, there were no major differences between the Control and Experimental Groups before classes began.

The following table summarizes Pretest and Posttest, Control Group and Experimental Group data. Independent repeated measures analyses of variance were used to analyze the data, and p-values associated with the two "Within-Subjects" factors (the pre/post main effect and the group by pre/post interaction) are summarized. As in the prior table, all p-values less than .10 are presented and those less than .05 are boldfaced.

There are two things to keep in mind when reviewing these data:

The most important scale is that labeled Proficiency in Spanish. For the combined groups, the scale score (i.e., the average score across all questions) rose significantly from pre- to posttest, and scores rose significantly on the following individual questions:

• Making reservations in Spanish.
• Persuading someone to do something in Spanish.
• Arguing my political perspective in Spanish.
• Sending e-mails in Spanish.
• Displaying vocabulary breadth in Spanish.
• Using reflexive verbs in Spanish.

Overall (i.e., on the scale score), the Control Group improved more than the Experimental Group, but not significantly so. On the individual Proficiency in Spanish questions, however, there were two with significant Group-by-Pre/Post interactions, and in both cases the Control Group displayed large improvements in scores from pretest to posttest, while the Experimental groups showed declines.

• Ordering a meal in Spanish.
• Making the correct distinctions between ser/estar in Spanish.

The opposite pattern was found, however, for two questions in the Comfort with limited Spanish scale. In both cases, the Experimental Group showed greater improvement in scores from pretest to posttest.

• I understand what’s been said in Spanish but I can’t respond.
• I notice errors in my classmates’ speech in Spanish.

These differences between the Experimental and Control Groups are minor, and there were no other noteworthy pre- to posttest changes or group differences in these questionnaire data. The results thus indicate parity between the RTA and traditional course formats.

The overall impression is that the Experimental group performed on par with the Control group. Proficiency improved significantly for both groups, and although it appears the proficiency improved slightly less for the Experimental than for the Control Group, the difference is neither large nor statistically significant.

Neither Control nor Experimental versions of the course appear to have had any impact on the other Scales. Interest in Spanish Immersion and Comfort with Limited Spanish actually dropped after the course, though not significantly so. And neither Cooperative Learning Style nor Use of Information Technology were significantly impacted.

Instructors naturally hope that their classes will inspire students to immerse themselves in the subject matter and give their students greater confidence to challenge ahead. The actual achievement of these goals is probably the exception rather than the rule, but continued improvement of RTA-supported curricula certainly hold this promise. That RTA failed to have a dramatic impact on cooperative learning style and the use of information technology is not surprising. Although the students in the RTA lab were thrown into cooperative learning situations, the curriculum was not set up to provide instruction and support in making cooperative learning work well. As a result, some students may have come out feeling more frustrated than pleased by their cooperative learning efforts. And during this particular pilot assessment of RTA, the curriculum did not include broad exposure to web and its benefits. It is intended that future instantiations of RTA’s foreign language course curriculum will include a greater use of web resources.

Comparisons Of The Control and Experimental Groups Taking The Japanese Language Course.

In the Japanese courses, a large Control Group was available and completed the pretest questionnaires, but since only three Control Group students completed the posttest questionnaires we cannot compare the extent of improvement over the quarter across Control and Experimental Groups. The following table therefore provides a comparison of the Control and Experimental Groups prior to the start of the class, and an evaluation of the changes in scores from pretest to posttest for the Experimental Group. On the pretest, the groups were compared using independent samples t-tests, and pretest/posttest changes in the Experimental Group were evaluated using the related samples t-test. As in prior tables, all p-values less that .10 are presented, and those less than .05 are shown in boldface.