Jo+Ann+Coco-Ripp+-+CRCU

Course information
TRC 4300 Research Methods and Evaluation Course instructor: Dr. Jo Ann Coco-Ripp Semester and year: Fall 2010

Context
__Course and major challenges__ The Research course was redesigned using the Big Idea approach. Students who are junior/senior level in the Therapeutic Recreation major take this required course. In the past, the course used content and a project as the approach. Within the curriculum, several major outcomes were primarily met through this course: critical thinking and evaluation/research concepts. Several standards for accreditation are met through this course. After teaching this course for one semester, it seemed that some revision was needed due to mediocre outcomes from the standardized Watson-Glaser test scores, the completed group evaluation projects, and the less than stellar performance on tests by the students. Conceptually, students just were not grasping what was intended from the course and were not exhibiting critical thinking skills at a level needed. __My preferred teaching methodology__ Within a course, my style or approach may vary but the overarching methodology falls within experiential education or seeking to meet multiple styles of learning. My courses tend to include some use of technology. Also, my course design does not depend very heavily upon lecture or use of traditional tests for assessment. A challenge in design of a course for me is to plot out the direct connections for each activity with the outcomes intended. The more concrete and objective I can create the assignment or learning activity, the more comfortable I am with implementing something new. __Students' major characteristics__ The students in this course are in the 3rd or beyond semester in the Therapeutic Recreation Program. They have all negotiated several years at WSSU or transferred here. Overall, writing skills are average and reading tends to be a chore. Many of them excel at social interaction skills, such as leading an activity or interviewing. Grasping big ideas or the broader scope of a concept is tends to be an area of difficulty for them. Critical thinking is important for out students and this course is a crucial one for assessment and development of that outcome. __Innovation for course, impact on student learning outcomes and assessment of impact__ On May 24-26, 2010 I participated in the Course Redesign Workshop led by Edmund Hansen and offered through WSSU Center for Excellence in Teaching and Learning (CETL). This workshop guided me in designing TRC 4300 from a Big Idea or conceptual approach instead of the topic framework that had been used previously. It took about a full day and a half before the "Ahah" moment of understanding the distinction in the two approaches. Then I struggled with conceptualizing the course for several days after the workshop. Eventually, I developed the course based on the three Big Ideas of Problem Development, Outcome Measurement, and Communication of Evidence. See the that outlines these three Ideas.

Processes
// Weeks 1 & 2 // Beginning a course at the start of the semester is generally filled with anticipation and all involved have a level of enthusiasm. Most of the students in TRC 4300 came to class with minimal preparation as compared to my expectations. Prior to the first class students are sent a letter of welcome asking them to visit Blackboard and to skim the syllabus. A check immediately before class time showed only one student had visited Blackboard. In addition, only two students had investigated what text was being used. One student had acquired the book; one student had acquired the details for the online text and needed help to access it. This assistance was provided so two students of nine enrolled had the text by the end of the first week of class. Despite these difficulties, as the instructor I maintained my enthusiasm for the redesigned course and plunged ahead with Interactive Assignment #1 (IA #1). The first week was filled with detailed review of syllabus and assignments. The course involved 10 sequenced IAs (30% of semester grade) focused on acquiring knowledge and skills that contribute to the Group Evaluation Project which is 40 % of overall grade for the course. Students were continually directed to read all materials carefully and ask questions. Most questions did not reflect a deep interest in learning but “what must I do to get an A.” The first two IAs were article reviews. In the past, article reviews were placed at the end of the semester and often not even done due to time constraints. In the course redesign approach using the Big Idea and conceptual understanding instead of the topic approach, I thought an overview of the entire process as demonstrated through scholarly articles describing evaluation and research projects of interest to students would be helpful at the start of the project work. Thus, there was no model for the first IA from previous courses. The assignment was designed to engage students in class and predicated on some level of preparation outside class. With the lack of outside preparation already demonstrated during the first week it was with some trepidation that I expected the students to read the article and fully engage in guided discussion during the third class period of the semester. Therefore, it was quite a surprise that the discussion seemed to go very smoothly with all students who attended class fully involved and somewhat cognizant of the facts from the article. The article review discussion also seemed to provide motivation for students to do their individual review which is IA#2 and not due until week 3. IA #3 mini debate was scheduled for Thursday of week 2. Momentum from the article review discussion was helpful but, once again, outside preparation did not meet expectations. Only two students had access to the text by the fourth day of class and much of the mini debate class activity was intended to be based upon application of information from reading outside class not drawing upon common sense or speculation. Thus comparing research and evaluation by using the mini debate activity was not as successful as anticipated.

// Weeks 3 & 4 // A visit to the library begins week 3. Students engaged in the session and seemed to benefit from the instruction by the librarian. The formation of groups and topics for mini lit review (IA# 4) appeared to be a point where the impetus or drive to stay focused hit a snag. What I mean is that students had not solidified their group formation or project ideas so the lit reviews were an exercise not as directly connected to the semester long evaluation project in some cases as it needed to be. Nevertheless, the lit reviews were done at a moderate to good quality. IA#5 had to be revised from the original design due to the instructor’s planned attendance at a conference. Students did the exercise individually as a worksheet submitted through Blackboard. The work was done but review of the submissions showed two obvious influences: students who performed better on the worksheets about problem development had access to the text and students who seemed to do the work at the last minute or without much understanding of the assignment had less than stellar quality work. Problem development is a critical part of the overall conceptual understanding so this assignment will be something to consider at the end of the semester after the group projects are completed. Review of individual article review submissions was done during this time period. Again, the distinction in barely acceptable work and reviews that showed understanding of the articles being read was apparent. Some of the students were not reading the text, paying close attention to the deeper ideas from the discussions in class and this showed on the article reviews. Another item to put on the list for the end of the semester consideration…with particular emphasis on making the connections to the elements that will be part of the Group Projects.

// Weeks 5 & 6 // By this time, test one had been held and it also confirmed the learning gap among the class. Students either were not reading the text or not able to take the information read then apply it to activities or in testing format. For example, a student might be able to list some reasons to undertake an evaluation project but not able to discuss or explain anything about the reasons. Critical thinking is a major outcome for this class and a review at the end of the semester will need to focus on this learning gap that is becoming apparent. At the end of this 2 week segment, students were expected to have read up to chapter 8 (of a total of 12) in the text. Students had formed their groups for the Evaluation Projects and were working on the tasks within the group area on Blackboard. There were a series of blogs due as well as getting formal letters from the agency where the group would be doing the evaluation. As it turned out, all 3 groups were using a campus organization for their projects. Establishing a site to do the evaluation had been one of the areas in the past that had been problematic. Two more IAs were done during this time period. The swap meet was a new idea and I was curious how it might work out. The activity required prior preparation and evidence of it by writing notes which they were supposed to show the teacher before the start of the activity. All students came prepared to do it … except one student stated he needed to print it. I did not give permission to leave class to do so thus that student had to rely on his memory. My hunch is this worked out for the best anyway. Most of the students had handwritten notes and that may be an addition I strongly suggest for next semester. The swap meet went fairly well. It might be better to have a larger group for this type of activity but overall it engaged them in exploring various forms of data collection methods. The summary discussion at the end went very well so the minor glitches in the actual swap meet did not interfere with the outcome to explain strengths and weaknesses of various methods of data collection.

// Weeks 7 & 8 // A test, midterm, and two IAs were scheduled to be done during this time frame. It was too much to fit into the time frame so only one test was offered instead of both a test and a midterm. The only distinction in the test and the midterm is the midterm was comprehensive and the test focused on select material. Deliberation on the purpose of tests (including midterm and final) in the overall course will need to be done at the end of the semester. Students were very focused on writing drafts of parts of the introduction for their projects. I gave feedback to all material posted in file exchange within the group area on Blackboard. Sometimes students would send me an email attachment but I redirected them to post in appropriate area. Another encouragement given to all is to use discussion area to document work within their groups. Some groups did much better than others on this aspect of the work. The two IAs were directly related to the group project; however, there was a lot of confusion related especially to the instrument development. For the most part, the students wanted to get right into collecting data from the pilot instrument instead of going through the process of developing the questionnaire, pilot testing it, getting feedback from instructor and one other person then making changes. This will need to be emphasized and clarified next semester. Due to the confusion and impatience of the students to skip the process and immediately collect data, some of the instruments were not developed as well as could have been. The process for development of the interview collection method was similar but did not have as much an impact as questionnaire development. Both activities will need to be carefully reviewed before next semester.

// Weeks 9 & 10 // During these two weeks, it was a bit of catch up for the group work. Students were working on various tasks within their project groups; however, it was not apparent from the discussion board or the file exchange on Blackboard. I checked with each group and member to determine progress or interactions. Introduction of the last IA was done then the completion of the activity will be done next week. IA #10 focuses on learning how to design observation methods for data collection. Instead of using group projects as the background for the exercise, we spent some time in class listing a few ideas for a data collection using observation that could be executed by all. What we decided to use is the number of times a student texts during a five minute time period in a non TRC course. The student would choose two classes they already attend and unobtrusively observe one student for this 5 minute time frame. They discussed and decided on what one unit of the observation measure would be to be consistent. After this discussion, the teacher and the students wrote down what would be done and chose a deadline by which to have this done.

// Weeks 11 & 12 // Sharing the data collected had to be done in two different class sessions since 3 students did not attend either the class discussion last week or the result sharing session during week 11. The original sharing date worked out much better than I anticipated. Students had data and information on context of data collection. I wrote the data on the board in a chart. Then we discussed the outcomes in relation to what class had the highest incidence, lowest or other patterns. It was an outstanding exercise and students were fully engaged. We also discussed a variety of ways to display these results in a report as well as how to use Excel spreadsheets for data in their own studies. Another part of the discussion was use of triangulation. Finally, groups were directed to post on discussion board any comments related to use of observation in their own group projects or other ideas related to triangulation. The second sharing session was held with only the three students. This session seemed to be less productive in terms of discussion yet students were engaged and involved in all aspects. Having two sessions is not the best practice but it was better than not engaging these students. Perhaps, my enthusiasm was not as high for a second session either and that may have impacted the discussion. Also, carry over from the original session may have influenced the second group. Nevertheless, the students all demonstrated some increase in understanding of use of observation for data collection.

// Last weeks of semester // The rest of the semester was spent on directing group projects in completing data collection and writing the reports. Also, the students and I discussed inviting a representative from their site to a presentation. This was something new and not part of the original plan for the semester. All agreed on the idea and worked on the plan. A date was chosen during the last week of classes. Other details included the fact that all members of each group would be part of the presentation. A visual such as powerpoint was strongly recommended. Points for the final exam would be awarded for the presentation. No other guideline except being part of the presentation on the date decided was used to award the 60 points. If this is used in the future then more distinction for quality work should be developed. One group had a guest. No one showed up for the other groups. Two groups used powerpoint; one group just stood and talked. After the presentation, the powerpoint and the notes from the group that did not have one were posted on Blackboard. Again, more guidelines would be needed for future use of a presentation. However, for this semester, the presentations were all acceptable and all appreciated seeing outcomes from the other group projects. Test three was held during the exam time since it had not been done earlier because the focus had been on doing the projects. All tests were done online with use of notes and texts allowed. Test 3 results were helpful in overall assessment of learning outcomes.

Products
In redesigning the course using the conceptual approach, tests, the semester long Evaluation Group Project, and 10 Interactive Assignments (IAs) were the learning activities developed/selected to elicit learning outcomes. The items posted here include the 10 IAs, rubric, and other additional support materials for them.

One rubric was created for all the IAs. [|Rubric for Interactive Assignments in TRC 4300 Research Methods and Evaluation.pdf] Rubric for Interactive Assignments in TRC 4300 Research Methods and Evaluation Standards for submission 1. Submit on time; if due date is not clear, please ask. 2. Submit in designated manner; directions for how to submit will be given with assignment; if directions are unclear or need further clarification, please ask; for example, an assignment may be due on Blackboard in the assignments area as an attachment or an assignment may be due at the start of class in written format. 3. Follow all guidelines for completion of assignment; for example, an assignment may ask for reflection of five items so the submission should include comments that show all five items; if guidelines need further clarification, please ask. 4. Completion of assignment shows understanding of stated purpose. Each assignment will have stated purpose(s) and the design is intended to provide opportunity to demonstrate that purpose. For example, the purpose may be to compare research and evaluation; for the demonstration each student will need to contribute during a class activity. If the student is present in class but remains silent then that does not fulfill this standard; if the student makes a statement that has marginal or tangential connection to the purpose, then less than full value is given; if the student makes a comment that includes facts and shows a connection or insight into the similarities or differences in the two terms then full points are awarded. 5. Show progress on achievement of learning outcomes. Each assignment is designed to provide opportunities to contribute to achieving overall learning outcomes for the course. For example, one learning outcome (LO) for this course is to carry out an individual interview. Progress on this LO may be shown by thoughtful reflection on the results of the interview. A reflection that merely states the questions and responses from the interview would demonstrate minimal progress. Points Earned
 * Standard || 3 || 2 || 1 ||  ||
 * 1 || On time || Under 7 days late || More than 7 days late ||  ||
 * 2 || In designated manner || Needs to submit again or does not follow fully || Needs more than one X or has serious deviations in manner submitted ||  ||
 * 3 || Follows guidelines || Needs redirection one time || Does not follow or needs more than one X ||  ||
 * 4 || Fully demonstrates insight into purpose || Partial understanding shown; some confusion evidenced || Minimal or no evidence of purpose in submission ||  ||
 * 5 || Outstanding evidence of LO achievement || Acceptable support for progress on LO || Minimal or no substantiation of LO ||  ||
 * TOTAL ||  ||

[|Interactive Assignment #10.pdf] [|Interactive Assignment #9.pdf] [|Interactive Assignment #8.pdf] [|Interactive Assignment #7.pdf] [|Interactive Assignment #6.pdf] [|Interactive Assignment #5.pdf] [|Interactive Assignment #4.pdf] [|Interactive Assignment #3.pdf] [|Interactive Assignment #2.pdf] [|Interactive Assignment #1.pdf]

[|Report on Interview for TRC 4300.pdf] [|Mini Debate Instructions for TRC 4300.pdf] [|Worksheet 6.pdf] [|Article Review Guidelines for TRC 4300 Research Methods and Evaluation.pdf] [|Feedback Form for IA7.pdf]

These items worked well to contribute to the learning outcomes for which they were designed. Student feedback on each was solicited as each was implemented. Students had very few suggestions. Any change suggested that was feasible will be used for next semester. It will take several semesters to determine a real impact on students from this redesign. However, the projects were much improved from the last semester from my perspective. Since the assignments were all new this time, it will be interesting to see how the students next semester perform on the IAs and the Group Project.

Lessons learned
In this course, a student being absent for whatever reason the first 3 class meeting dates of the semester has a huge impact on the learning for that student and group formation for the rest of the class. This semester the student who missed the first 3 days of semester did not participate in the first interactive assignment (class discussion/article review) which turned out to be a foundational activity that set the framework for conceptual understanding.

Textbook access was a problem that needs to be resolved before the end of the first week. This was the first semester I used this textbook which was offered as a digital text. I did not anticipate that students would be so resistive to using the e-text version. It is about half the cost of the print copy and it is able to be placed in two locations such as a laptop and another device. The student does not need to carry the book itself. However, students asked for the hard copy! I think it is due to several reasons. Lack of familiarity with technology may have prompted some of the resistance as well as cost not being an immediate factor since many use “vouchers” from loans to purchase their books. Several students blatantly chose not to get the book. This possible demonstration of a lack of investment in their education is a concern. Nevertheless, problems with use of an e-text are now known and can be resolved before class starts or the first day. This textbook is also available in hard copy at the campus bookstore. Even though this text was new this semester and there were a few problems it is a good choice for this class and will be used next semester.

Looking back on article review in class discussion, it is important to make even more connections with group projects students will do such as how results are presented. In the final reports, several groups took the suggestion to use graphs and charts too literally—their results section only had charts and graphs with no text, narrative or other use of words to explain or reference the visual displays. So pointing out how graphs are used as part of the provision of results is a connection that can be made. Another area to further emphasize is background information. Even though students did literature reviews related to their projects, hardly any of the information found was used in the project reports in any areas even though I continually reinforced use of background information in several areas such as introduction or choice of data collection methods.

Forming groups is based on a combination of who is in enrolled in the course and individual interests. Form groups early and teach about group behavior (form, norm, storm, etc.) for this class and to use in TR groups later as well as collaboration as a professional in the future. Evidence of group work on Blackboard in both file exchange and discussion board needs to be emphasized. Resources on group work will be placed in the documents area for future use.

Problem development which was the focus of IA #5 is a critical idea that needs further attention in the course. Discovery of how important understanding this concept is confirms the first Big Idea and several of the essential questions, specifically 2, 3 and 6 (see below).

Essential Questions
 * 1) What is the purpose of the information to be gathered?
 * 2) What is the connection of finding the right problem or need to evaluate or research and quality service?
 * 3) Why is it so important to become familiar with the organization for which the evaluation project is being done?
 * 4) What might happen if systematic development of a problem is not followed?
 * 5) Why conduct a literature review?
 * 6) Why is the phrasing of the problem so critical?

Use of the Big Idea approach to redesign this course seemed to bring the concepts into sharper focus. Finding the gaps such as problem development was very helpful and will improve outcomes for future students.

** Future Actions ** Prior to teaching this class the instructor should review the three Big Ideas along with the Enduring Understandings, Misconceptions, Essential Questions, Guiding concepts, and Learning Outcomes for each one: 1. Problem Development 2. Outcome Measurement 3. Communication of Evidence Another preparation task is to review the 10 Interactive Assignments, especially in relation to needed competencies and learning outcomes as well as the performance criteria for all IAs. In order to facilitate group work, several key resources need to be posted on Blackboard. Two or three brief times for discussion of group behavior needs to be scheduled throughout the semester. Connections to group work in the course as well as beyond should be part of the discussion. Several relevant suggestions for current sites for the group evaluation projects should be gathered to assist in brainstorming sessions for forming groups. These can be used in addition to the list already available from this semester. In the future, I would suggest continuing the presentation activity at the end of the semester. An assessment of this will need to be developed and explanation of expectations for all groups. If this activity is incorporated in the current system of point values, I would offer this activity in lieu of the final exam.