Project 7: The Pilot Study

Developing Tools to Assess the value of integrating classroom instruction with the social network Facebook.

How does the integration of the social network Facebook with classroom instruction affect student retention of course material?

Initial Pilot Study:

In order to pursue design research in the effectiveness of connecting classroom instruction with technology mediated social networks a simple pilot study was developed. A course exercise was moved from the classroom to the social network Facebook. All students were already a part of the network and a Facebook group of the course CM297: A/V Production was setup. In order to ensure student confidence in participating the group was closed to only students registered for the course. The study involved four key phases and several methodologies:

1) Observe student behavior and use of Facebook outside the classroom. Engage the students in general conversation about their use of Facebook when observed using it in computer and video editing labs. Through these observations gage candidates for personal interview based on the usage, habits, openness and comfort level with research conductor.

2) Pre-Interview (several videotaped) students regarding their use of the social network Facebook and their opinions on it value as an educational tool. Standard questions avoiding yes/no responses include:

• Describe how you use Facebook in your everyday life. Are you an active or passive user? Explain why you consider yourself either.
• Why do you participate in social network Facebook? Explain the benefits you receive by participating within the social network?
• Discuss negative aspects of your participation within Facebook.
• What are areas integrating Facebook with classroom instruction would benefit you?
• Describe a classroom lesson or activity that would work well as an exercise on Facebook.

3) Develop an online class exercise to take place in Facebook. This activity is to be based off of the responses of the initial interviews. In other words, let the students select the item they feel would be most beneficial to take place within Facebook. Setup the exercise and monitor activity, establishing the goals of the learning exercise before it begins

4) Post-Interview (several videotaped) the same students regarding the experience. Standard questions avoiding yes/no responses include:

• Explain the appropriateness of the course activity within Facebook.
• Quantify your personal level of participation and buy-in to the assignment.
• How did the process help you better reach the educational goals of the project?
• What adjustments to the statements you made in the pre-interview regarding using Facebook in an educational setting would you make?

• What effect did the exercise have on your participation and retention of course materials

• What would you change about the project that would improve retention?

Initial Pilot Study Results:

The pre-interviews provided a excellent foundation for the design research pilot study. Not only did the interviews make available a clear Facebook project idea for the classroom, it confirmed the use of Facebook within the sea of social and educational networks. When questioned about the use of Facebook in an educational setting senior college student Rebekkah Breitbarth responded, Facebook is where the students are. It’s what they use. Freshman Marcus Taplin added, if want them to use it; you need to meet the students on their own terms. There are too many systems to learn and be a member of. A key to participation success is using the system students are familiar with.

Pre-interviews also indicated a high level of communication apprehension by students in the course. Most of the students felt they and others in the course tended to simply sit in classroom having minimal interpersonal contact with the instructor or students. This seemed especially true during sessions where student work was critiqued leading to an exercise idea, online viewing of student work combined with in class and online discussion. In a purely online educational environment student work is always critiqued through social networking. However, none of these traditional classroom freshmen and sophomore students had engaged in this type of critique session.

The Facebook group for the course was setup, all students invited to participate and the lesson scheduled to begin during a class session and hopefully continue beyond. Students were explained clear learning objectives of the exercise before it was started. It was absolutely essential students were aware of what they should be learning through the exercise before it began so at least limited assessment of retention of knowledge could be measured.

1) To view other students media projects (normally done in class) and comment on them

2) To begin and foster online social discussion of assignments

3) To learn proper critiquing techniques both positive and constructive including areas to address, how to address them and what to expect

4) To create comfortable dialog and a positive learning environment, not simply a support environment.

Students were given a class period to work on formulating and posting their responses and the professor would also actively participate. Initial responses to the exercise indicated a much higher level of participation than in class discussion rendered with even the quietest student demonstrating something valuable to contribute. Responses were not kept to one or two sentences but generally short paragraphs of information that contained critical commentary.

Post-interviews are ongoing as the Facebook group continues to function until the end of the academic term, which is May 14, 2008. However, preliminary post-interviews with participants have revealed thoughts and trends of the success of the pilot study. With twelve members of the course actively participating 25% indicated the exercise did get them to interact with the process of critique beyond the level they expected. When interviewed student Marcus Taplin, previously interviewed supported the success of one of the learning goals. He responded his understanding of methods of critiquing students have a much higher retention rate as he is able to return and review comments by both fellow students and professors. He also indicated communication apprehension was not a problem because being able to write out comments gave him more time to think. He developed a conclusion the lack of discussion during in class critiques may have more to do with a fear of public speaking and peer relationships than a lack of thought or opinion. While it is far to early to determine if connecting classroom instruction with continued conversation through the social network Facebook will have any significant effect on the fear of public address in the classroom students interviewed clearly made the connection.

Problems Identified Through the Pilot Study:

Problems and limitations of the case study where quickly identified through the interviewing and observation process. Many of these problems were related to limited scope of the pilot study due to time and resource limitations. Several other problem areas where identified for use in the development of a new pilot study review outlined in the final research design proposal.

• A comprehensive examination of the educational goals was not conducted and therefore precise measurements of retention could not be established. Student success and retention was measured through their perceptions as reveled through interviews. Instructor observation also provided feedback on whether the educational goals were better learned, but specific statistical data is not available.

• The results of the case study were not measured against a control group who did not complete the social network exercise.

• With only one exercise conducted and measured the potential for possible abnormalities is high. Drawing conclusions on retention based on one class exercise cannot provide any broad and conclusive evidence.

• Student background was not available. Key factors including current GPA, and academic potential were not directly applied to the results.

Conclusions:

The initial pilot study revealed several potential weaknesses of media design research but also achieved a major goal.  Clear evidence was collected that there is need and value to developing assessment tools on the integration of social network and classroom based instruction.  The pilot study has paved the way for a clear research proposal, which includes two additional phases to produce measurements of retention when expanding the classroom into the social network.  These phases will be outlined in the final research proposal.