Researcher’s Log 2008-02-18

Data collection for my Delphi study was completed as planned on February 8th. (As if on cue, Clark was born the next morning!)

I received 12 completed surveys in response to the final consensus check. Thankfully, this was the minimum number I set out to gather – so the level of attrition was acceptable, especially through rounds 2 and 3 and the final consensus check. Twenty-four experts initially agreed to participate in the study. Of these, only 15 people actually completed round one. Thirteen completed round 2, and twelve completed both round 3 and the final consensus check. So, following round 1, the attrition rate was only 20%. Those who left the study were not among the significant dissenting opinions.

In the final consensus check, consensus was defined in the following way:

For the purposes of this study, consensus is defined as the state in which the results are “at least acceptable to every member [of the expert panel], if not exactly as they would have wished.” (Reid, 1988, as cited in Williams & Webb, 1994, p. 182).

Participants were then asked to rate their level of consensus with each of six summaries on the following scale (Participants were also invited to leave additional comments after rating their level of consensus with each summary):

  • 5. Complete Consensus – I am in agreement with everything stated in this summary. The results are acceptable to me, if not exactly as I would have wished.
  • 4. High Level of Consensus – I agree with most of what is stated in this summary, and I disagree in only minor or insignificant ways. The results are acceptable to me.
  • 3. Moderate Level of Consensus – I agree with much of what is stated in this summary, but I also disagree in some ways. The results are acceptable to me.
  • 2. Low Level of Consensus – I agree with some of what is stated in this summary, but I also disagree in some major or significant ways. However, the results are still acceptable to me.
  • 1. No consensus – I disagree with most or all of what is stated in this summary. The results are not acceptable to me.

These ratings were used to find a level of consensus between the participants. Though many participants selected “5. Complete Consensus” in response to individual items, no items received that rating from all participants, so it would be inaccurate to report that there was complete consensus on any of the summarized themes. However, the participants ratings were averaged and the following scale was used to determine the level of consensus among the participants:

  • 5.0 Complete Consensus
  • 4.50-4.99 Very High Level of Consensus
  • 3.50-4.49 High Level of Consensus
  • 2.50-3.49 Moderate Level of Consensus
  • 1.50-2.49 Low Level of Consensus
  • 0.00-1.49 No Consensus
By this scale, there was a high level of consensus on four themes and a very high level of consensus on two additional themes. Out of 72 individual responses (12 participants responded to 6 summarized themes), 34 were “complete consensus,” 27 were “high level of consensus,” 10 were “moderate level of consensus,” and only one was “low level of consensus.” In other words, only one participant responded with a low level of consensus, and even then only responded in this way to one theme. No participants selected “no consensus” in response to any themes. Most dissenting opinions were minor and all will be addressed in the final report.

At this point in the process I have several new tasks ahead of me.

  • Final Data Analysis: I will code and complete analysis of the Final Consensus Check comments. These will be used to modify the summaries used in the final consensus check for their final appearance in my report.
  • Confirmability Measures: I will work with two of my colleagues, both of whom have recently completed in a doctoral dissertation focused on educational technology. They will aid me in a peer debriefing, in serving as devil’s advocates, and (in one case) in serving as an external auditor of my Delphi methodology.
  • I will complete a literature realignment to account for the months between completion of my proposal and the beginning of this dissertation draft.

Then I can move on to rewrite chapters 1 through 3 to reflect the actual implementation of this study – and to compose chapters 4 and 5 to report my results and discuss their implications. Though I had hoped to have a draft of my dissertation by March 1st, I don’t think that is possible at this point. However, I still hope to have a draft completed sometime in March. I think it’s time to take on no new work until this is done… especially with a new baby in the house.

9 Responses to “Researcher’s Log 2008-02-18”

  1. Educational Technology and Life » Blog Archive » MMORPGs in Education: Motivation & Engagement Says:

    [...] In a final consensus check survey, the participating experts indicated a very high level of consensus with this summary: Summary of Participant Responses Theme 1 Motivation and Engagement [...]

  2. Educational Technology and Life » Blog Archive » MMORPGs in Education: Context-Embedded Learning Says:

    [...] In a final consensus check survey, the participating experts indicated a very high level of consensus with this summary: Summary of Participant Responses Theme 2 Context-Embedded Learning [...]

  3. Educational Technology and Life » Blog Archive » MMORPGs in Education: Social Learning Says:

    [...] In a final consensus check survey, the participating experts indicated a high level of consensus with this summary: Summary of Participant Responses Theme 3 Social Learning [...]

  4. Educational Technology and Life » Blog Archive » MMORGPs in Education: 21st Century Skills Says:

    [...] In a final consensus check survey, the participating experts indicated a very high level of consensus with this summary: Summary of Participant Responses Theme 4 21st Century Skills [...]

  5. Educational Technology and Life » Blog Archive » MMORPGs in Education: Reflection Says:

    [...] In a final consensus check survey, the participating experts indicated a very high level of consensus with this summary: Summary of Participant Responses Theme 5 Reflection [...]

  6. Educational Technology and Life » Blog Archive » MMORPGs in Education: Infrastructure & Logistics Says:

    [...] In a final consensus check survey, the participating experts indicated a high level of consensus with this summary: MMORPGs may require fewer hardware resources compared to many other video game genres, but implementing MMORPGs in existing schools would include many challenges related to infrastructure and logistics. With current student to computer ratios, students might experience limited access to the game at school. Many computers in schools might not meet the hardware needs of modern MMORPGs. The bandwidth available at the school might also be limited. Technical problems with the software, hardware, and network as well as the logistical and cognitive overhead necessary to play the games might outweigh the positive learning experience. (Outside of the school, many socio-economically disadvantaged students might also have limited access to the equipment necessary to play an MMORPG.) Filtering games for age appropriate content may also be a concern. [...]