Category: Dissertation

Today I completed the summaries and emailed the (link to the) final consensus check to all 12 remaining participants. I also looked forward to the confirmability measures I mentioned in the proposal. Some I have covered, such as member-checking, rich thick description, negative or discrepant information, and “prolonged time in the field.” Some I need to look up in order to address them: triangulation and bias clarification. And some I need to recruit help with. For instance, I will need to recruit as many as three peers to serve as a devils advocate for the results, to aid me in a peer debriefing, and to perform an external audit. I have some peers in mind, but am prepared to use as few as one other person for these roles, because I do not expect to find very many peers willing to make the necessary time commitment. I will send out emails to these people next time I work – Wednesday at the latest.

In any case, I will send a reminder to all participants on Wednesday because the final consensus check is due on Friday (the 8th). I plan to complete the final analysis on the 9th and to begin the remaining confirmability measures immediately. I will also begin rewriting chapters 1-3 and drafting the final chapters of the dissertation. I still hope to complete a draft of the dissertation by March 1st.

Of course, all of this is contingent on when Eva has the baby, which is due in two days on the 5th.

1:30 PM I sent out the third round questions on the 25th. I sent reminders on the 29th and again on the 31st.. Today 11 of 13 have completed the study, and one other just contacted me to say she would complete it today, so I anticipate having the 12 participants complete this 3rd round that I had hoped for. Now I hope I won’t see any additional attrition during the final consensus check; it should, however, require less effort (though at least as much reading) as the previous rounds.

My analysis method will be different for this final round. Coding is not necessary. Round 3 presented summaries (by theme) and asked participants to respond to the statement in the summary that they disagreed with most strongly. In some cases participants replied that they did not disagree with anything on the page… and in other cases they indicated that they did not disagree “strongly” and then provided a response to some elements that they had minor concerns about – or else elaborated on a point. So, in this case I will simply deal with each response one at a time. I will treat each response in one of three ways:

1. Alter the summary to reflect the concern addressed in the response.
2. Catalog the response as a dissenting opinion, to be reported in the dissertation
3. Catalog the response as repetitive, off-topic, or an insignificant contribution.

I think I’ll use TAMS on my first run to tag items for these purposes: {alter}, {dissenting}, {repetitive}, {offtopic}, {insig}. (Or perhaps {discard} for an answer of type 3 that is difficult to classify.)

2:44 PM I used this system, but also needed another set of codes to group responses that addressed the same issues, which were few in number.

6:28 Earlier today I completed the Motivation and Engagement summary. Just now I’ve completed putting together the final consensus check online questionnaire. Due to space limitations in the survey tool, I am no longer embedding the summaries, but am linking to them instead. All that remains is to complete analyzing the responses to Round 3 questions 2 through 6 and then to complete (and upload) the final summaries (at the URLs I specified in the online questionnaire).

1:22 pm – Yesterday I completed the summary for the motivation and engagement section. It took approximately two hours to complete. My first summary today, the context section, took about an hour. Social Learning is taking longer. I hope to finish all of the summaries this afternoon and send out the Round 3 survey today.

1:44 pm – Social Learning took about 90 minutes to compose. I believe that 21st century skills and reflection will go much more quickly… and I am considering cutting the “other” category from this round. I think I fished out any interesting responses last time… and with the extra added reading for questions 1 through 5, I’d love to be able to cut one question again.

3:06 pm – The reflection section was very short, so I’m considering still keeping the “other” question. Working on it now.

4:43 pm – I just emailed out the Round 3 questionnaire. In the end I decided to focus the final (6th) question on the issues that the participants focused on in Round 2. So, question 6 focused on logistics, including cost, resistance, and organizational change. Other more minor benefits and concerns were dropped from this round. I look forward to the results. Perhaps in the intervening week I will read more for the literature realignment.

2:02 pm – Back to coding again. I’m noting that there is a developing consensus. Some participants are saying things such as “all the responses above are very valid,” “nothing to add,” and “there is absolutely no question.”

3:56 pm – For question five, I created a slightly difference coding scheme using one of the following letters as the second character in the code:

a – agree
d – disagree
i – idea

4:53 pm – I’ve completed coding. I’m unsure how to proceed with composing Round 3. I used an outliner to arrange and group codes for Round 2. But there are many more codes for Round 3 and I’m unsure if that approach will work, but I am attempting it to begin with.

5:12 pm – The outline approach appears promising. I also ran a “Count” report in TAMS, which lists all codes and their frequency of appearance. I imported this into a numbers spreadsheet and sorted by the number of appearances. 86 of the 119 codes I’ve used only appear once, so my first two characters (which categorize the codes) are critical in making the information useful. Those characters are driving my organization of the codes in my outliner, which will in turn drive my composition of the round three questions. It is already clear that my six Round 3 questions will focus on these topics:

1. Motivation and Engagement
2. Context-Embedded Learning
3. Social Learning
4. 21st Century Skills (for which I am seeking a different name.)
5. Reflection
6. Logistics (including resistance and time.)

The outline went well and I think I’m ready to compose new summaries (tomorrow). I still wonder about how best to approach the questions. Perhaps I will ask them what statement they disagree with (or agree with) most strongly and why.

8:36 pm – Last night I setup TAMS Analyzer for Round 2 analysis. I needed to re-learn how to create a new xtprj file and begin populating the code browser. With that preparation out of the way, tonight I begin coding the Round 2 responses so I can compose round three. In the previous round, the direction was very clear. Participant responses to my initial questions provided the detail necessary to write summaries for Round 2 and ask more focused questions. Now, though, the amount of detail looks overwhelming. How do I proceed?

In the case of the “other” question (question #5 in Round 2), the direction is clear. The issues that were ignored will not reappear in Round 3. There were two or three clear “favorites” (wether as benefits or concerns) that will reappear in Round 3. However, with the first four questions (motivation, context, inquiry, and social learning), the participants have provided still more data. Perhaps after I’ve coded the responses the direction will be clear. If I were to predict at this point I would guess that Round 3 might take this form: longer more detailed summaries followed by a question similar to Round 2 number 5 that asks them to take issue with one benefit and/or one concern. That way, what I will be left with is an even better “summary” for the final consensus check, when they will be asked to rate their level of consensus with the final summary. I’ll move forward with this in mind.

8:44 pm – My coding scheme has improved. I now include a two letter sequence before the primary word of the code. The first letter denotes the question topic (such as “m” for motivation) and the second letter denotes the response (such as “b” for benefit). Additional responses are listed below:

b – benefit
c – concern
n – need
i – idea

Due to the holidays I have extended the deadline for Round 1 of the study. I have sent a reminder to all participants and asked them to complete the survey by Friday, January 4th (if they haven’t already). I currently have 9 responses. However, posting on my blog a reflection on the process prompted many other potential participants to come forward, and prompted my existing participants to suggest colleagues who might be interested in participating. At this point I now have 18 participants who have returned a consent form, 2 more who intend to do so, and the 1 dropout. So, I am hopeful that I can reach at least 12 responses, if not as many as 20, by January 4th. This will give me greater confidence that as I proceed I will be able to finish the study with the minimum number of participants that I recommended in my proposal. If, however, this is not the case, I am sure that the data I am collecting will still be valuable. My early analysis turned up a variety of meaningful perspectives already, and it still remains for me to code the most recent four responses.

It has become less likely I will finish collecting data before the baby is born. I now project finishing data collection sometime between Feb 1 and Feb 15. The baby is due on the 5th. I am also concerned about making the March 1st “deadline” for having a draft of my results ready. Hopefully I can push that back and still be able to graduate in May.

This has been posted after the fact (on 2008-02-08) in order to protect the integrity of the study.

Further early analysis on the first five responses has generated additional material for existing codes, and additional codes. Much more has been submitted related to collaboration and social elements of games, and I am now seeing more mentions of the creative or expressive benefits of games. Also, more was written about the ability of games to be differentiated for various learners. In addition, I’ve seem more related to 21st century skills, and new material that has introduced the potential of MMORPGs to effect positive social change, which also came up in the literature review. Again, this analysis is related to the first question, which covers the potential benefits of MMORPGs as constructivist learning environments.

Question two asks about the potential drawbacks, and my early analysis has also resulted in more codes than I expected. A good deal has been submitted about educational organizations’ resistance to change, a topic that was cut from my final proposal. Some has been written about the drawbacks of constructivist pedagogies in general, as well as additional ways learning in an MMORPG might be difficult to assess (at least in the eyes of traditional educational establishments). In addition, of course, the issues of addiction and anti-social behavior have come up, as have concerns over the amount of time the games can take to play (particularly in an educational setting) and the lack of necessary infrastructure in schools. Also, several respondents have mentioned that videogames are not for everyone and that not all games are attractive to all gamers.