Researcher’s Log 2008-02-03

Today I completed the summaries and emailed the (link to the) final consensus check to all 12 remaining participants. I also looked forward to the confirmability measures I mentioned in the proposal. Some I have covered, such as member-checking, rich thick description, negative or discrepant information, and “prolonged time in the field.” Some I need to look up in order to address them: triangulation and bias clarification. And some I need to recruit help with. For instance, I will need to recruit as many as three peers to serve as a devils advocate for the results, to aid me in a peer debriefing, and to perform an external audit. I have some peers in mind, but am prepared to use as few as one other person for these roles, because I do not expect to find very many peers willing to make the necessary time commitment. I will send out emails to these people next time I work – Wednesday at the latest.

In any case, I will send a reminder to all participants on Wednesday because the final consensus check is due on Friday (the 8th). I plan to complete the final analysis on the 9th and to begin the remaining confirmability measures immediately. I will also begin rewriting chapters 1-3 and drafting the final chapters of the dissertation. I still hope to complete a draft of the dissertation by March 1st.

Of course, all of this is contingent on when Eva has the baby, which is due in two days on the 5th.

Researcher’s Log 2008-02-02

1:30 PM I sent out the third round questions on the 25th. I sent reminders on the 29th and again on the 31st.. Today 11 of 13 have completed the study, and one other just contacted me to say she would complete it today, so I anticipate having the 12 participants complete this 3rd round that I had hoped for. Now I hope I won’t see any additional attrition during the final consensus check; it should, however, require less effort (though at least as much reading) as the previous rounds.

My analysis method will be different for this final round. Coding is not necessary. Round 3 presented summaries (by theme) and asked participants to respond to the statement in the summary that they disagreed with most strongly. In some cases participants replied that they did not disagree with anything on the page… and in other cases they indicated that they did not disagree “strongly” and then provided a response to some elements that they had minor concerns about – or else elaborated on a point. So, in this case I will simply deal with each response one at a time. I will treat each response in one of three ways:

1. Alter the summary to reflect the concern addressed in the response.
2. Catalog the response as a dissenting opinion, to be reported in the dissertation
3. Catalog the response as repetitive, off-topic, or an insignificant contribution.

I think I’ll use TAMS on my first run to tag items for these purposes: {alter}, {dissenting}, {repetitive}, {offtopic}, {insig}. (Or perhaps {discard} for an answer of type 3 that is difficult to classify.)

2:44 PM I used this system, but also needed another set of codes to group responses that addressed the same issues, which were few in number.

6:28 Earlier today I completed the Motivation and Engagement summary. Just now I’ve completed putting together the final consensus check online questionnaire. Due to space limitations in the survey tool, I am no longer embedding the summaries, but am linking to them instead. All that remains is to complete analyzing the responses to Round 3 questions 2 through 6 and then to complete (and upload) the final summaries (at the URLs I specified in the online questionnaire).

Researcher’s Log 2008-01-25

1:22 pm – Yesterday I completed the summary for the motivation and engagement section. It took approximately two hours to complete. My first summary today, the context section, took about an hour. Social Learning is taking longer. I hope to finish all of the summaries this afternoon and send out the Round 3 survey today.

1:44 pm – Social Learning took about 90 minutes to compose. I believe that 21st century skills and reflection will go much more quickly… and I am considering cutting the “other” category from this round. I think I fished out any interesting responses last time… and with the extra added reading for questions 1 through 5, I’d love to be able to cut one question again.

3:06 pm – The reflection section was very short, so I’m considering still keeping the “other” question. Working on it now.

4:43 pm – I just emailed out the Round 3 questionnaire. In the end I decided to focus the final (6th) question on the issues that the participants focused on in Round 2. So, question 6 focused on logistics, including cost, resistance, and organizational change. Other more minor benefits and concerns were dropped from this round. I look forward to the results. Perhaps in the intervening week I will read more for the literature realignment.

Researcher’s Log 2008-01-23

2:02 pm – Back to coding again. I’m noting that there is a developing consensus. Some participants are saying things such as “all the responses above are very valid,” “nothing to add,” and “there is absolutely no question.”

3:56 pm – For question five, I created a slightly difference coding scheme using one of the following letters as the second character in the code:

a – agree
d – disagree
i – idea

4:53 pm – I’ve completed coding. I’m unsure how to proceed with composing Round 3. I used an outliner to arrange and group codes for Round 2. But there are many more codes for Round 3 and I’m unsure if that approach will work, but I am attempting it to begin with.

5:12 pm – The outline approach appears promising. I also ran a “Count” report in TAMS, which lists all codes and their frequency of appearance. I imported this into a numbers spreadsheet and sorted by the number of appearances. 86 of the 119 codes I’ve used only appear once, so my first two characters (which categorize the codes) are critical in making the information useful. Those characters are driving my organization of the codes in my outliner, which will in turn drive my composition of the round three questions. It is already clear that my six Round 3 questions will focus on these topics:

1. Motivation and Engagement
2. Context-Embedded Learning
3. Social Learning
4. 21st Century Skills (for which I am seeking a different name.)
5. Reflection
6. Logistics (including resistance and time.)

The outline went well and I think I’m ready to compose new summaries (tomorrow). I still wonder about how best to approach the questions. Perhaps I will ask them what statement they disagree with (or agree with) most strongly and why.

Researcher’s Log 2008-01-22

8:36 pm – Last night I setup TAMS Analyzer for Round 2 analysis. I needed to re-learn how to create a new xtprj file and begin populating the code browser. With that preparation out of the way, tonight I begin coding the Round 2 responses so I can compose round three. In the previous round, the direction was very clear. Participant responses to my initial questions provided the detail necessary to write summaries for Round 2 and ask more focused questions. Now, though, the amount of detail looks overwhelming. How do I proceed?

In the case of the “other” question (question #5 in Round 2), the direction is clear. The issues that were ignored will not reappear in Round 3. There were two or three clear “favorites” (wether as benefits or concerns) that will reappear in Round 3. However, with the first four questions (motivation, context, inquiry, and social learning), the participants have provided still more data. Perhaps after I’ve coded the responses the direction will be clear. If I were to predict at this point I would guess that Round 3 might take this form: longer more detailed summaries followed by a question similar to Round 2 number 5 that asks them to take issue with one benefit and/or one concern. That way, what I will be left with is an even better “summary” for the final consensus check, when they will be asked to rate their level of consensus with the final summary. I’ll move forward with this in mind.

8:44 pm – My coding scheme has improved. I now include a two letter sequence before the primary word of the code. The first letter denotes the question topic (such as “m” for motivation) and the second letter denotes the response (such as “b” for benefit). Additional responses are listed below:

b – benefit
c – concern
n – need
i – idea

Researcher’s Log 2008-01-02

Some of my research log cannot be shared here – at least not until the Delphi process is finished. In particular I can’t share the entries about the content of the participants responses. However, here is my latest entry, describing the logistics of the study, which may be of interest to those who may conduct a similar Delphi study or to those interested in the progress of my particular study.

Here is where I stand with my study. I have sent out 71 invitations to participate. Of these, 24 have indicated interest in participating. Only 22 have actually sent in consent forms and received a link to the first round questions. One of these has dropped out. Currently, I have only 11 responses to the first round questionnaire. I have extended the deadline for Round 1 to this Friday due to the holidays. If I receive even one more response, I’ll have what I considered my “minimum” amount. Regardless, despite the fact that responses have varied in quality (of course), there is no question I have plenty of material to proceed with the study.

My preliminary coding process has revealed several topics I did not originally foresee including in Round 2, so the process of beginning with a broad question is working. Also, happily, much of what they have brought up has been related to issues I cut from my literature review during the revisions that shortened my proposal. I expect I may be able to reuse some of that research and material in the discussion of my results in chapters 4 and/or 5.

Once Round 1 has completed on Friday, I plan to complete my analysis and send out Round 2 questions as soon as Monday the 7th or as late as Friday the 11th. This puts me a bit behind schedule (of course), but if I continue to do preliminary analysis as responses come in, I should be able to make up some time between each round.

Thank you again to all of you who offered assistance after my last post. I wish I were able to use more of that. I can’t wait to post the results here and receive open feedback from all of you as well.

Researcher’s Log 2007-12-28

Due to the holidays I have extended the deadline for Round 1 of the study. I have sent a reminder to all participants and asked them to complete the survey by Friday, January 4th (if they haven’t already). I currently have 9 responses. However, posting on my blog a reflection on the process prompted many other potential participants to come forward, and prompted my existing participants to suggest colleagues who might be interested in participating. At this point I now have 18 participants who have returned a consent form, 2 more who intend to do so, and the 1 dropout. So, I am hopeful that I can reach at least 12 responses, if not as many as 20, by January 4th. This will give me greater confidence that as I proceed I will be able to finish the study with the minimum number of participants that I recommended in my proposal. If, however, this is not the case, I am sure that the data I am collecting will still be valuable. My early analysis turned up a variety of meaningful perspectives already, and it still remains for me to code the most recent four responses.

It has become less likely I will finish collecting data before the baby is born. I now project finishing data collection sometime between Feb 1 and Feb 15. The baby is due on the 5th. I am also concerned about making the March 1st “deadline” for having a draft of my results ready. Hopefully I can push that back and still be able to graduate in May.

Researcher’s Log 2007-12-19

This has been posted after the fact (on 2008-02-08) in order to protect the integrity of the study.

Further early analysis on the first five responses has generated additional material for existing codes, and additional codes. Much more has been submitted related to collaboration and social elements of games, and I am now seeing more mentions of the creative or expressive benefits of games. Also, more was written about the ability of games to be differentiated for various learners. In addition, I’ve seem more related to 21st century skills, and new material that has introduced the potential of MMORPGs to effect positive social change, which also came up in the literature review. Again, this analysis is related to the first question, which covers the potential benefits of MMORPGs as constructivist learning environments.

Question two asks about the potential drawbacks, and my early analysis has also resulted in more codes than I expected. A good deal has been submitted about educational organizations’ resistance to change, a topic that was cut from my final proposal. Some has been written about the drawbacks of constructivist pedagogies in general, as well as additional ways learning in an MMORPG might be difficult to assess (at least in the eyes of traditional educational establishments). In addition, of course, the issues of addiction and anti-social behavior have come up, as have concerns over the amount of time the games can take to play (particularly in an educational setting) and the lack of necessary infrastructure in schools. Also, several respondents have mentioned that videogames are not for everyone and that not all games are attractive to all gamers.

Researcher’s Log 2007-12-18

In the methods chapter of my proposal, one procedure I stated I would follow during the data collection and analysis phases of my study was to keep a research log. Because I am not revealing any sensitive data or sharing results that might skew the study, I have decided to share my experiences here as well. (Entries will appear in an edited form in order not to influence the study if participants happen to read this blog.)

UPDATE: It’s now 2008-01-02 and with the conclusion of data collection I am now adding back in a paragraph that does discuss specific results below. It begins with the word “Specifically”.

The first round of the study was originally scheduled to conclude tomorrow. However, I have only collected five responses. Out of more than sixty invitations to participate there are now fourteen confirmed participants, with a potential for two to three more. The good news is that their levels of expertise are very much what I had hoped for, which will add to the credibility of the study, though of course their identities will remain anonymous. However, the minimum number of responses I called for in my proposal was twelve, so I plan to send out reminders today and extend the deadline to Friday the 21st at least. It is a difficult time of year to conduct surveys. I knew this would be the case and I know I will need to be flexible in order to finish in time to graduate this May.

Nevertheless, once I received my first three responses I began to organize and prepare data for analysis. Also, I began early data analysis, using Tams Analyzer for OS X to create an initial coding scheme from the first three responses. Already the categories (and thus potential questions) I may include in the second round of the Delphi have already grown beyond my original six. I’m sure I will need to synthesize and condense the results to allow for a manageable and productive second round.

Specifically, there has been a focus on active learning, depth of learning, and differentiated learning, all of which may fall under my category of constructivist learning, as problem solving might, too. There has been some focus on hard fun, as well as the expected discussion of motivation and engagement. The importance (and inherent educational value) of gameplay has also been mentioned. There has been little mention so far of social benefits, other than some discussion of the natural marriage of games and the ZPD. One category discarded during the proposal stage was 21st Century Skills, but those issues are making an appearance in participant answers, particularly risk taking. Role playing has also reappeared in participant answers as well. Note that this early analysis has focused on question 1, which focuses on the potential benefits of MMORPGs in education. I have not yet begun analysis on question 2, which focuses on the potential drawbacks of MMORPGs in education.

This morning, I will be sending an email to the participants thanking those who have completed the first round and prompting others to complete the survey. A few participants who joined later will be receiving their round 1 questions this morning. And finally, a few others I expect might still want to join will receive an invitation or prompt for response.

I also plan to add two most recent responses to my Tams Analyzer project and add their content to my coding scheme for question 1. I will also begin reading and analyzing responses to question 2.

In addition, as I review my methods chapter I am looking ahead to identifying a colleague familiar with the subject matter to serve as a devil’s advocate to the results, and to identifying a colleague familiar with the method to serve as an external auditor.

Walden University Exploits Students for Money

UPDATE 04/23/08: It turns out I didn’t actually want to bring people out of the woodwork who have had troubles with Walden. I know the issues you’ll read about in the comments below can happen at any University. Also, though this was a frustrating time for me, I know I might have been as frustrated or more in a traditional Ph.D. program. Since that time my committee has been heroic in their efforts to push my submissions through the system – and just last night I had my dissertation approved with minor edits following my oral defense. If I can complete all the form and style revisions quickly enough I expect to graduate next month and to walk this summer. I’m thrilled. And I say again unequivocally that Walden was an excellent learning experience for me over five years, despite this one quarter of frustration. I know other individual’s experiences may very, but I can whole heartedly recommend the school and the Educational Technology Ph.D. program. In keeping with good blogging ethics, though, I won’t be removing this post or the comments below. I’ll let this update set the story straight.

Here are my last three posts on twitter about what happened today:

WOO HOO! Got approval from the research office to proceed with my study! Oh wait, that’s not IRB approval. ARRRGGGG! Are u kidding me!?????

The IRB office has had my docs since August 15th… but couldn’t review them in the meantime, or even concurrently with the research office.

I understand working with bureaucracies, but I feel as if I’m being exploited for $. That’s not good education & it’s not good business.

At the earliest I might get approval from the IRB next week (though it will more likely be in ten business days… plus some). If I get approval next week, I will have spent an entire quarter’s time and tuition waiting for approval to proceed with my study. I’ve done less than 40 hours of work on my dissertation in three months time (this was mostly minor – often contradictory – revisions). I don’t even feel like I’m in a Ph.D. program anymore.

Yes, this post is more emotionally charged than most I share here.. particularly the title. It borders on being unprofessional. But, this blog is my forum for my voice to be heard, and as I clearly have no other recourse with the university (and I’m not going to quit and stop paying them at this point), here is where I will voice my criticisms of Walden University. (That phrase should be Google-able.)

Incidentally, I’m looking forward to the end of the quarter evaluations, too.

In any case, I hope this post matters as much as the times I’ve posted about the benefits of Walden’s program. I’ve lead many prospective students their way. I hope this sends others looking elsewhere. I will need to see evidence of change in the approval process to re-endorse this program. I understand I am not alone in these frustrations.

I hope my feelings on this matter level off in the coming months, especially after I’ve finally completed my study. If not, I will be sad that my long journey ends with this sort of experience.

Note: For more details on much time I’ve spent waiting for approval, see my post from the 17th.