Research approach
I have two research objectives:
- to determine the salient influences to package maintainers' adoption or rejection decisions, develop an unambiguous and orthogonal (uncorrelated) terminology of factors to capture these influences, and identify clusters of factors to produce a framework.
- to apply this framework in the context of a number of previously adopted or rejected tools or techniques, as well as some which have only recently appeared on the radar screens, and to asses to what degree such instantiations of the framework reflect the real-world adoption rates.
If you dislike the use of the word "framework" (as I do), or you do not know what I am talking about, please have a look at my explanation of what I mean with framework in my research.
My approach is exploratory and qualitative, centred around a particular type of technique known as the Delphi method.
Introducing the Delphi method
The Delphi method is a moderated group communication technique, optimised to allow a group of people to deal with a complex problem at a distance. W. Timothy Weaver in 1971 nicely coined the underlying principle of the Delphi method as: "several heads are better than one in making subjective conjectures about the future, […] and that experts will make conjectures based upon rational judgement rather than merely guessing […]". James Surowiecki captures the idea in numerous case studies and anecdotes in his book The Wisdom of Crowds.
I intend to use the Delphi method over email, as follows. The process is designed to reduce the effort from each participant to a minimum:
- invite a fixed number of participants to the discussion panel;
- serve the same question to each panelist and obtain the reply, possibly following up to ask for clarification, if necessary;
- anonymise and paraphrase the responses, possibly asking the authors to acknowledge that their positions are still properly represented;
- send a collated summary of all responses back to every participant, giving them a chance to revise their previous answer in the light of the group response, meaning everyone can change their opinion without admitting that they were wrong or simply persuaded because they act anonymously;
- repeat the process with the same panel and a modified or new question.
It usually takes 2-3 rounds until the group opinion has settled, and consensus, or the degree of differing opinions has been established.
The study depends a lot on the care and effort put in by each participant. I have found a number of sponsors to help compensate the panelists for their time. If you would like to sponsor this research as well, please get in touch with me.
Snowball-sampling a discussion panel
It is crucial to the study for me to assemble a diverse and balanced discussion group. To meet this requirement, I have chosen a snowball sampling strategy to populating the panel: I identify people who seem interested in the topic of this research by scanning discussions on mailing lists and other media and ask them to nominate candidates for the discussion group. I also encourage them to nominate themselves, and to write a bit about why they nominate each person.
From the responses, it should be possible to identify a core group of candidates, namely those who are nominated multiple times or have outstanding recommendations. I then write to each member of the group:
- telling them about the study,
- outlining the requirements and estimating the effort,
- mentioning the compensation,
- asking them whether they would be interested in participating.
If a candidate is interested, s/he should answer a few (optional) questions about his/her background and involvement in Debian. In addition, interested candidates need to position themselves on the following four spectra:
- involved in a team vs. working alone
- maintaining many different packages vs. maintaining only a few or mainly similar packages
- using similar approaches for all packages vs. using diverse tools
- interested in workflow and meta-issues vs. just trying to get work done.
Using these self-ratings, and the number of nominations of each candidate as a weighting factor, I select panelists in a way to maximise diversity across the aforementioned spectra, a technique known as stratified purposeful sampling.
Consulting the oracle
I cannot disclose any information on the Delphi study at this moment. This is in part because the process will be in flux throughout, but mostly because I cannot risk the participants looking beyond the task of each round, which would invariably influence their responses.
Application and verification of the framework
In the final phase of my research, I seek to test the framework I developed in close collaboration with the Delphi panel. Even though applying it prescriptively to a new tool or technique to see how it performs, I leave that for further research as it is outside the scope of my current research. Instead, I intend to investigate the past diffusions of two classes of methods in the Debian Project: package build helpers (debhelper, cdbs, and yada), and patch management and version control systems for package maintenance (dpatch, quilt, SVN, and Git).
