Research approach

I pursue five research objectives:

  1. to determine the salient influences to package maintainers' adoption or rejection decisions;
  2. to present these influences in a meaningful way, and develop a terminology of labels for these influences which makes every attempt at being unambigious;
  3. to crystallise a number of implications for practice from these influences, to help increase the rate of diffusion of improved tools and techniques in the Debian Project, to foster competition and progress, and finally to enable the project to scale better with its growth;
  4. to provide stepping stones for future work, by making all data available under a Free licence, speculating research ideas and identifying niches for further work;
  5. to further the use of the Delphi method in FLOSS environment, by highlighting its applicability in the FLOSS context, meticulously documenting the design of my approach, and analysing the performance.

My approach is exploratory and qualitative, centred around a particular type of technique known as the Delphi method.

Introducing the Delphi method

The Delphi method is a moderated group communication technique, optimised to allow a group of people to deal with a complex problem at a distance. W. Timothy Weaver in 1971 nicely coined the underlying principle of the Delphi method as: "several heads are better than one in making subjective conjectures about the future, […] and that experts will make conjectures based upon rational judgement rather than merely guessing […]". James Surowiecki captures the idea in numerous case studies and anecdotes in his book The Wisdom of Crowds.

I intend to use the Delphi method over email, as follows. The process is designed to reduce the effort from each participant to a minimum:

  1. invite a fixed number of participants to the discussion panel;
  2. serve the same question to each panelist and obtain the reply, possibly following up to ask for clarification, if necessary;
  3. anonymise and paraphrase the responses, possibly asking the authors to acknowledge that their positions are still properly represented;
  4. send a collated summary of all responses back to every participant, giving them a chance to revise their previous answer in the light of the group response, meaning everyone can change their opinion without admitting that they were wrong or simply persuaded because they act anonymously;
  5. repeat the process with the same panel and a modified or new question.

It usually takes 2-3 rounds until the group opinion has settled, and consensus, or the degree of differing opinions has been established.

The study depends a lot on the care and effort put in by each participant. I have found a number of sponsors to help compensate the panelists for their time. If you would like to sponsor this research as well, please get in touch with me.

Snowball-sampling a discussion panel

It is crucial to the study for me to assemble a diverse and balanced discussion group. To meet this requirement, I have chosen a snowball sampling strategy to populating the panel: I identify people who seem interested in the topic of this research by scanning discussions on mailing lists and other media and ask them to nominate candidates for the discussion group. I also encourage them to nominate themselves, and to write a bit about why they nominate each person.

From the responses, it should be possible to identify a core group of candidates, namely those who are nominated multiple times or have outstanding recommendations. I then write to each member of the group:

  • telling them about the study,
  • outlining the requirements and estimating the effort,
  • mentioning the compensation,
  • asking them whether they would be interested in participating.

If a candidate is interested, s/he should answer a few (optional) questions about his/her background and involvement in Debian. In addition, interested candidates need to position themselves on the following four spectra:

  1. involved in a team vs. working alone
  2. maintaining many different packages vs. maintaining only a few or mainly similar packages
  3. using similar approaches for all packages vs. using diverse tools
  4. interested in workflow and meta-issues vs. just trying to get work done.

Using these self-ratings, and the number of nominations of each candidate as a weighting factor, I select panelists in a way to maximise diversity across the aforementioned spectra, a technique known as stratified purposeful sampling.

Consulting the oracle

Details on the Delphi study will be available as soon as my thesis has been written up and published. Please come back often.

Table of contents

Please cite this web page as follows: Krafft, Martin F. "Research approach." Process evolution in large open-source projects:. 14 Jun 2011. Martin Krafft, University of Limerick. <>. Last accessed: [DATE ACCESSED].

This page was last modified Tue Nov 2 12:45:31 2010.

Site design and content are copyright © 2005–9 Martin F. Krafft. Please see the imprint.
I shall not be held liable for offending content in pages linking to or linked from site, nor do I endorse their content unless I stated otherwise.

This webpage wouldn't be possible without Debian, Python, rest2web, Apache, and other excellent pieces of Free Software.

Valid XHTML 1.0 Strict Valid CSS