.. This document is © 2006 Martin F. Krafft It is available under the terms of the Artistic Licence. ========================================================================== Improving cooperation in volunteer projects ========================================================================== Workflow, communication, tools |mdash| and how to get people to adopt them -------------------------------------------------------------------------- .. |event| replace:: FrOSCon 2006, Sankt Augustin, Germany .. |talkdate| replace:: 24 June 2006 .. |author| replace:: Martin F. Krafft .. |footer| replace:: |event| |bull| |talkdate| .. container:: author Martin F. Krafft * Ph.D. student, University of Limerick, Ireland * Developer with the Debian, Zope, and Plone projects .. container:: event |event| |talkdate| Background ========== * My Ph.D. topic * Origin in the Debian project, main point of focus * Important implications in case of success - open source projects - software development - corporations - ... * So far, mainly ideas and "vapourware" Overview ======== .. class:: current * The breed of volunteers * The need for change * Thoughts on possible improvements * Tools that have made a difference * Tools that have failed * Characteristics of successful tools * Discussion Volunteers ========== In gardening and botanical terminology, a volunteer is a plant that *grows on its own*, rather than being deliberately planted by a human farmer or gardener (`Wikipedia`_). .. _Wikipedia: http://en.wikipedia.org/wiki/Volunteer_%28botany%29 * Seems to square well with volunteers in open-source projects: * developers arrive at working for a project because they want to. * bringing in people to do specific things (for money) rarely works (and can cause serious friction if it does work). Volunteers ========== Less botanical: One who enters into, or offers for, any service *of his own free will* (`Webster's (via dict.die.net)`_). .. _Webster's (via dict.die.net): http://dict.die.net/volunteer/ .. container:: vcenter center *Any* service? Volunteers ========== .. container:: vcenter center I need a volunteer, please... Motivation ========== Sorry; moving on... One who enters into, or offers for, any service *of his own free will* (`Webster's (via dict.die.net)`_). Other works on motivation (e.g. `The Cathedral and the Bazaar`_), we'll concentrate on one main aspect: * Many volunteers in open-source projects *help because they want to* (for whatever reason) * People (and geeks especially) like to do things *their own way* * Put differently: *you cannot tell a volunteer what or how to do things* (or you're not dealing with a voluneer anymore) .. _The Cathedral and the Bazaar: http://www.catb.org/~esr/writings/cathedral-bazaar/ Giving direction ================ If "volunteers do just what they want", how can projects follow a direction? * common vision (e.g. Debian Social Contract) * accepted leader(s), governors, decision makers (e.g. Plone, Apache) * "benevolent dictators" (e.g. Linux, Python, Perl, Ubuntu, Fedora) * "chaotically tolerant" (e.g. Gentoo) * "passive" leadership (e.g. Debian) (this list is not supposed to be flame bait!) Size matters ============ In smaller projects, leaders look up to governors who are decision makers. As the project grows more diverse, * governors tend to fade away. * leaders take a more passive role. * decision makers are only used to resolve conflicts. Diverse projects have a strong self-dynamics. It is impossible to change existing processes *by force* in volunteer projects. Overview ======== * The breed of volunteers .. class:: current * The need for change * Thoughts on possible improvements * Tools that have made a difference * Tools that have failed * Characteristics of successful tools * Discussion Goal ==== .. container:: hcenter vcenter What am I actually trying to do? Revolutionise project leadership? Goal ==== .. container:: hcenter vcenter No. (you can stop worrying now) Goal: improve cooperation ========================= ... where "improve" primarily means "facilitate" or "streamline". Why? What problem am I trying to solve? Stagnation and inflexibility due to high inertia Stagnation and inflexibility in Debian ====================================== * The Debian project turns 14 this year * Processes are in flux, but fundamentals have not changed much * Much time is lost Claim: Debian could be further and more agile (I don't want to paint a black picture though) I have two hours to work on a Debian package, I don't want to spend most of that on learning how a certain package is managed! Overview ======== * The breed of volunteers * The need for change .. class:: current * Thoughts on possible improvements * Tools that have made a difference * Tools that have failed * Characteristics of successful tools * Discussion Improving Cooperation ===================== How can cooperation be facilitated, streamlined? Some ideas: * Technical: version control (VCS), issue tracking, standardisation * Social: tolerance/variety, integration, convincing of technical merits * Organisational: workflow, best practices, policy Example: package maintenance ============================ * Maintaining packages has not changed fundamentally in years * Various bottlenecks call for co-maintenance * Complexity calls for smart(er) patch management * Different approaches and their learning curves, confusion * Manual bug triage The principles should be: Don't do manually what you can automate! Keep it simple, stupid! Top-down vs. bottom-up ====================== * Top-down (in this context) means: * we are at the current state * we need to change the processes to get to the desired state * Bottom-up (in this context) means: * the foundation was geared for the desired state from the start * the processes emerged from the foundation Comparisons: Plone and Debian ============================= Differentiate between projects, take Plone and Debian as examples: ============ ============================= =================================== Criterium Plone Debian ============ ============================= =================================== Age relatively young relatively aged Jumpstart sprang from Zope started from scratch (pioneer) Size small number of contributors large number of contributors VCS usage used *one* VCS from the start tried to introduce VCS later, no standard Bug tracking in flux, standardised now early introduction of ``debbugs`` Workflow simple(r), one product Various, many constituents ============ ============================= =================================== Main points of focus ==================== Debian package maintenance: * Version control * Bug tracking * Quality assurance with distributed contributors * Inherent/emergent workflow Improving cooperation in Debian =============================== Some ideas: * Standardise the use of VCS, but not the type of VCS (if possible) * Switch users to VCS by making VCS more intuitive, not by expecting users to learn * Integrate with other aspects of package maintenance, for example: * ``debian/changelog`` * Bug tracking * Upload coordination The problem with VCS ==================== * VCS are a technical solution to a social problem * generally more geared towards technical soundness than usability * Thus: steep learning curve of concepts * So far no integration with other development processes (in Debian) How to get people to adopt? =========================== We cannot just revolutionise the process. Remember: volunteers don't like to be told how to do things Two strategies for adoption: * seamless integration with existing processes * technical merits of alternatives to attract users Seamless integration ==================== * First phase of my research is the assessment of existing processes * One approach involves "live command-line sessions" on IRC:: i want to work on the mutt package downloading mutt (1.5.9-2) to ./mutt-1.5.9 ... done. i want to fix bug #654321 preparing isolated environment for fixing of bug #654321 ... done. < martin records intent ... < oracle edits a file, fixes the bug i have fixed the bug thank you. please give a brief description of the bug fix. < martin spawns editor-of-choice < oracle describes the bug fix and closes editor fix recorded in change log, what else can i do for you? please prepare a new release < [...] * Other approaches include: mockup interfaces, investigation and exchange with other similarly complex projects, such as the BSDs What to integrate? ================== * Generally, this question should be answered only after extensive research of existing methods. * Some ideas already include the integration of VCS with: * ``debian/changelog`` * Bug tracking * Upload coordination Integration of VCS with ``debian/changelog`` ============================================ * VCS changesets have log messages. * ``debian/changelog`` entries should ideally correspond to one or more changesets each. * Figure out a good, unobtrusive way to reuse log entries Idea: * Use of branches to group related changes between versions * Merging a branch into the main tree requires a commit and thus a log entry * Those log entries (combined with the log entry corresponding to the branch creation) could make up Integration of VCS with ``debbugs`` =================================== * At the moment only through ``debian/changelog`` * Everything else is manual, mainly associating patches with bugs and keeping track of bugs and versions that fixed them. Idea: * One branch per bug, automatic branching (?) * Automatic submission of patches to BTS (if still necessary) The word "automatic" does cause chills among many. Things should be automated the Debian way, as in: facilitated but requiring manual action. Integration of VCS with the upload process ========================================== Problem: maintenance is distributed, but we still need one developer for the upload. This can lead to a myriad of problems, ranging from bottlenecks to quality control. Idea: * Upload requests instead of direct upload * Archive server fetches source, building on *all* architectures * Per-package policy requiring quality control/authenticity certificates Overview ======== * The breed of volunteers * The need for change * Thoughts on possible improvements .. class:: current * Tools that have made a difference * Tools that have failed * Characteristics of successful tools * Discussion Technical merits ================ What are technical merits that will lead users to adopt? Tools that have been successful =============================== * Subversion: CVS-compatible, broad user base, ecological niche * wikis: easy to use, highly flexible * debbugs: e-mail based, simple, ecological niche, Debian-specific * debhelper: modular, consistent, well documented, ecological niche, Debian-specific * alioth (?): cumbersome, but fast turn-around, integrates other tools Successful introduction: debian-installer ========================================= * Project started around 2001 * Consistent use of VCS from the beginning, first CVS, soon SVN * Most contributors didn't want to learn VCS, project decided to keep it very simple * First project to make consistent use of a wiki in Debian Successful introduction: debhelper ================================== * Consistent set of scripts to fill an ecological niche. Compare `this debian/rules file`_ to `this one using debhelper`_. * Logical modularity: one command per packaging aspect * Consistent yet various specification of parameters * Early introduction (1997) * Bottom-up (building block) vs. top-down (c.f. ``cdbs`` (which *uses debhelper*); see `this cdbs rules file`_). .. _this debian/rules file: debian_rules.util-linux .. _this one using debhelper: debian_rules.suspend2 .. _this cdbs rules file: debian_rules.agp Tools that have been less successful ==================================== * GNU arch (and, to some degree, CVS); too complex, buggy * bugzilla, mantis: web-based, too complex (at least for Debian) * cdbs: obscure, badly documented (better now) Unsuccessful introduction: pkg-zope =================================== * Around 100 packages and 30 maintainers (maybe 5 active) * Infected by the hype around distributed VCS, I tried to introduce GNU arch * Despite efforts and documentation, no adoption * Instance of "useful but not usable", developer-centric approach Successful tools, improving cooperation ======================================= What are the characteristics of successful tools? .. container:: vcenter .. What other ways are there to improve cooperation? This is only the beginning... ============================= Thank you for your attention! .. container:: author Martin F. Krafft * http://martin-krafft.net/phd/ .. include:: licence.rst .. include:: .. include:: .. include:: .. include:: .. footer:: |footer|