Doc. no.:  N1631=04-0071
Date:         11 April 2004
Project:     Programming Language C++
Reply to:   David Abrahams <dave@boost-consulting.com>
                 Beman Dawes <bdawes@acm.org>
                 Jeremy Siek <jsiek@osl.iu.edu>

Electronic Review Process

Introduction
Process Overview
Prerequisites
What to include in Review Comments
Results
Notes for Proposal Authors
Notes for Review Managers
FAQ

Introduction

This document describes an electronic review process designed to augment the physical review process used by the C++ committee's working groups (WG's) at meetings to reach a consensus.

Under ISO, national body, and committee rules, official action on an issue or proposal is taken only by formal vote of the full committee.

To ensure that proposals which come before the full committee are truly ready for a formal vote, working group review occurs among domain experts meeting physically. Proposals are only brought forward to the full committee when the WG chair determines that sufficient working group consensus exists to warrant a full committee vote.

The committee wishes to extend this process to allow reviews to occur electronically between meetings. The intent is to improve the process by ensuring:

A separate "review" mailing list reflector will be set up to encourage wide participation in electronic reviews.

Regardless of whether review occurs electronically between meetings or in an actual working group meeting, final acceptance of any proposal always requires a formal vote by the full committee.

Process Overview

  1. Proposal authors request on the review reflector that a review be scheduled. WG chairs may also nominate issues (with their proposed resolutions) for electronic review.
  2. The appropriate WG chair appoints a review manager.
  3. The review manager coordinates the review schedule with authors and WG chairs via private email. Reviews are usually scheduled to last at least 14 days, to allow full participation, but may be shorter or longer as experience and the needs of specific proposals dictate.
  4. The review manager begins the review period by posting a notice of the review, including a summary of the proposal, on both the review reflector and appropriate WG reflector.
  5. Review comments are posted by review reflector participants.
  6. After the conclusion of the review period, the review manager posts results on the review and WG reflectors.
  7. At the next meeting, the WG chair in consultation with the working group makes the final determination to submit the proposal to the full committee for a formal vote.

Prerequisites

What to include in Review Comments

A reviewer's comments may be brief or lengthy, but basically the review manager needs your evaluation of the proposal. You can even vote with no additional commentary, though comments are valuable and strongly encouraged. If you identify problems along the way, please note if they are minor, serious, or showstoppers.

Here are some points you might want to touch on in your review:

And finally, every review should answer this question: "Should the proposal be accepted by the committee?"  Be sure to say this explicitly so that your other comments don't obscure your overall opinion. It may also be helpful if your answer is complete enough to give guidance for further work. Possible responses:

Results

At the conclusion of the review period, the review manager will post a message with the final tallies of the reviewer's votes and a summary of the main issues that were raised during the review. This vote plays the same role as the straw polls that occur during working group meetings; it provides a mechanism for measuring consensus. As always, the final determination of whether or not sufficient consensus exists to warrant forwarding a proposal to the full committee is made by working group chairs in consultation with their WG's.

If there are suggestions or conditions that must be met in a revised version of a proposal, they should be stated in the review summary.

Notes for Proposal Authors

A proposal should remain stable during the review period; it will just confuse and irritate reviewers if a revised document appears during a review period.

Authors should respond to reviewer's detailed comments. For example, if a reviewer suggests some change in the proposal, the authors might:

Notes for Review Managers

Before a proposal can be scheduled for electronic review, someone not connected with it must volunteer to be the "Review Manager" for the proposal.

The review manager for an electronic review performs similar functions to a working group chair during in-person working group reviews; they coordinate the process and tally the votes for whether or not enough consensus exists to forward the proposal to the full committee for a formal vote. They also mediate if discussion becomes overheated.

It is expected that working group chairs may act as review managers for proposals relevant to their working group. WG chairs may ask someone else to act as review manager when the chair is the proposal's author, or to spread the workload.

The review manager:

In other words, it is the review manager's responsibility to make sure the review process works smoothly.

FAQ

Who can participate in an Electronic Review? Anyone with access to the review reflector. As with any committee reflector traffic, committee officers monitor discussion to ensure acceptable use. And as with other reviews, WG chairs factor in any divergence between straw voters and voting members when determining if enough consensus exists to warrant a full committee vote.

Why are the review periods so short/long? The Boost experience with electronic reviews is that there is a tension between the desire to give reviewers plenty of time to prepare reviews, and practical factors such as reviewers delaying until near the end of the review period, workload for submitters and review managers, and lack of discussion focus when review periods are lengthy. The length of review periods can be adjusted as committee experience with electronic reviews grows.

Why aren't the reviews scheduled right after between meeting mailings? The initial strategy is to spread the committee's workload out over time, rather than concentrate it in a few specific time periods, and to decouple reviews from other events like mailings to simplify the process. Schedule timeslots can be adjusted as committee experience with electronic reviews grows.

Why aren't documents to be reviewed posted on the committee web site? Logistics. We don't want the webmaster to become a bottleneck or to be overloaded. Perhaps committee web site procedures can be worked out, or a permanent committee Wiki can be established, but we don't want to delay progress until these administrative details are resolved.

The prerequisites are tilted toward mature proposals. Won't this inhibit discussion of "half-ready" proposals? Yes, and that is intended. The individual working group reflectors are the appropriate place for discussions of "good ideas", "trial balloons", "half-ready" proposals, and the like. Electronic reviews involve enough expenditure of committee effort that they are best reserved for mature proposals, or for less-mature proposals important enough to justify electronic review.

How can an electronic review replace all meeting review time? It can't. The goal of electronic reviews isn't to eliminate all meeting review time, but rather to allow  meetings to be more productive. Electronic review allows authors to refine proposals, meeting participants to vote in confidence, and with minimal time be spent in meetings on review aspects which can be handled well electronically.


Revised 11 April, 2004