A Modest Proposal
Summary
There’s been considerable discussion in various places about the auto-resolution of UNCO bugs, and the longstanding desire to have a READY state, with arguments on both sides as to whether the local hack for Bugzilla is worth the effort. To quote, er, someone: “Bugzilla was created to serve the needs of mozilla.org. We need to make that it continues to do so.” These changes would involve local hacks, some of which I’m sure could be lifted from Red Hat’s Bugzilla code or other places.
Desired Outcomes
- Provide a means to get more people involved on the ground level of bug triage by making it easier to do useful things. This solution is threefold, starting with simplifying the initial process to confirming that the bug is reproducable and not an obvious dupe, making canconfirm easier to get and giving it the additional ability to mark WFM or DUP, and granting that priv far more widely.
- Free up more experienced and knowledgable QA people to work on a smaller set of bugs and drive bugs into a fixable state.
- Help developers by creating a pool of bugs that are ready to be fixed, with appropriate testcases/design specs in place.
Implementation
This requires workflow and thus code-level changes to bugzilla. This is a scary thought for some, but we need to be willing to let mozilla.org’s needs drive Bugzilla work, at least in part. One thing is that canconfirm needs to be able to mark bugs as DUPLICATE and WORKSFORME.
Thanks to help from Mike Beltzner, master of the flowchart, and feedback from a few others, I’ve put together a modified workflow for dealing with bugs in such a way that we can achieve the desired outcomes. This mitigates the need for periodic expiry, but I think we may still want to expire bugs after a few months if still unconfirmed and abandoned.
Please check this out, and comment here for now, and we can wiki up the results.
The modest proposal isn’t entirely clear about what this new READY state means. I’ve seen the term bandied about, mostly by bz, but no one’s explicitly said what the state means (as opposed to the other bug states).
It seems like the “testcase required” check needs to be earlier, since it often requires a testcase to be confident that a bug is not duplicate or invalid (although in many cases a bug can be demonstrated to be duplicate or invalid without a testcase).
Comments:
1) Marking a bug worksforme should involve testing on the same OS at least… I’ve seen VMS-specific bugs worksforme’d by Windows folks.
2) I’m not convinced that filing with canconfirm should default the bug to NEW. Then again, I guess we’re redefining what NEW means, so that might be ok.
3) How is the “bug is valid?” decision get made? This seems like something that would require module owner feedback. Also, “Text case required?” comes before you can tell whether a bug is valid — for a typical layout/dom/whatever bug (which is where we need testcases) there’s no way to tell what the heck is going on from the several tens or hundreds of K of markup and script the bugs usually start with.
So I propose checkin for enhancement request. If not, check for testcase. Then try to decide validity.
4) It also seems like for enhancements we need a way to request owner approval.
I would like to read more detailed description of “bug is valid?” and “Test case required?”.
questions about a form of the presentation:
Could you form the contets of decision blocks as questions?
Can you strighten the line between “bug is valid?” and “invalid”?
(there is a jump around the “No” label)
The very first choice-diamond (“filed with editbugs”) should be removed. If a user with editbugs filed as UNCO then it was presumably so that the triage team would pick it up and look for dupes or attempt to recreate.
This is partly semantics, but it seems to me what is needed is more a state (“In Triage”?) between UNCO and NEW, not a state between NEW and ASSI.
To me, at least, NEW indicates a valid, should-be-fixed bug that is more-or-less ready to fix and is simply waiting for a developer with time to fix it.
NEW does not mean a bug that is simply “not a dupe or obvious WFM but which still needs more triaging.”
I can’t speak for the rest of the tree, but in Camino we leave bugs UNCO util they are generally reproducible *Camino* bugs or RFEs that have gotten the nod of pinkerton or smfr or sometimes the consensus of the triage team. When we send bugs over to Layout and such, we send them over as UNCO, partly because we don’t know Layout dupes very well but also because we don’t know if it’s a valid bug per specs or just a bad site.
NEW has a very clear meaning in terms of a bug’s validity (and to laymen generally), and I don’t think we should redefine the meaning of “new”.
I am completely signed up to making Bugzilla serve mozilla.org; and I hope no further convincing of anyone should be required there.
Can you please elaborate on “make canconfirm easier to get”? The current rules don’t seem like a very high barrier to me:
http://www.gerv.net/hacking/before-you-mail-gerv.html#bugzilla-permissions
Or are you saying more people should be encouraged to apply for it earlier in their Bugzilla life?
Am I right in saying you are requesting exactly two code changes – changes to the abilities of canconfirm, and addition of READY? The code changes to allow those with canconfirm to do a little bit more are pretty trivial. Adding READY is a bit more work; I’d need to investigate exactly how much more.
You need to elaborate on “bug is valid?”. How is this decided, and by whom?
In order to work out exactly what states and keywords are required, it might be good to label some of the states with queries. So, for example, if a module owner needs to approve all NEW enhancements, and then they move on to requiring a spec, how do they query for all “unapproved” NEW enhancements? Do we need a “mo-approved” keyword? How do you search for all bugs which need testcases making? Do we use a “needs-testcase” keyword?
Smokey: I agree that redefining NEW creates problems, but the alternative is to mass-change 50,000 bugs from NEW to the new TRIAGING state. Which probably wouldn’t be ideal. Having a READY state, starting with no bugs in it, allows it to be populated by humans in an incremental fashion.
Personally, I think you should add READY, and rename NEW to CONFIRMED.
Then modules such as layout who want a fine distinction can tell between ‘there is a bug here somewhere’ and ‘bug explicitly identified’, and everyone else who simply used NEW as CONFIRMED, can do so.
A few comments: An enhancement request shouldn’t necessarily go to NEW as the diagram shows. The enhancement has to:
Not already be fixed.
Not be invalid.
Also, having canconfim or editbugs doesn’t mean that all your bugs should be filed as NEW. The choice should still be there.
re #8
> An enhancement request shouldn’t necessarily go to NEW
> The enhancement has to:
> Not already be fixed.
drop “is rfe” from graph, reword “can reproduce” to be inclusive of rfes?
> Not be invalid.
I think the intent is for the more value- than mechanical-flavored INVA and WONT decisions to not be the concern of the UNCO tier in the optimal workflow for bugs and rfes alike, though INVA does rather fall in between in that regard.
Is the text of a state (READY, NEW, etc…) is coded into the database, or is it a code that Bugzilla just associates with the right human-readable state ?
Provided the second case is the right one, then having “TRIAGING” associated in place of current “NEW” should’nt be very difficult, and “NEW” in place of proposed “READY”. No need to bugspam everybody… But a _single_ mail should be sent to every account in bugzilla to explain the change of nomenclature.
> Free up more experienced and knowledgable QA people to work on a smaller set of bugs and drive bugs into a fixable state.
If it’s any consolation, this is one of the stated goals of the current QA automation effort.
There was a time when QA staff were able to help with triage, but sadly that hasn’t been the case for a while now. This isn’t due to the volume of bugs per se, but rather the demands of daily testing (with frequent release testing, it seems) being spread between too few people. People like Marcia, Tracy, and Bob just don’t have the time to do concerted sweeps through Bugzilla like in times past.
By automating the really mind-numbing daily tests, i.e. BFTs and smoketests, we should free up the QA staff to help police Bugzilla again.
I like the look of this in general. Certainly having this process documented is a good thing, so far as I know the current process doesn’t exist in this form and it would help new people starting work in triage or development.
A couple of concerns, I think a check for the bug validity may be needed sooner. For example in this process, filing a bug that only contains “Firefox is silly!” would end up getting confirmed before the validity check is performed!
Secondly for enhancements a design spec is made up before the bug is “READY”. Such a thing might be useful in some of the more complex normal bugs.
For me READY seems a bit unclear. Isn’t what we want a UNCO->CONFIRMED->TRIAGED thus changing NEW to CONFIRMED and add TRIAGED instead of READY.
Enhancement bugs are requests and should never have the UNCO status (there’s nothing unconfirmed about a request).
They should have status NEW by default and either be chanched to DUPE or WONTFIX or ACCEPTED (a new status for enhancement bugs only).
From ACCEPTED the status can only be changed to DUPE, WONTFIX or ASSIGNED.
It wouldn’t be a bad idea to have an “enhancement day” (like bugs day) once in a while to make sure enhancement bugs don’t rot forever.
The reporter shouldn’t get the impression that the “great” idea he/she had is something we simply don’t care about.
Why isn’t there an OPEN state, that tells people that a bug is actively worked upon ? The name READY doesn’t really ring a bell to me.
Hi!
How are you?