As a product developer you realize that quality has recently been slipping. Customer's are no longer as thrilled with your products, which are now buggy, slow to market, and missing the magic of wow!  A self examination reveals un-structured product development processes that you aren't very proud of.  A loose structure worked well when your organization was younger and smaller, but now you have several products in various phases of development or maintenance and the lack of structure is causing problems.

Everyone concludes it's time to crank up the structure, hire a process pro, bring some consultants in, and execute a top down re-org to insert layers of management above all the moving parts. What used to be a small highly integrated team of multi-disciplined professionals working together evolves into departments of specialized skill-sets matrixed across multiple projects.  You inject the rigor of ISO 9000 derived quality control into the development pipeline by erecting phase gates, compulsory reviews, and formal verification and validation.  Acronyms ending with the word "Board" become the dread of every productive human in the organization.

You get encouragement from upper management because these steps are convincing proof that action is being taken to correct what everyone sees as a chronic problem with quality. 

Screech!

Fast forward a couple of years and you have a constipated organization that can no longer deliver innovation to customers. Everything has a bottleneck and there is a bottleneck for everything.  Product launches are few and far between, and when they do happen there's little or no enthusiasm in the marketplace because the technology is behind the times, depleting to users, and generally bland.  Adding insult to injury, despite all the phase gate checkpoints, compulsory reviews, and formal verification and validation - quality is as bad or worse. Essentially, you increased development costs and schedules and all you got in return was a poke in the eye.

What went wrong?

You needed structure but reached for the wrong kind, you reached for the kind of structure that makes things worse rather than better. High structure comes in various flavorings, some are really good and others are poisonous.  Fixing a broken product development organization involves more than just reaching for structure, it involves selecting the correct structure.

As John Hajdukiewicz from MIT/Honeywell points out, "Innovation and structured processes are at odds with each other and need to be managed with different approaches. Innovation processes require creative isolation from high structure to open degrees of freedom and the design space".

To wrap this up, as we've pointed out many times here before - product development consists of two main parts: You need a vibrant front end for innovation and creative design but you also need a highly structured back end for development. Agile tends to combine the front end and back end, which injects mediocrity into product design.  Waterfall and structured systems engineering processes impose a common work-breakdown flowchart around everything - resulting in numerous problems, one of which is really bad product design.

A good front end has a very special and in many ways "loose" structure. Creative talent (Product Designers) need to be isolated from the high structure used to crank out a red-shifted high quality delivery.

At the back end you want high structure, a machine that runs efficiently and swiftly under intense pressure.  You want the back end to wrap things up quickly, no more than 10 months for most products. Don't try to incorporate a lot of product-level innovation in the back-end, just focus on writing great code.  Also, don't try to iterate major elements of product design at the back end...development will be hosed.  If the back-end doesn't feel a little like a pressure cooker then there's a problem.  If your back-end staff doesn't completely grok the product design then you're also hosed.  This is why so much offshore outsourcing produces garbage.

Great software development floors are run at a constant but intense pace, where best practices are a religion and peer pressure is a force that drives things to be done right.  XP/Scrum practices work well at the back end, but you need a talented "floor boss" to drive quality and productivity.  Organizations that think the only way to control quality is with a heavy verification & validation process generally don't have these disciplines; that said, even great organizations benefit from having independent verification and validation disciplines. 

At the front end, the best innovation is driven from Design not requirements, so your process structure should be centered around Product Design not systems analysis.  Requirements capture and analysis disciplines are necessary but completely insufficient for great product innovation. Design is one-third art, one-third technology, and one-third modern anthropology.  Keep the creative pieces isolated from the pressure cooker of the back end, but make sure you have a "structure for creativity" and assess your design talent at every step.  The most talented designers should be recognized and used on the most visible projects. Less talented designers should get the scraps or be moved away from the creative front end.