Rapid Development

Steve McConnell

Mentioned 12

Project managers, technical leads, and Windows programmers throughout the industry share an important concern--how to get their development schedules under control. Rapid Development addresses that concern head-on with philosophy, techniques, and tools that help shrink and control development schedules and keep projects moving. The style is friendly and conversational--and the content is impressive.

More on Amazon.com

Mentioned in questions and answers.

If you could go back in time and tell yourself to read a specific book at the beginning of your career as a developer, which book would it be?

I expect this list to be varied and to cover a wide range of things.

To search: Use the search box in the upper-right corner. To search the answers of the current question, use inquestion:this. For example:

inquestion:this "Code Complete"

Applying UML and Patterns by Craig Larman.

The title of the book is slightly misleading; it does deal with UML and patterns, but it covers so much more. The subtitle of the book tells you a bit more: An Introduction to Object-Oriented Analysis and Design and Iterative Development.

Masters of doom. As far as motivation and love for your profession go: it won't get any better than what's been described in this book, truthfully inspiring story!

Beginning C# 3.0: An Introduction to Object Oriented Programming

This is the book for those who want to understand the whys and hows of OOP using C# 3.0. You don't want to miss it.

alt text

Mastery: The Keys to Success and Long-Term Fulfillment, by George Leonard

It's about about what mindsets are required to reach mastery in any skill, and why. It's just awesome, and an easy read too.

Pro Spring is a superb introduction to the world of Inversion of Control and Dependency Injection. If you're not aware of these practices and their implications - the balance of topics and technical detail in Pro Spring is excellent. It builds a great case and consequent personal foundation.

Another book I'd suggest would be Robert Martin's Agile Software Development (ASD). Code smells, agile techniques, test driven dev, principles ... a well-written balance of many different programming facets.

More traditional classics would include the infamous GoF Design Patterns, Bertrand Meyer's Object Oriented Software Construction, Booch's Object Oriented Analysis and Design, Scott Meyer's "Effective C++'" series and a lesser known book I enjoyed by Gunderloy, Coder to Developer.

And while books are nice ... don't forget radio!

... let me add one more thing. If you haven't already discovered safari - take a look. It is more addictive than stack overflow :-) I've found that with my google type habits - I need the more expensive subscription so I can look at any book at any time - but I'd recommend the trial to anyone even remotely interested.

(ah yes, a little obj-C today, cocoa tomorrow, patterns? soa? what was that example in that cookbook? What did Steve say in the second edition? Should I buy this book? ... a subscription like this is great if you'd like some continuity and context to what you're googling ...)

Database System Concepts is one of the best books you can read on understanding good database design principles.

alt text

Algorithms in C++ was invaluable to me in learning Big O notation and the ins and outs of the various sort algorithms. This was published before Sedgewick decided he could make more money by dividing it into 5 different books.

C++ FAQs is an amazing book that really shows you what you should and shouldn't be doing in C++. The backward compatibility of C++ leaves a lot of landmines about and this book helps one carefully avoid them while at the same time being a good introduction into OO design and intent.

Here are two I haven't seen mentioned:
I wish I had read "Ruminations on C++" by Koenig and Moo much sooner. That was the book that made OO concepts really click for me.
And I recommend Michael Abrash's "Zen of Code Optimization" for anyone else planning on starting a programming career in the mid 90s.

Perfect Software: And Other Illusions about Testing


Perfect Software: And Other Illusions about Testing by Gerald M. Weinberg

ISBN-10: 0932633692

ISBN-13: 978-0932633699

Rapid Development by McConnell

The most influential programming book for me was Enough Rope to Shoot Yourself in the Foot by Allen Holub.

Cover of the book

O, well, how long ago it was.

I have a few good books that strongly influenced me that I've not seen on this list so far:

The Psychology of Everyday Things by Donald Norman. The general principles of design for other people. This may seem to be mostly good for UI but if you think about it, it has applications almost anywhere there is an interface that someone besides the original developer has to work with; e. g. an API and designing the interface in such a way that other developers form the correct mental model and get appropriate feedback from the API itself.

The Art of Software Testing by Glen Myers. A good, general introduction to testing software; good for programmers to read to help them think like a tester i. e. think of what may go wrong and prepare for it.

By the way, I realize the question was the "Single Most Influential Book" but the discussion seems to have changed to listing good books for developers to read so I hope I can be forgiven for listing two good books rather than just one.

alt text

C++ How to Program It is good for beginner.This is excellent book that full complete with 1500 pages.

Effective C++ and More Effective C++ by Scott Myers.

Inside the C++ object model by Stanley Lippman

I bough this when I was a complete newbie and took me from only knowing that Java existed to a reliable team member in a short time

Not a programming book, but still a very important book every programmer should read:

Orbiting the Giant Hairball by Gordon MacKenzie

The Pragmatic programmer was pretty good. However one that really made an impact when I was starting out was :

Windows 95 System Programming Secrets"

I know - it sounds and looks a bit cheesy on the outside and has probably dated a bit - but this was an awesome explanation of the internals of Win95 based on the Authors (Matt Pietrek) investigations using his own own tools - the code for which came with the book. Bear in mind this was before the whole open source thing and Microsoft was still pretty cagey about releasing documentation of internals - let alone source. There was some quote in there like "If you are working through some problem and hit some sticking point then you need to stop and really look deeply into that piece and really understand how it works". I've found this to be pretty good advice - particularly these days when you often have the source for a library and can go take a look. Its also inspired me to enjoy diving into the internals of how systems work, something that has proven invaluable over the course of my career.

Oh and I'd also throw in effective .net - great internals explanation of .Net from Don Box.

I recently read Dreaming in Code and found it to be an interesting read. Perhaps more so since the day I started reading it Chandler 1.0 was released. Reading about the growing pains and mistakes of a project team of talented people trying to "change the world" gives you a lot to learn from. Also Scott brings up a lot of programmer lore and wisdom in between that's just an entertaining read.

Beautiful Code had one or two things that made me think differently, particularly the chapter on top down operator precedence.


@Juan: I know Juan, I know - but there are some things that can only be learned by actually getting down to the task at hand. Speaking in abstract ideals all day simply makes you into an academic. It's in the application of the abstract that we truly grok the reason for their existence. :P

@Keith: Great mention of "The Inmates are Running the Asylum" by Alan Cooper - an eye opener for certain, any developer that has worked with me since I read that book has heard me mention the ideas it espouses. +1

I found the The Algorithm Design Manual to be a very beneficial read. I also highly recommend Programming Pearls.

This one isnt really a book for the beginning programmer, but if you're looking for SOA design books, then SOA in Practice: The Art of Distributed System Design is for you.

For me it was Design Patterns Explained it provided an 'Oh that's how it works' moment for me in regards to design patterns and has been very useful when teaching design patterns to others.

Code Craft by Pete Goodliffe is a good read!

Code Craft

The first book that made a real impact on me was Mastering Turbo Assembler by Tom Swan.

Other books that have had an impact was Just For Fun by Linus Torvalds and David Diamond and of course The Pragmatic Programmer by Andrew Hunt and David Thomas.

In addition to other people's suggestions, I'd recommend either acquiring a copy of SICP, or reading it online. It's one of the few books that I've read that I feel greatly increased my skill in designing software, particularly in creating good abstraction layers.

A book that is not directly related to programming, but is also a good read for programmers (IMO) is Concrete Mathematics. Most, if not all of the topics in it are useful for programmers to know about, and it does a better job of explaining things than any other math book I've read to date.

For me "Memory as a programming concept in C and C++" really opened my eyes to how memory management really works. If you're a C or C++ developer I consider it a must read. You will defiantly learn something or remember things you might have forgotten along the way.


Agile Software Development with Scrum by Ken Schwaber and Mike Beedle.

I used this book as the starting point to understanding Agile development.

Systemantics: How Systems Work and Especially How They Fail. Get it used cheap. But you might not get the humor until you've worked on a few failed projects.

The beauty of the book is the copyright year.

Probably the most profound takeaway "law" presented in the book:

The Fundamental Failure-Mode Theorem (F.F.T.): Complex systems usually operate in failure mode.

The idea being that there are failing parts in any given piece of software that are masked by failures in other parts or by validations in other parts. See a real-world example at the Therac-25 radiation machine, whose software flaws were masked by hardware failsafes. When the hardware failsafes were removed, the software race condition that had gone undetected all those years resulted in the machine killing 3 people.

It seems most people have already touched on the some very good books. One which really helped me out was Effective C#: 50 Ways to Improve your C#. I'd be remiss if I didn't mention The Tao of Pooh. Philosophy books can be good for the soul, and the code.

Discrete Mathematics For Computer Scientists

Discrete Mathematics For Computer Scientists by J.K. Truss.

While this doesn't teach you programming, it teaches you fundamental mathematics that every programmer should know. You may remember this stuff from university, but really, doing predicate logic will improve you programming skills, you need to learn Set Theory if you want to program using collections.

There really is a lot of interesting information in here that can get you thinking about problems in different ways. It's handy to have, just to pick up once in a while to learn something new.

I saw a review of Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools on a blog talking also about XI-Factory, I read it and I must say this book is a must read. Altough not specifically targetted to programmers, it explains very clearly what is happening in the programming world right now with Model-Driven Architecture and so on..

Solid Code Optimizing the Software Development Life Cycle

Although the book is only 300 pages and favors Microsoft technologies it still offers some good language agnostic tidbits.

Managing Gigabytes is an instant classic for thinking about the heavy lifting of information.

My vote is "How to Think Like a Computer Scientist: Learning With Python" It's available both as a book and as a free e-book.

It really helped me to understand the basics of not just Python but programming in general. Although it uses Python to demonstrate concepts, they apply to most, if not all, programming languages. Also: IT'S FREE!

Object-Oriented Programming in Turbo C++. Not super popular, but it was the one that got me started, and was the first book that really helped me grok what an object was. Read this one waaaay back in high school. It sort of brings a tear to my eye...

My high school math teacher lent me a copy of Are Your Lights Figure Problem that I have re-read many times. It has been invaluable, as a developer, and in life generally.

I'm reading now Agile Software Development, Principles, Patterns and Practices. For those interested in XP and Object-Oriented Design, this is a classic reading.

alt text

Kernighan & Plauger's Elements of Programming Style. It illustrates the difference between gimmicky-clever and elegant-clever.

to get advanced in prolog i like these two books:

The Art of Prolog

The Craft of Prolog

really opens the mind for logic programming and recursion schemes.

Here's an excellent book that is not as widely applauded, but is full of deep insight: Agile Software Development: The Cooperative Game, by Alistair Cockburn.

What's so special about it? Well, clearly everyone has heard the term "Agile", and it seems most are believers these days. Whether you believe or not, though, there are some deep principles behind why the Agile movement exists. This book uncovers and articulates these principles in a precise, scientific way. Some of the principles are (btw, these are my words, not Alistair's):

  1. The hardest thing about team software development is getting everyone's brains to have the same understanding. We are building huge, elaborate, complex systems which are invisible in the tangible world. The better you are at getting more peoples' brains to share deeper understanding, the more effective your team will be at software development. This is the underlying reason that pair programming makes sense. Most people dismiss it (and I did too initially), but with this principle in mind I highly recommend that you give it another shot. You wind up with TWO people who deeply understand the subsystem you just built ... there aren't many other ways to get such a deep information transfer so quickly. It is like a Vulcan mind meld.
  2. You don't always need words to communicate deep understanding quickly. And a corollary: too many words, and you exceed the listener/reader's capacity, meaning the understanding transfer you're attempting does not happen. Consider that children learn how to speak language by being "immersed" and "absorbing". Not just language either ... he gives the example of some kids playing with trains on the floor. Along comes another kid who has never even SEEN a train before ... but by watching the other kids, he picks up the gist of the game and plays right along. This happens all the time between humans. This along with the corollary about too many words helps you see how misguided it was in the old "waterfall" days to try to write 700 page detailed requirements specifications.

There is so much more in there too. I'll shut up now, but I HIGHLY recommend this book!

alt text

The Back of the Napkin, by Dan Roam.

The Back of the Napkin

A great book about visual thinking techniques. There is also an expanded edition now. I can't speak to that version, as I do not own it; yet.

Agile Software Development by Alistair Cockburn

Do users ever touch your code? If you're not doing solely back-end work, I recommend About Face: The Essentials of User Interface Design — now in its third edition (linked). I used to think my users were stupid because they didn't "get" my interfaces. I was, of course, wrong. About Face turned me around.

"Writing Solid Code: Microsoft's Techniques for Developing Bug-Free C Programs (Microsoft Programming Series)" by Steve MacGuire.

Interesting what a large proportion the books mentioned here are C/C++ books.

While not strictly a software development book, I would highly recommend that Don't Make me Think! be considered in this list.

As so many people have listed Head First Design Patterns, which I agree is a very good book, I would like to see if so many people aware of a title called Design Patterns Explained: A New Perspective on Object-Oriented Design.

This title deals with design patterns excellently. The first half of the book is very accessible and the remaining chapters require only a firm grasp of the content already covered The reason I feel the second half of the book is less accessible is that it covers patterns that I, as a young developer admittedly lacking in experience, have not used much.

This title also introduces the concept behind design patterns, covering Christopher Alexander's initial work in architecture to the GoF first implementing documenting patterns in SmallTalk.

I think that anyone who enjoyed Head First Design Patterns but still finds the GoF very dry, should look into Design Patterns Explained as a much more readable (although not quite as comprehensive) alternative.

Even though i've never programmed a game this book helped me understand a lot of things in a fun way.

How influential a book is often depends on the reader and where they were in their career when they read the book. I have to give a shout-out to Head First Design Patterns. Great book and the very creative way it's written should be used as an example for other tech book writers. I.e. it's written in order to facilitate learning and internalizing the concepts.

Head First Design Patterns

97 Things Every Programmer Should Know

alt text

This book pools together the collective experiences of some of the world's best programmers. It is a must read.

Extreme Programming Explained: Embrace Change by Kent Beck. While I don't advocate a hardcore XP-or-the-highway take on software development, I wish I had been introduced to the principles in this book much earlier in my career. Unit testing, refactoring, simplicity, continuous integration, cost/time/quality/scope - these changed the way I looked at development. Before Agile, it was all about the debugger and fear of change requests. After Agile, those demons did not loom as large.

One of my personal favorites is Hacker's Delight, because it was as much fun to read as it was educational.

I hope the second edition will be released soon!

You.Next(): Move Your Software Development Career to the Leadership Track ~ Michael C. Finley (Author), Honza Fedák (Author) link text

I've been arounda while, so most books that I have found influential don't necessarily apply today. I do believe it is universally important to understand the platform that you are developing for (both hardware and OS). I also think it's important to learn from other peoples mistakes. So two books I would recommend are:

Computing Calamities and In Search of Stupidity: Over Twenty Years of High Tech Marketing Disasters

Working Effectively with Legacy Code is a really amazing book that goes into great detail about how to properly unit test your code and what the true benefit of it is. It really opened my eyes.

I am looking for studies about applied code quality, comparing costs before and after. They should show a clear benefit in cost (or maybe a negative effect of too much cost). I need hard facts like (entirely fictional):

After we added static code analysis to our build, the number of defects reduced to half. So we saved approx. 10 developer days effort per iteration on bug fixing. Extra cost by buying and setting up the analysis was x$. Development was slowed down 0.1% by obeying the analysis results, increasing the total development effort by 5 days per iteration. During the first half year the initial cost was returned. etc. Now we save approx. y$ per iteration.

I only know one such story given in Code Complete 2nd Ed. It is talking about Boeing that defects decreased after adding reviews during the QA process (AFAIK). Unfortunately most shops would not compare with Boeing, so studies from Boeing do not count.

Do you know such studies or do you have any hard data from your shop?

There is a related question, but does not give any hard data.

The books Code Complete 2 and Rapid Development have lots of examples from real life case studies and experiments. Almost everything they argue is backed up with hard facts.

I need a urgent crash course on software development. We are about to start a major project at work. The project is a huge web application and I have been chosen to lead the team. The problem now is, I am not familiar with the software development process. The way I used to go about my development is think it and do it, but now I won't be able to do that because I have to collaborate with others. I will very much appreciate if someone can help with a link to a crash course. Thanks.

You want to look at Construx. It's owned by Steve McConnell, the author of Code Complete.

You'll probably want to read Rapid Development.

Bug tracking, source control and code reviews are your friend. You might want to read Agile Software Development and adapt some of those processes to your needs.

It seems to be an article of faith that top-flight programmers are several times as productive as mediocre programmers. Can anyone point to studies or reports which support this with hard data?

It's hard to find hard data on anything related to programming, but I'll second the suggestion by Shiraz Bhaiji that you look at Code Complete by Steve McConnell. You might want to look at the next book he wrote, Rapid Development, as well. It also has some statistics.

As comments above pointed out, you need a good definition of productivity. Some people say that experienced and inexperienced programmers both write about the same number of lines of code per day but experienced programmers keep more lines of code per day. The inexperienced programmers will spin their wheels and rewrite most of today's code tomorrow.

There's some reference on this point in The Mythical Man-Month; I don't have my copy with me right now to see if the information is well-sourced, though.

I've never written functional specs, I prefer to jump into the code and design things as I go. So far its worked fine, but for a recent personal project I'm writing out some specs which describe all the features of the product, and how it should 'work' without going into details of how it will be implemented, and I'm finding it very valuable.

What are your thoughts, do you write specs or do you just start coding and plan as you go, and which practice is better?

If you are working in an XP (or similar) environment, you'll use stories to guide development along with lots of unit and hallway useability testing (I've drunk the Kool-Aid, I guess).

However, there is one area where a spec is absolutely required: when coordinating with an external team. I had a project with a large insurance company where we needed to have an agreement on certain program behaviors, some aspects of database design and a number of file layouts. Without the spec, I was wide open to a creative interpretation of what we had promised. These were good people - I trusted them and liked working with them. But still, without that spec it would have been a death march. With the spec, I could always point out where they had deviated from the agreed-to layout or where they were asking for additional custom work ($$!). If working with a semi-antagonistic relationship, the spec can save you from even worse: a lawsuit.

Oh yes, and I agree with Kieveli: "jumping right to code" is almost never a good idea.

For someone who "jumps into the code" and "design[s] as they go", I would say writing anything including a functional spec is better than your current methods. A great deal of time and effort can be saved if you take the time to think it through and design it before you even start.

  • Requirements help define what you need to make.
  • Design helps define what you are planning on making.
  • User Documentation defines what you did make.

You'll find that most places will have some variation of these three documents. The functional spec can be lumped into the design document.

I'd recommend reading Rapid Development if you're not convinced. You truely can get work done faster if you take more time to plan and design.

Background: All of my experience in developing software and managing projects has been related to applications (not counting a few trivial hacked-together websites here and there). Process-wise, I typically start off templating Rapid Development and the Software Project Survival Guide and then tweaking the plan to suit the needs, resources, etc. of the project. This has worked well for me.

Soon I'm going to start building a website to supplement an application that I've been working on that will require a lot more planning and foreknowledge than sites I've made previously (think something akin to Newgrounds). There will be 2 other developers on the project, and we're all on separate continents (only reason I mention this is to dissuade answers involving "small team doesn't need any kind of formal process").

My question is, what changes should I anticipate when moving from the planning of an application to the planning of a website (though the site will involve web applications as well. using django.)?

I typically distinguish between web sites and web applications. If the site is mostly content and a UI to find and manipulate the content, then I consider that to be a web site.

If, on the other hand, you actually do things on the site, or affect other parts of the world, then that's an application that happens to be delivered over the web, hence a "web application" to my mind.

I would largely develop a web application the same way as any other: best practices, separation of concerns, automated unit tests if not test-driven development, etc. The only place you should be seeing a difference is in the nature of the UI.

Depending on your platform, you may not have the same tools available for testing the actual Web UI as you'd be used to for testing a desktop UI. To my mind, if you've thoroughly covered all the non-UI code, then this is not such a big deal.

OTOH, if you are testing a UI that is rich in JavaScript, then you may want to look into any testing tools for JavaScript. I believe there are some, but I'm afraid I can't tell you where they are. In the absence of automated tools, I would create test pages that exercise the individual controls or other JavaScript classes individually.

I am currently a college student and I don't feel that any of my classes have touched on the Software Development Life Cycle (SDLC) nearly enough.

I've been interning a company for a few years and I've been learning about SDLC from the internship, but I wanted to poll the crowd so I can branch out from that.

What SLDC resources (books, websites, magazines, newsletters, etc.) do you use?

These two are great resources on SPM and methods therein:
Software Project Survival Guide - Steve McConnell
Rapid Development - Steve McConnell

This is a good resource on SCRUM, an agile method of development
Agile Software Development with Scrum

Does anyone know of excellent (in your opinion) books/websites that go over in details how to pragmatically use a bug/task/project management tool to help improve software development process? Ideally, I would like "from-the-trench" kind of resource, not theory.

Examples of such mgmt tools are: JIRA/Confluence combo, Visual Studio Team System / Sharepoint, FogBugz, etc.

I think I'm looking for books that are like the 3 Pragmatic book series (source control, unit testing, and build automation) Come to think of it, I wish the pragmatic guys also released a book on how to integrate project mgmt tools to their software development process.

I know of a few that might fit as the answers to my question:

Painless Project Management with FogBugz, Second Edition Visual Studio Team System: Better Software Development for Agile Teams Software Engineering with Microsoft Visual Studio Team System

Much like how we learn different programming languages to broaden our thought process, I dont really mind what tool the book/website uses. What I'm most interested in is HOW the tool is integrated into software development process to help improve it.

When I installed MS-TFS 2008, I started to get myself prepared to use the Agile Process Guidance template, that was shipped with the TFS server. With a little googling, I went through Mike Cohn materials:

  1. I watched his conference in youtube "sponsored by google":
  2. Read his book "Agile Estimating and Planning"
  3. Watched the video series in his website: http://www.mountaingoatsoftware.com/presentations-tag/video-recorded

I was very happy while absorbing and eating the techniques he uses with the teams and how agile and Scrum are both such a great software process/methodology, until I saw Mike answer a question regarding the architect role and talking about the requirements document... at that point everything start falling apart, due to the following:

  1. Last year I had been assigned to make full analysis "including requirements gathering" for big project, "very high priority project".
  2. Within 2 months of hard work, dedication and commitment I delivered the whole analysis with full satisfaction of the customer and my BOSS and ZERO amendments.
  3. Later on, the project entered the architecture, development ... phases.
  4. Due to the fact that the system included many competitive and exciting features, I requested patenting it and its going in the process...

So imagine you are the kind of person who used to love facing all kinds of challenges and returning with excellent experience and results for the stakeholders and yourself, How fairly agile and scrum processes will credit and admit your talent and passion while the scrum master/coach treat the team as one unit that accomplish user stories and converge through trial and error approach??!!!!

With those dark thoughts about agile and Scrum I found many people "anti agile" and on top of them is "Crispin Rogers Johnson":

That guy made an anti-statement for everything Mike Cohn used to talk about.

I really don't know what to do next! So any guidance will be appreciated.


For every project there is a correct development strategy. If NASA used agile or scrum they would not have the 1 in 100,000 lines of code defect rates that satelite system requires. You can't release and iterate away the bugs. If you do you wind up watching your system crash into Mars.

That said, you shouldn't have to spec out in excruciating detail every nuance of a website which is related to some Hollywood mogul's dog or fansite. That's something you'd iterate and fix while the customer gave you feedback.

There is a balance to everything and every a balance. Perhaps you should read a book like, Rapid Development. While it's slightly dated so is the mythical man month but both have lasting values. What these should teach is that there is no one way to do things but many. The project should dictate your approach not some evangelical software guru.

Disclaimer: This is in no way meant to imply that non-Agile are for "real" uses while Agile should be relegated to Scruffy McPointless projects.

I'm looking for a book that tells you how to split up a software (or anything; doesn't have to be software-specific) task or project into discrete chunks, and helps elucidate all the issues and nuances that aren't apparent when you just start programming.

Should I get a book on use cases? If so, what?

What I'm looking for is a book that can help me go from a simple story like "The user wants to be able to specify an email address for themselves, or specify that they don't have an email address" to a UI diagram, along with a list of coding tasks, and testing requirements (so writing unit tests becomes very straightforward).

I'm tired of starting on a task only to be flummoxed in the middle of it because it wasn't thought through properly.


While not specific to software, the PMI's PMBOK is a great overview on how the project planning process works.


Steve McConnell's book Rapid Development: Taming Wild Software Schedules is used in my university's undergraduate course on Software Engineering Process and Project Management. It's very much a companion to Code Complete. CC deals with the coding issues, while Rapid Development talks about management issues.

I highly recommend looking into mindmapping - here's a good book: Mapping Inner Space: Learning and Teaching Visual Mapping

and open source mindmapping tool: FeeMind

but don't get stuck in wrestling with any software - you can use crayons. mindmapping is sort of a WBS (work breakdown structure) and sort of free formed organized thinking. When breaking a BIG application (project) to smaller pieces you often find that you have repeated functionality down different branches, very defined and very undefined areas and external interactions to make the system work...graphically representing this with some pics also helps

Scott Ambler's The Object Primer is also a useful overview for the topics you mention.

I've seen lots of references to such studies, but can't find any links right now.

Steve McConnell's book Rapid Development is an excellent treatment of software quality assurance practices. Although not a "study" per se, it does bring together the conclusions reached by many studies on software quality. (The boxed area at the top of page 72, "How Much Does It Cost Not to Find a Defect?", gives a direct answer to your question).

For the studies themselves, the book's bibliography is a good resource. Also, if you have memberships to ACM and/or IEEE, their libraries are where many of the studies mentioned in Rapid Development can be found.

In addition to McConnell, other prolific authors in the area of software quality include Barry Boehm, Capers Jones, Gerald Weinberg and Tom Gilb.

So as a developer I am seeking to improve not only my coding skills but my design and management skills. Because of this I'm starting to pay more attention to software engineering practices but i'm not sure where agile development fits into the picture. I can appreciate agility in projects but I wonder whether this conflicts with the traditional ways of software engineering practices and research.

If you haven't read Rapid Development, I highly recommend it. What makes it super pertinent now is how it leaves off just as it should start talking about Agile. So it gives you a thorough background of "how we got here". When you read the sections on iterative development and iterative prototyping, it will make any additional research you do on Agile that much more meaningful.