Enterprise Integration Patterns

Gregor Hohpe, Bobby Woolf

Mentioned 32

Would you like to use a consistent visual notation for drawing integration solutions? "Look inside the front cover." Do you want to harness the power of asynchronous systems without getting caught in the pitfalls? "See "Thinking Asynchronously" in the Introduction." Do you want to know which style of application integration is best for your purposes? "See Chapter 2, Integration Styles." Do you want to learn techniques for processing messages concurrently? "See Chapter 10, Competing Consumers and Message Dispatcher." Do you want to learn how you can track asynchronous messages as they flow across distributed systems? "See Chapter 11, Message History and Message Store." Do you want to understand how a system designed using integration patterns can be implemented using Java Web services, .NET message queuing, and a TIBCO-based publish-subscribe architecture? "See Chapter 9, Interlude: Composed Messaging." Utilizing years of practical experience, seasoned experts Gregor Hohpe and Bobby Woolf show how asynchronous messaging has proven to be the best strategy for enterprise integration success. However, building and deploying messaging solutions presents a number of problems for developers. " Enterprise Integration Patterns " provides an invaluable catalog of sixty-five patterns, with real-world solutions that demonstrate the formidable of messaging and help you to design effective messaging solutions for your enterprise. The authors also include examples covering a variety of different integration technologies, such as JMS, MSMQ, TIBCO ActiveEnterprise, Microsoft BizTalk, SOAP, and XSL. A case study describing a bond trading system illustrates the patterns in practice, and the book offers a look at emerging standards, as well as insights into what the future of enterprise integration might hold. This book provides a consistent vocabulary and visual notation framework to describe large-scale integration solutions across many technologies. It also explores in detail the advantages and limitations of asynchronous messaging architectures. The authors present practical advice on designing code that connects an application to a messaging system, and provide extensive information to help you determine when to send a message, how to route it to the proper destination, and how to monitor the health of a messaging system. If you want to know how to manage, monitor, and maintain a messaging system once it is in use, get this book. 0321200683B09122003

More on Amazon.com

Mentioned in questions and answers.

If you could go back in time and tell yourself to read a specific book at the beginning of your career as a developer, which book would it be?

I expect this list to be varied and to cover a wide range of things.

To search: Use the search box in the upper-right corner. To search the answers of the current question, use inquestion:this. For example:

inquestion:this "Code Complete"

Applying UML and Patterns by Craig Larman.

The title of the book is slightly misleading; it does deal with UML and patterns, but it covers so much more. The subtitle of the book tells you a bit more: An Introduction to Object-Oriented Analysis and Design and Iterative Development.

Masters of doom. As far as motivation and love for your profession go: it won't get any better than what's been described in this book, truthfully inspiring story!

Beginning C# 3.0: An Introduction to Object Oriented Programming

This is the book for those who want to understand the whys and hows of OOP using C# 3.0. You don't want to miss it.

alt text

Mastery: The Keys to Success and Long-Term Fulfillment, by George Leonard

It's about about what mindsets are required to reach mastery in any skill, and why. It's just awesome, and an easy read too.

Pro Spring is a superb introduction to the world of Inversion of Control and Dependency Injection. If you're not aware of these practices and their implications - the balance of topics and technical detail in Pro Spring is excellent. It builds a great case and consequent personal foundation.

Another book I'd suggest would be Robert Martin's Agile Software Development (ASD). Code smells, agile techniques, test driven dev, principles ... a well-written balance of many different programming facets.

More traditional classics would include the infamous GoF Design Patterns, Bertrand Meyer's Object Oriented Software Construction, Booch's Object Oriented Analysis and Design, Scott Meyer's "Effective C++'" series and a lesser known book I enjoyed by Gunderloy, Coder to Developer.

And while books are nice ... don't forget radio!

... let me add one more thing. If you haven't already discovered safari - take a look. It is more addictive than stack overflow :-) I've found that with my google type habits - I need the more expensive subscription so I can look at any book at any time - but I'd recommend the trial to anyone even remotely interested.

(ah yes, a little obj-C today, cocoa tomorrow, patterns? soa? what was that example in that cookbook? What did Steve say in the second edition? Should I buy this book? ... a subscription like this is great if you'd like some continuity and context to what you're googling ...)

Database System Concepts is one of the best books you can read on understanding good database design principles.

alt text

Algorithms in C++ was invaluable to me in learning Big O notation and the ins and outs of the various sort algorithms. This was published before Sedgewick decided he could make more money by dividing it into 5 different books.

C++ FAQs is an amazing book that really shows you what you should and shouldn't be doing in C++. The backward compatibility of C++ leaves a lot of landmines about and this book helps one carefully avoid them while at the same time being a good introduction into OO design and intent.

Here are two I haven't seen mentioned:
I wish I had read "Ruminations on C++" by Koenig and Moo much sooner. That was the book that made OO concepts really click for me.
And I recommend Michael Abrash's "Zen of Code Optimization" for anyone else planning on starting a programming career in the mid 90s.

Perfect Software: And Other Illusions about Testing

TITLE Cover

Perfect Software: And Other Illusions about Testing by Gerald M. Weinberg

ISBN-10: 0932633692

ISBN-13: 978-0932633699

Rapid Development by McConnell

The most influential programming book for me was Enough Rope to Shoot Yourself in the Foot by Allen Holub.

Cover of the book

O, well, how long ago it was.

I have a few good books that strongly influenced me that I've not seen on this list so far:

The Psychology of Everyday Things by Donald Norman. The general principles of design for other people. This may seem to be mostly good for UI but if you think about it, it has applications almost anywhere there is an interface that someone besides the original developer has to work with; e. g. an API and designing the interface in such a way that other developers form the correct mental model and get appropriate feedback from the API itself.

The Art of Software Testing by Glen Myers. A good, general introduction to testing software; good for programmers to read to help them think like a tester i. e. think of what may go wrong and prepare for it.

By the way, I realize the question was the "Single Most Influential Book" but the discussion seems to have changed to listing good books for developers to read so I hope I can be forgiven for listing two good books rather than just one.

alt text

C++ How to Program It is good for beginner.This is excellent book that full complete with 1500 pages.

Effective C++ and More Effective C++ by Scott Myers.

Inside the C++ object model by Stanley Lippman

I bough this when I was a complete newbie and took me from only knowing that Java existed to a reliable team member in a short time

Not a programming book, but still a very important book every programmer should read:

Orbiting the Giant Hairball by Gordon MacKenzie

The Pragmatic programmer was pretty good. However one that really made an impact when I was starting out was :

Windows 95 System Programming Secrets"

I know - it sounds and looks a bit cheesy on the outside and has probably dated a bit - but this was an awesome explanation of the internals of Win95 based on the Authors (Matt Pietrek) investigations using his own own tools - the code for which came with the book. Bear in mind this was before the whole open source thing and Microsoft was still pretty cagey about releasing documentation of internals - let alone source. There was some quote in there like "If you are working through some problem and hit some sticking point then you need to stop and really look deeply into that piece and really understand how it works". I've found this to be pretty good advice - particularly these days when you often have the source for a library and can go take a look. Its also inspired me to enjoy diving into the internals of how systems work, something that has proven invaluable over the course of my career.

Oh and I'd also throw in effective .net - great internals explanation of .Net from Don Box.

I recently read Dreaming in Code and found it to be an interesting read. Perhaps more so since the day I started reading it Chandler 1.0 was released. Reading about the growing pains and mistakes of a project team of talented people trying to "change the world" gives you a lot to learn from. Also Scott brings up a lot of programmer lore and wisdom in between that's just an entertaining read.

Beautiful Code had one or two things that made me think differently, particularly the chapter on top down operator precedence.

K&R

@Juan: I know Juan, I know - but there are some things that can only be learned by actually getting down to the task at hand. Speaking in abstract ideals all day simply makes you into an academic. It's in the application of the abstract that we truly grok the reason for their existence. :P

@Keith: Great mention of "The Inmates are Running the Asylum" by Alan Cooper - an eye opener for certain, any developer that has worked with me since I read that book has heard me mention the ideas it espouses. +1

I found the The Algorithm Design Manual to be a very beneficial read. I also highly recommend Programming Pearls.

This one isnt really a book for the beginning programmer, but if you're looking for SOA design books, then SOA in Practice: The Art of Distributed System Design is for you.

For me it was Design Patterns Explained it provided an 'Oh that's how it works' moment for me in regards to design patterns and has been very useful when teaching design patterns to others.

Code Craft by Pete Goodliffe is a good read!

Code Craft

The first book that made a real impact on me was Mastering Turbo Assembler by Tom Swan.

Other books that have had an impact was Just For Fun by Linus Torvalds and David Diamond and of course The Pragmatic Programmer by Andrew Hunt and David Thomas.

In addition to other people's suggestions, I'd recommend either acquiring a copy of SICP, or reading it online. It's one of the few books that I've read that I feel greatly increased my skill in designing software, particularly in creating good abstraction layers.

A book that is not directly related to programming, but is also a good read for programmers (IMO) is Concrete Mathematics. Most, if not all of the topics in it are useful for programmers to know about, and it does a better job of explaining things than any other math book I've read to date.

For me "Memory as a programming concept in C and C++" really opened my eyes to how memory management really works. If you're a C or C++ developer I consider it a must read. You will defiantly learn something or remember things you might have forgotten along the way.

http://www.amazon.com/Memory-Programming-Concept-C/dp/0521520436

Agile Software Development with Scrum by Ken Schwaber and Mike Beedle.

I used this book as the starting point to understanding Agile development.

Systemantics: How Systems Work and Especially How They Fail. Get it used cheap. But you might not get the humor until you've worked on a few failed projects.

The beauty of the book is the copyright year.

Probably the most profound takeaway "law" presented in the book:

The Fundamental Failure-Mode Theorem (F.F.T.): Complex systems usually operate in failure mode.

The idea being that there are failing parts in any given piece of software that are masked by failures in other parts or by validations in other parts. See a real-world example at the Therac-25 radiation machine, whose software flaws were masked by hardware failsafes. When the hardware failsafes were removed, the software race condition that had gone undetected all those years resulted in the machine killing 3 people.

It seems most people have already touched on the some very good books. One which really helped me out was Effective C#: 50 Ways to Improve your C#. I'd be remiss if I didn't mention The Tao of Pooh. Philosophy books can be good for the soul, and the code.

Discrete Mathematics For Computer Scientists

Discrete Mathematics For Computer Scientists by J.K. Truss.

While this doesn't teach you programming, it teaches you fundamental mathematics that every programmer should know. You may remember this stuff from university, but really, doing predicate logic will improve you programming skills, you need to learn Set Theory if you want to program using collections.

There really is a lot of interesting information in here that can get you thinking about problems in different ways. It's handy to have, just to pick up once in a while to learn something new.

I saw a review of Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools on a blog talking also about XI-Factory, I read it and I must say this book is a must read. Altough not specifically targetted to programmers, it explains very clearly what is happening in the programming world right now with Model-Driven Architecture and so on..

Solid Code Optimizing the Software Development Life Cycle

Although the book is only 300 pages and favors Microsoft technologies it still offers some good language agnostic tidbits.

Managing Gigabytes is an instant classic for thinking about the heavy lifting of information.

My vote is "How to Think Like a Computer Scientist: Learning With Python" It's available both as a book and as a free e-book.

It really helped me to understand the basics of not just Python but programming in general. Although it uses Python to demonstrate concepts, they apply to most, if not all, programming languages. Also: IT'S FREE!

Object-Oriented Programming in Turbo C++. Not super popular, but it was the one that got me started, and was the first book that really helped me grok what an object was. Read this one waaaay back in high school. It sort of brings a tear to my eye...

My high school math teacher lent me a copy of Are Your Lights Figure Problem that I have re-read many times. It has been invaluable, as a developer, and in life generally.

I'm reading now Agile Software Development, Principles, Patterns and Practices. For those interested in XP and Object-Oriented Design, this is a classic reading.

alt text

Kernighan & Plauger's Elements of Programming Style. It illustrates the difference between gimmicky-clever and elegant-clever.

to get advanced in prolog i like these two books:

The Art of Prolog

The Craft of Prolog

really opens the mind for logic programming and recursion schemes.

Here's an excellent book that is not as widely applauded, but is full of deep insight: Agile Software Development: The Cooperative Game, by Alistair Cockburn.

What's so special about it? Well, clearly everyone has heard the term "Agile", and it seems most are believers these days. Whether you believe or not, though, there are some deep principles behind why the Agile movement exists. This book uncovers and articulates these principles in a precise, scientific way. Some of the principles are (btw, these are my words, not Alistair's):

  1. The hardest thing about team software development is getting everyone's brains to have the same understanding. We are building huge, elaborate, complex systems which are invisible in the tangible world. The better you are at getting more peoples' brains to share deeper understanding, the more effective your team will be at software development. This is the underlying reason that pair programming makes sense. Most people dismiss it (and I did too initially), but with this principle in mind I highly recommend that you give it another shot. You wind up with TWO people who deeply understand the subsystem you just built ... there aren't many other ways to get such a deep information transfer so quickly. It is like a Vulcan mind meld.
  2. You don't always need words to communicate deep understanding quickly. And a corollary: too many words, and you exceed the listener/reader's capacity, meaning the understanding transfer you're attempting does not happen. Consider that children learn how to speak language by being "immersed" and "absorbing". Not just language either ... he gives the example of some kids playing with trains on the floor. Along comes another kid who has never even SEEN a train before ... but by watching the other kids, he picks up the gist of the game and plays right along. This happens all the time between humans. This along with the corollary about too many words helps you see how misguided it was in the old "waterfall" days to try to write 700 page detailed requirements specifications.

There is so much more in there too. I'll shut up now, but I HIGHLY recommend this book!

alt text

The Back of the Napkin, by Dan Roam.

The Back of the Napkin

A great book about visual thinking techniques. There is also an expanded edition now. I can't speak to that version, as I do not own it; yet.

Agile Software Development by Alistair Cockburn

Do users ever touch your code? If you're not doing solely back-end work, I recommend About Face: The Essentials of User Interface Design — now in its third edition (linked). I used to think my users were stupid because they didn't "get" my interfaces. I was, of course, wrong. About Face turned me around.

"Writing Solid Code: Microsoft's Techniques for Developing Bug-Free C Programs (Microsoft Programming Series)" by Steve MacGuire.

Interesting what a large proportion the books mentioned here are C/C++ books.

While not strictly a software development book, I would highly recommend that Don't Make me Think! be considered in this list.

As so many people have listed Head First Design Patterns, which I agree is a very good book, I would like to see if so many people aware of a title called Design Patterns Explained: A New Perspective on Object-Oriented Design.

This title deals with design patterns excellently. The first half of the book is very accessible and the remaining chapters require only a firm grasp of the content already covered The reason I feel the second half of the book is less accessible is that it covers patterns that I, as a young developer admittedly lacking in experience, have not used much.

This title also introduces the concept behind design patterns, covering Christopher Alexander's initial work in architecture to the GoF first implementing documenting patterns in SmallTalk.

I think that anyone who enjoyed Head First Design Patterns but still finds the GoF very dry, should look into Design Patterns Explained as a much more readable (although not quite as comprehensive) alternative.

Even though i've never programmed a game this book helped me understand a lot of things in a fun way.

How influential a book is often depends on the reader and where they were in their career when they read the book. I have to give a shout-out to Head First Design Patterns. Great book and the very creative way it's written should be used as an example for other tech book writers. I.e. it's written in order to facilitate learning and internalizing the concepts.

Head First Design Patterns

97 Things Every Programmer Should Know

alt text

This book pools together the collective experiences of some of the world's best programmers. It is a must read.

Extreme Programming Explained: Embrace Change by Kent Beck. While I don't advocate a hardcore XP-or-the-highway take on software development, I wish I had been introduced to the principles in this book much earlier in my career. Unit testing, refactoring, simplicity, continuous integration, cost/time/quality/scope - these changed the way I looked at development. Before Agile, it was all about the debugger and fear of change requests. After Agile, those demons did not loom as large.

One of my personal favorites is Hacker's Delight, because it was as much fun to read as it was educational.

I hope the second edition will be released soon!

You.Next(): Move Your Software Development Career to the Leadership Track ~ Michael C. Finley (Author), Honza Fedák (Author) link text

I've been arounda while, so most books that I have found influential don't necessarily apply today. I do believe it is universally important to understand the platform that you are developing for (both hardware and OS). I also think it's important to learn from other peoples mistakes. So two books I would recommend are:

Computing Calamities and In Search of Stupidity: Over Twenty Years of High Tech Marketing Disasters

Working Effectively with Legacy Code is a really amazing book that goes into great detail about how to properly unit test your code and what the true benefit of it is. It really opened my eyes.

I am new to Messaging and want to know the difference between ActiveMQ, Mule, ServiceMix and Camel

Anyone knows how each of these product is different?

Thanks in advance !

EDIT: Also would like to know any good place/resource to learn these things.

ActiveMQ is a message broker which implements the JMS API and supports a number of cross language clients and network protocols. It lets you implement queues or topics and write listeners or subscribers to respond to queue events.

Mule and ServiceMix are open source ESB (enterprise service bus). An ESB has capabilities beyond JMS: queuing is key, but so are transformation, routing, logging, security, etc.

Apache Camel is an implementation of the patterns in Enterprise Integration Patterns. It can use JMS and other components to put the ideas from that book into place.

JMS is fundamental to the other technologies, like JDBC is the foundation for Hibernate, iBatis, etc.

JMS is a Java API and a specification and TCK (part of Java EE). ActiveMQ is a particular implementation of it.

Camel can use ActiveMQ (and Camel is bundled inside the ActiveMQ broker so you can easily route to and from JMS to the other components Camel supports).

Camel doesn't use Mule or ServiceMix directly; though ServiceMix uses Camel as its preferred routing engine and EIP implementation. Camel does have a ton of different components though that use various different technologies.

I am an inexperienced Java developer trying to wrap my head around some fundamental middleware/SOA concepts and technologies, specifically:

  • Service-Oriented Architecture (SOA)
  • Message-Oriented Middleware (MOM)
  • Message Queue
  • Apache Camel
  • Mule
  • EJBs
  • Endpoints & Routes
  • Service Bus/ESB
  • JMS

After looking each of these up online/on Wikipedia, I was able to get (for the most part) decent definitions for each of these. What I am not understanding is how all of these technologies/concepts work together on the backend to provide a 2nd/business tier solution.

Can someone please give an example of an architecture that would use all of these technologies/concepts, and explain what role each of them play in the overall solution? Once I see a working example I'm sure it will help me connect most of the dots.

Edit: Since I added the bounty, I've had several answers that suggest reading books. Although I appreciate all feedback here, I simply can't part ways with 300 reputation points for an answer that, essentially, boils down to "RTM" (especially when I'm flat broke and can't afford the manual!) To reiterate, the bounty and definitive answer will go to someone who can hit all of these bullets in a meaningful, practical example. This does not have to be a middleware compendium!!! Just a paragraph or two that shows how all these can be used together in harmony to produce a Java business-tier solution. Thanks again.

You mix a lot of different concepts and technologies with different abstraction levels. But all of your concepts have something to do with (enterprise) application integration. I will try to comment your definitions:

  • Service-Oriented Architecture (SOA)
    SOA provides a set of principles and methodologies to integrate existing applications as loosely coupled units. From the Enterprise Integration Patterns (see below): "SOAs blur the line between integration and distributed aplications".
  • Service Bus/ESB
    The ESB is a main concept of SOA to reduce the dependencies within the applications in a SOA. Instead of a lot of dependencies between the applications each application is connected to the ESB.
  • Message-Oriented Middleware (MOM)
    MOM is a infrastructure for sending and receiving messages between distributed systems. This is used to integrate applications. MOM was the golden hammer before the SOA hype came up. Since both are useful, big integration suites provides both ESB and MOM (or use MOM inside their ESB).
  • Message Queue
    A message queue is just a technical detail aspect in MOM architecture. When message sending/receiving is decoupled, message are stored in queues until the recipient is ready.
  • Apache Camel
    When the book Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions hit the market, some software solutions have been created which provides implementation for the patterns in this book. Apache Camel is one of them. Camel is also a part of Apache ServiceMix which is also an open source ESB. FuseSource and Talend are packaging Apache ServiceMix, Apache Camel and Apache Active MQ (MOM) to bundles with commercial support.
  • Mule
    Mule is also an open source ESB and integration platform.
  • EJBs
    From Wikipedia: Enterprise JavaBeans (EJB) is a managed, server-side component architecture for modular construction of enterprise applications. This means EJB is a component within an application and has primary nothing to do with integrating applications.
  • Endpoints & Routes
    When you work with Apache Camel you are designing routes between endpoints, see a tutorial. In short, message are entering/leaving your system via endpoints and are processed in a flow defined by a route.
  • JMS
    JMS or Java Message Service is a Message Oriented Middleware (MOM) with an standardized Java API.

I've got the GOF sitting on my desk here and I know there must be some kind of design pattern that solves the problem I'm having, but man I can't figure it out.

For simplicities sake, I've changed the name of some of the interfaces that I'm using.

So here's the problem, on one side of the wire, I've got multiple servers that send out different types of messages. On the other side of the wire I have a client that needs to be able to handle all the different types of messages.

All messages implement the same common interface IMessage. My problem is, when the client gets a new IMessage, how does it know what type of IMessage its received?

I supposed I could do something like the following, but this just FEELS awful.

TradeMessage tMessage = newMessage as TradeMessage;
if (tMessage != null)
{
    ProcessTradeMessage(tMessage);
}

OrderMessage oMessage = newMessage as OrderMessage;
if (oMessage != null)
{
    ProcessOrderMessage(oMessage);
}

The second thought, is to add a property to IMessage called MessageTypeID, but that would require me to write something like the following, which also FEELS awful.

TradeMessage tMessage = new TradeMessage();
if (newMessage.MessageTypeID == tMessage.MessageTypeID)
{
    tMessage = newMessage as TradeMessage;
    ProcessTradeMessage(tMessage); 
}

OrderMessage oMessage = new OrderMessage();
if (newMessage.MessageTypeID == oMessage.MessageTypeID)
{
    oMessage = newMessage as OrderMessage;
    ProcessOrderMessage(oMessage);
}

I know this general problem has been tackled a million times, so there has to be a nicer way of solving the problem of having a method that takes an interface as a parameter, but needs different flow control based on what class has implemented that interface.

You might want to take a look through Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf. It has a good catalog of patterns for message processing.

Can you please suggest some books on Software Architecture, which should talk about how to design software at module level and how those modules will interact. There are numerous books which talks about design patterns which are mostly low level details. I know low level details are also important, but I want list of good design architecture book.

Please also suggest some books which talks about case studies of software architecture.

I think this is the book that came to mind when I first read this question. It talks about various architectural styles like pipes-and-filters, blackboard systems, etc. It's an oldie, and I'll let you judge whether it's a 'goodie'.

Pattern Oriented Software Architecture

I also particularly like these two, especially the first. The second starts to dig into lower level design patterns, but it's still awesome in various spots:

Enterprise Integration Patterns

Patterns of Enterprise Application Architecture

I hope these are what you had in mind.

Where can you get knowledge about software architecture? One place is your experience building systems. Another is conversations with other developers or reading their code. Yet another place is books. I am the author of a book on software architecture (Just Enough Software Architecture) but let me instead point you to some classics:

  • Software Architecture in Practice (Bass, Clements, Kazman). This book from the Software Engineering Institute (SEI) describes how architects should think about problems. It describes the importance of quality attributes (performance, security, modifiability, etc.) and how to make tradeoffs between them, since you cannot maximize all of them.
  • Documenting Software Architectures (lots of SEI/CMU authors). The title of this book is a bit scary, because many people are trying to avoid writing shelfware documents. But the wonderful thing about the book is that it describes the standard architectural styles / patterns, notations for describing structure and behavior, and a conceptual model of understanding architectures. All these are valuable even if you only ever sketch on a whiteboard.
  • Software Systems Architecture (Rosanski and Woods). Goes into detail about how to think about a system from multiple perspectives (views). What I like particularly is that it gives checklists for ensuring that a particular concern (say security) has been handled.
  • Essential Software Architecture (Gorton). Small, straightforward book on IT architecture. Covers the different kinds of things you'll see (databases, event busses, app servers, etc.)

That's just a short list and just because I didn't list something doesn't mean it's a bad book. If you are looking something free to read immediately, I have three chapters of my book available for download on my website.

For message based passing systems, what are your "message design patterns" e.g.

  • Limit directed messages (i.e. specific destination)

  • Avoid long cascade chains (i.e. react to MsgA with MsgB, MsgC etc.)

  • Have a system "heartbeat" message

Other examples?

If you are implementing a message based system, I suggest reading the canonical resource to get insight on messaging architectures: Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe y Bobby Woolf.

A short summary of each pattern is available online at http://www.eaipatterns.com/toc.html At the end of the page two case studies are available.

The book is a great resource, you will find there problems and situations you don't even imagine before, with a good analysis of the strategy to solve it.

I like reading open-source code to see how particular problems have been solved, or simply to study how to create nice and clean code.

I was wondering if there actually was any good book out there that goes through excerpts of different (preferably in Java) open-source projects to show how patterns have been used and implemented, some nifty tricks, etc.?

Update: I took design patterns as an example, but it’s not really what I am after. I read a lot of open-source code, and I often find interesting ways of doing things which I didn’t think of before – my question is whether there is a book that shows some “cool” open-source code, a bit like how Clean Code uses FitNesse code to illustrate refactoring and other points.

Well, you can read just book about open source project, and authors can show what and how. E.g. I read Pro Spring 2.5 and there is even chapter called "Spring Patterns" with explanation of why and how they call files, directories, design patterns they use and invite you to use etc. Probably take a look at the books about open source projects you investigate.

UPDATE (just that paragraph insert to extend answer:) : as I told previously, you can look through the books about open source projects and find there information how this project is developed, architectural issues etc. Once I looked for real good example of "Open/Closed Principle" using, theory is nice but I would not say that it is so clear how to use it. The only book which helped me was Expert Spring MVC and Web Flow (I already pointed you to google books with that chapter I read). I didn't read the whole book, but I am sure it contains a lot of such things (what and how was used to make Spring done). From book Clean Code: A Handbook of Agile Software Craftsmanship I remember that there is a chapter, when they take a code from one open source project and make it better (unfortunately open source does not mean that it is good enough); that book also contain reference to how some other projects where written (e.g. jUnit). I don't know any book, which is completely dedicated to any project, there are just books where is "theory", and books about project which can have some explanation of what is inside. I would recommend books from APress (Pro and Expert series, afaik "beginning" books does not contain such explanation), and Packt Publishing - their main topic is open source.

UPDATE 2 (just can't stop :) Just remembered new series of books from OReilly "Beautiful ...." - I never read it, but I looked through. It actually consists of different stories behind development, real stories from life. I see there is also Beautiful Code: Leading Programmers Explain How They Think - I should contain what you need. Other "beautiful" books can be worth of your time too.

About patterns in general there are a lot of good books, and it depends on what exactly are you looking for. E.g. Head First Design Patterns is quite good to explain common patterns in fun way. If you need more, you can take a look at Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions to see how different applications, systems etc. can be integrated (and how open source projects do it too). Etc. Just specify what patterns or tricks do you want to read about, community will make its suggestion ;)

Some web links out of my bookmarks:

  1. Anti Patterns Catalog - know what you are not going to do!
  2. Design Patterns and Martin Fowler's Catalog - that's shortly about what is good to use and know about
  3. SOA patterns - community site for SOA patterns (integration etc.)

And few more books to take a look at:

  1. Head First Object-Oriented Analysis and Design
  2. Agile Software Development, Principles, Patterns, and Practices
  3. Code Complete - this book is just a bible of how to write code...

people, who develop open source definitely know what are those books about ;)

Hope that helps

UPDATE (25-05-2011) Just found one book, which I definitely going to buy, and I think it perfectly fits your question:

The Architecture of Open Source Applications
enter image description here

Architects look at thousands of buildings during their training, and study critiques of those buildings written by masters. In contrast, most software developers only ever get to know a handful of large programs well — usually programs they wrote themselves — and never study the great programs of history. As a result, they repeat one another’s mistakes rather than building on one another’s successes. This book’s goal is to change that. In it, the authors of twenty-five open source applications explain how their software is structured, and why. What are each program's major components? How do they interact? And what did their builders learn during their development? In answering these questions, the contributors to this book provide unique insights into how they think. Electronic versions of The Architecture of Open Source Applications for e-readers will be available for download soon.

I've been looking at Mass Transit for a couple of months now, and I'm really intrigued by the possibilities. However, I don't seem to be able to get the concepts quite right. I've looked the code, and I've gone through the documentation, but I just don't feel that I"m understanding it.

The examples in general show;

    Bus.Initialize( sbc =>
                       {
                           sbc.UseMsmq( );
                           sbc.VerifyMsmqConfiguration( );
                           sbc.UseMulticastSubscriptionClient( );
                           sbc.ReceiveFrom( "msmq://localhost/myqueue" );
                       } );

Now, I understand what this is doing, but I don't think my brain is taking the concept further than this. Here's what I do understand;

  • Messages can be published from the software, and subscribed to within the service bus for action/s to be performed on receipt of that message.
  • The Service Bus itself, sits on top of a messaging queue (either RabbitMQ or MSMQ in MT)

I just want to understand a little more about this. I don't think I"m getting it. Do I need a server configured, listening? Do I set it up in my software then just publish messages, they get picked up and processed from within?

First off, getting start with MassTransit...

The idea is that you have multiple systems communicating. Now that's just messaging and doesn't really require MassTransit. A talks to B. When we start talking about pub/sub it can become more interesting. A publishes msg CreateOrder which B is listening for. When B receives that CreateOrder message it can take whatever steps is required to handle a new order. This leaves the services decoupled, the only interaction point is a pretty simple message, the CreateOrder.

Now the joy of pub/sub is that A and B are going back and forth for a while, and we have C that wants to listen into CreateOrder messages so it can prepare stock for shipping before B completes all its tasks. We can drop C into the bus, it subscribes to the CreateOrder message and neither A nor B need to change any code. This can be done while they are actively sending messages back and forth. You need to upgrade one of the members involved? Just stop that service, drop in the new one, and restart it, allowing it to pick it where it left off.

If you have more questions on this topic, I'd try hitting up the mailing list. I'd like to believe we are rather responsive when we can be. Additionally you can hit up a couple related questions and books ...

Enterprise Integration Patterns is a great book, even if was written more with Java in mind.

I'm looking forward to start developing a new server side enterprise communication framework in Java and I'm wondering if anyone knows a good book on the subject? Some best practices and advice would be welcome.

Thanks.

I've searched for a while for a good book which covers server designed patterns. I'm looking for something along the lines of Gang of Four.

Concepts include:

-- Threaded vs Process vs combo based solutions
-- How to triage requests properly. i.e. I expect only limited requests from any domain, so I may only allocate a certain number of workers per domain.
-- Worker timeouts
-- poll/select/epoll use cases
-- And those things I don't know!

Any suggestions please!

Thanks!

Advanced Programming in the Unix Environment, 2nd Edition is a fantastic resource for learning the details of Unix systems programming. It's extremely well-written (one of my favorite books in the English Language), the depth is excellent, and the focus on four common environments (at the time of publication) help ensure that it is well-rounded. It's not too badly out of date -- new features in newer operating systems may be fantastic for specific problems, but this book really covers the basics very well.

The downside, of course, is that APUE2nd misses out on some fantastic third-party tools such as libevent, which can make programming sockets-based servers significantly easier. (And automatically picks the 'best' of select(2), poll(2), epoll(4), kpoll, and Windows event handling, for the platform.)

As for choosing between threads and processes, it comes down to: how much memory sharing do you want / need between tasks? If each process can run relatively isolated, processes provide better memory protection and no speed penalty. If processes need to interact with each other's objects, or objects 'owned' by a single thread, then threads provide better primitives for sharing data. (But many would argue that the shared memory of threads is an invitation to fun and exciting bugs. It Depends.)

Two very useful books:

The book Enterprise Integration Patterns provides a consistent vocabulary and visual notation to describe large-scale integration solutions across many implementation technologies. It also explores in detail the advantages and limitations of asynchronous messaging architectures. You will learn how to design code that connects an application to a messaging system, how to route messages to the proper destination and how to monitor the health of a messaging system. The patterns in the book are technology-agnostic and come to life with examples implemented in different messaging technologies, such as SOAP, JMS, MSMQ, .NET, TIBCO and other EAI Tools.

I am trying to use a DocumentDb write as a part of a transaction like below -

using (var scope = new TransactionScope)
{
//first transaction

//write to document db

//third transaction
}

I observed that if the third transaction fails, documentDb write is not rolled back and I still see the document in the collection. The first transaction (NEventStore in this case) rolls back perfectly. Does anyone know if DocumentDb supports TrnasactionScope. What if I have a nested transaction?

Thanks!

Edit: So looks like TransactionScope is not supported with DocumentDb and it knows nothing about them. Is there a way to make DocumentDb transactions part of an external transaction from C#? Has anyone come across this use case before?

Edit 2: Follow-up question and answer here as suggested

Lots of SQL users don't understand what to do where transactions are not available.

You should implement compensation logic your own or use frameworks like Windows Workflow Foundation. Compensation logic is related to Enterprise Integration Patterns. You also may use correlation ID pattern to check if big operation was done.

SQL people manage big operations in the same way when it is necessary to make long running transaction. https://www.amazon.com/Enterprise-Integration-Patterns-Designing-Deploying/dp/0321200683/ref=sr_1_1?ie=UTF8&qid=1480917322&sr=8-1&keywords=integration+patterns

I'm building a T4 template that will help people construct Azure queues in a consistent and simple manner. I'd like to make this self-documenting, and somewhat consistent.

  1. First I made the queue name at the top of the file, the queue names have to be in lowercase so I added ToLower()

  2. The public constructor uses the built-in StorageClient API's to access the connection strings. I've seen many different approaches to this, and would like to get something that works in almost all situations. (ideas? do share)

  3. I dislike the unneeded HTTP requests to check if the queues have been created so I made is a static bool . I didn't implement a Lock(monitorObject) since I don't think one is needed.

  4. Instead of using a string and parsing it with commas (like most MSDN documentation) I'm serializing the object when passing it into the queue.

  5. For further optimization I'm using a JSON serializer extension method to get the most out of the 8k limit. Not sure if an encoding will help optimize this any more

  6. Added retry logic to handle certain scenarios that occur with the queue (see html link)

  7. Q: Is "DataContext" appropriate name for this class?

  8. Q: Is it a poor practice to name the Queue Action Name in the manner I have done?

What additional changes do you think I should make?

public class AgentQueueDataContext
{
    // Queue names must always be in lowercase
    // Is named like a const, but isn't one because .ToLower won't compile...
    static string AGENT_QUEUE_ACTION_NAME = "AgentQueueActions".ToLower();

  static bool QueuesWereCreated { get; set; }

    DataModel.SecretDataSource secDataSource = null;

    CloudStorageAccount cloudStorageAccount = null;
    CloudQueueClient cloudQueueClient = null;
    CloudQueue queueAgentQueueActions = null;

    static AgentQueueDataContext()
    {
        QueuesWereCreated = false;
    }

    public AgentQueueDataContext() : this(false)
    {
    }
    public AgentQueueDataContext(bool CreateQueues)
    {
        // This pattern of setting up queues is from:
        // ttp://convective.wordpress.com/2009/11/15/queues-azure-storage-client-v1-0/
        //
        this.cloudStorageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
        this.cloudQueueClient = cloudStorageAccount.CreateCloudQueueClient();
        this.secDataSource = new DataModel.SecretDataSource();

        queueAgentQueueActions = cloudQueueClient.GetQueueReference(AGENT_QUEUE_ACTION_NAME);

        if (QueuesWereCreated == false || CreateQueues)
        {
            queueAgentQueueActions.CreateIfNotExist();
            QueuesWereCreated = true;
        }
    }

  // This is the method that will be spawned using ThreadStart
   public void CheckQueue()
    {
        while (true)
        {
            try
            {
                CloudQueueMessage msg = queueAgentQueueActions.GetMessage();

                bool DoRetryDelayLogic = false;

                if (msg != null)
                {
                    // Deserialize using JSON (allows more data to be stored)
                    AgentQueueEntry actionableMessage = msg.AsString.FromJSONString<AgentQueueEntry>();

                    switch (actionableMessage.ActionType)
                    {
                        case AgentQueueActionEnum.EnrollNew:
                            {
                                // Add to 
                                break;
                            }
                        case AgentQueueActionEnum.LinkToSite:
                            {
                                // Link within Agent itself

                                // Link within Site

                                break;
                            }
                        case AgentQueueActionEnum.DisableKey:
                            {
                                // Disable key in site

                                // Disable key in AgentTable (update modification time)

                                break;
                            }
                        default:
                            {
                                break;
                            }
                    }

                    //
                    // Only delete the message if the requested agent has been missing for 
                    // at least 10 minutes
                    //
                    if (DoRetryDelayLogic)
                    {
                        if (msg.InsertionTime != null)
                            if (msg.InsertionTime < DateTime.UtcNow + new TimeSpan(0, 10, 10))
                                continue;

                        // ToDo: Log error: AgentID xxx has not been found in table for xxx minutes.   
                        //                  It is likely the result of a the registratoin host crashing.
                        //                  Data is still consistent.  Deleting queued message.
                    }


                    //
                    // If execution made it to this point, then we are either fully processed, or 
                    // there is sufficent reason to discard the message.
                    //
                    try
                    {
                        queueAgentQueueActions.DeleteMessage(msg);
                    }
                    catch (StorageClientException ex)
                    {
                        // As of July 2010, this is the best way to detect this class of exception
                        // Description: ttp://blog.smarx.com/posts/deleting-windows-azure-queue-messages-handling-exceptions
                        if (ex.ExtendedErrorInformation.ErrorCode == "MessageNotFound")
                        {
                            // pop receipt must be invalid
                            // ignore or log (so we can tune the visibility timeout)
                        }
                        else
                        {
                            // not the error we were expecting
                            throw;
                        }
                    }
                }
                else
                {
                   // allow control to fall to the bottom, where the sleep timer is...
                }
            }
            catch (Exception e)
            {
                // Justification: Thread must not fail.
                //Todo: Log this exception

                // allow control to fall to the bottom, where the sleep timer is...
                // Rationale: not doing so may cause queue thrashing on a specific corrupt entry
            }

            // todo: Thread.Sleep() is bad
            //       Replace with something better...
            Thread.Sleep(9000);
        }

Q: Is "DataContext" appropriate name for this class?

In .NET we have a lot of DataContext classes, so in the sense that you want names to appropriately communicate what the class does, I think XyzQueueDataContext properly communicates what the class does - although you can't query from it.

If you want to stay more aligned to accepted pattern languages, Patterns of Enterprise Application Architecture calls any class that encapsulates access to an external system for a Gateway, while more specifically you may want to use the term Channel in the language of Enterprise Integration Patterns - that's what I would do.

Q: Is it a poor practice to name the Queue Action Name in the manner I have done?

Well, it certainly tightly couples the queue name to the class. This means that if you later decide that you want to decouple those, you can't.

As a general comment I think this class might benefit from trying to do less. Using the queue is not the same thing as managing it, so instead of having all of that queue management code there, I'd suggest injecting a CloudQueue into the instance. Here's how I implement my AzureChannel constructor:

private readonly CloudQueue queue;

public AzureChannel(CloudQueue queue)
{
    if (queue == null)
    {
        throw new ArgumentNullException("queue");
    }

    this.queue = queue;
}

This better fits the Single Responsibility Principle and you can now implement queue management in its own (reusable) class.

I'm interested in open-source Java/C/C++ framework for pipes-and-filters (like described in that book) paradigm. Can you recommend some?

EDIT: I'm looking for as much "pure" (or lightweight) frameworks as possible: frameworks which encourage programming in "pipes-and-filters" technique, without reliance on any standards, such as XML or JMS, but on language only.

I just discovered Spring integration framework. Looks like the sample of what I was asking for.

I was reading a blog, and one of the points was 'if you're using queues, you messed up', in the context of JMS.

I was thinking, do we even need JMS? A simple alternative would be, if you need to do something asynchronously, why not just put a job request in a table somewhere, and have some process(es) polling the db every X time units looking for new jobs?

This approach is simpler than JMS, its easy to understand, and basically removes a dependency from an application.

What am I losing if I use the alternative I described? Perhaps one loses the possibility of using JMX to be able to administer things, but if your job 'queue' is fed off a table, you can write some simple code to 'manage' the processing.

For very simple requirements of just asynchronous processing, you can use any suitable class from java.util.concurrent package.

If you need transactions or the guarantee that a job must be successfully completed once submitted, even in case of system failures (software or even hardware crashes), or you want to offload job processing to another process you need some other solution.

JMS approach can provide a very sophisticated solution with relatively less effort.

Messaging (or JMS) is a very standard integration solution which can solve the problem of keeping submission of jobs asynchronous and keeping submission of tasks decoupled from the actual processing. Messaging based solution can be easily scaled by increasing the number of 'job processor' threads listening on the job queue. The traffic will be automatically load balanced. Messaging systems can also provide transaction support, i.e. automatically put the message back on the queue if job processing fails so that it can be retried.

Many enterprise integration patterns are based on Messaging systems (Message oriented middlewares). This book on Enterprise Integration Patterns by Gregor Hohpe has most popular patterns of how to use messaging in your applications.

The database approach requires another process to

1) Poll the table for 'new jobs', update the status of the row when job processing starts by the processing application and eventually either update the row status of the job to 'done' (or delete the job altogether from the table). 2) If something goes wrong during the job processing, the job status should be changed back to 'new' on the table, so that the 'polling' mechanism can pick up the job again. Also, need will arise to write some 'recovery thread' on system start-up to find out jobs which might be in an inconsistent state and put them back in 'new' state to start the processing again.

Bottom line is, it takes a lot of effort to build an integration solution which is based on a database. It also tightly couples both 'job submitter' and 'job processor' applications with the database schema which breaks the encapsulation of your database. If your 'job processor' has multiple threads (which you will probably need if you want to scale) then you need to make sure that only one thread picks up the job and updates 'that' row.

A JMS solution solves this problem very easily without hand-coding all this logic.

Surely, if you are using queues you are NOT messed up. But you should have some valid use-case to introduce a messaging middleware.

One of the jobs I'm looking at requires good experience with EAI.

Was wondering if anyone can recommend good EAI books / resources?

I've had this book for several years, and I still regard it as a valuable reference.

Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions

The author also has a list of books on his website.

What would be best practice if you have an event-driven architecture and a service subscribing to events has to wait for multiple event (of the same kind) before proceeding with creating the next event in the chain?

An example would be a book order handling service that has to wait for each book in the order to have been handled by the warehouse before creating the event that the order has been picked so that the shipping service (or something similar) picks up the order and starts preparing for shipping.

You can use the Aggregator pattern, also called Parallel Convoy.

Essentially you need to have some way of identifying messages which need to be aggregated, and when the aggregated set as a whole has been recieved, so that processing can start.

Without going out and buying the book*, the Apache Camel integration platform website has some nice resource on implementing the aggregator pattern. While this is obviously specific to Camel, you can see what kind of things are involved.

* disclaimer, I am not affiliated in any way with Adison Wesley, or any of the authors of the book...

Ever since the publication of Enterprise Integration Patterns people have been using the notation introduced in that book for documenting asynchronous heterogenous messaging systems.

But our shop is more or less standardized around a proprietary documentation tool that does exclusively UML. Is one of the standard UML diagrams appropriate for documenting asynchronous messaging systems, including transformers, routers et al.? If yes, which one? If not, what would be a killer argument against the tweaking of UML for this purpose?

UML provides a mechanism for extension through profiles

A profile allows you to specify stereotypes, tagged values, and constraints.

Every stereotype can have an optional stereotype icon.

Perhaps there is an EIP profile for UML you could download?

If not, you can build your own profile if your UML tool supports this, and use the icons available as Visio shapes from Gregor Hohpe's website

Hope this helps.

I'm working on a project where we're working with both Spring Integration & AMQP for the first time.

I'm noticing there appears to be a lot of XML configuration involved in wiring everything together. Our config files are already very messy, and it's getting tricky to keep across how everything is hanging together. I'm worried we're about to make some preventable mistakes.

What resources are available that provide guidance on how to structure large-scale, real-world spring integration & AMQP projects.

eg:

  • What naming conventions should we follow for channels / queues / exchanges
  • When to employ XML config vs Annotation based config
  • Is it ok to mix & match annotations & xml, or is this a recipe for disaster?
  • How to layout the bean definitions for ease of use

Note - I'm not looking for guidance on what the various components do, more - how to structure your project to foster maintainability.

Ideally, I'm looking for:

  • Blog posts
  • Books
  • StackOverflow answers (answers inline would be great, too)

It is not SI and AMQP specific but, I think reading
Enterprise Integration Patterns : Designing, Building, and Deploying Messaging Solutions by would be very helpful. Also this link has also good resources.

I've just started messing around with AWS DynamoDB in my iOS app and I have a few questions.

Currently, I have my app communicating directly to my DynamoDB database. I've been reading around lately and people are saying this isn't the proper way to go about getting data from my database.

By this I mean is I just have a function in my code querying my Dynamo database and returning the result.

How I do it works but is there a better way I should be going about this?

From what you say, I can guess that you are talking about a way you can distribute data to many clients (ios apps).

There are few integration patterns (a very good book on this: Enterprise Integration Patterns), one of which is called shared database. It is essentially about using a common database for multiple clients to share the data. Main drawback for that pattern (in your case) is that you are doing assumption about how the database schema looks like. It can potentially bring you some headache supporting the schema in the future, if your business logic changes.

The more advanced approach would be sending events on every change in your data instead of directly writing changes to the database from client apps. This way you can add additional processing to the events before the data they carry is written to the database. For example, you may want to change the event format in the new version of your app, but still want to support legacy users, so you add translation procedure which transforms both types of events to the format which fits the database schema. It's basically a question of whether to work with diffs vs snapshots.

You should be aware of added complexity of working with events, and it can be an overkill if your app is simple and changes in schema are unlikely.

Also consider that you can do data preprocessing using DynamoDB Streams, which gives you some advantages of using events still keeping it simple to implement.

The classic "Design Patterns: Elements of Reusable Object-Oriented Software" actually introduced most of us to the idea of design patterns.

However these days I find a book such as "Patterns of Enterprise Application Architecture" (POEA) by Martin Fowler, much more useful in my day to day work.

In discussions with fellow developers, many make the (fair) point that frameworks like .NET are slowly starting to provide many of the patterns in the GOF book, and so why re-invent the wheel?

It seems many developers think that the GOF book is the only reference worth having on design patterns. So their logic goes that because frameworks (.NET etc) are providing many GOF patterns for us, patterns have seen their day and are no longer as important.

Surprisingly (to me at least) when I mention the patterns descibed in POEA, I am often met with blank stares.

POEA proves that patterns are more than just Interators, Singletons, Commands etc. I would also say that the patterns in GOF are really just a distinct "category" of patterns, applicable to a different (non-overlapping) level of design, than those in POEA.

So, what other "categories" of patterns have applied in your development? What level of design do these patterns apply to? At what point in the development process are they normally used? Is there a book or reference for these patterns?

CategoryPatterns on Ward's wiki contains a categorized list of patterns.

The first three are the GoF patterns

  • Creational
  • Structural
  • Behavioural

Then there are problem specific problems

  • Security
  • Concurrency
  • RealTime

Fowler's pattern are Enterprise Application Patterns. There are also Enterprise Integration Patterns. UI patterns also exist.. and so on...

A client has a number of disparate systems that they are planning to link together and have asked for a set of system interface specifications that would document the data and protocols used to interface the different parts.

The interfaces are between processes not between users.

Any recommendations for a template that we could use to document the system interfaces?

This book might have the sort of guidance you're looking for: Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions

How to start learning Java Message Service (JMS)?

I would say to read Enterprise Integration Patterns, it's a bit high level but you must understand integration to use JMS smartly. Next you can Java Message Service, a good book for technical questions.

I'm currently trying to get comfortable with multi tier applications (Server / Client Architecture). For that I created Service Interfaces and the associated Service Implementations.

Both modules - the Client and the Server - knowing the Interfaces (included via build path).

The purpose of the application is that the client can receive data from the server and also send data to the server where it become stored into a database.

It seems so that I need a communication into both directions. Is the "standard" Java RMI the correct approach for that or do I need to use a JMS implementation like ActiveMQ?

I can't see the main difference between the two approaches? Is the only one that RMI is synchronous and JMS asynchronous or is there more than that?

Futhermore, can you recommend an ActiveMQ tutorial or even book?

We're talking of two different technologies here, each with different usage scenarios. RMI is a Java application programming interface (API) which enables remote invocation of methods between Java programs, and its intended use is for building synchronous, distributed applications. RMI is the Java equivalent of remote procedure calls (RPC).

On the other hand, ActiveMQ is a product, a message oriented middleware (MOM), useful for receiving and processing asynchronous messages (queues or topics), and Java applications would typically communicate with such a system by using the JMS API.

You would use RMI when you need to communicate with another Java application in a distributed, synchronous fashion, whereas you would use JMS for sending messages to be processed asynchronously, and in principle the processing could involve systems written using any other technology, not just Java (unlike RMI).

Asynchronous messaging is typically used for solving the architectural problem of integrating disparate systems, a very good (technology-agnostic) book on the subject is Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions.

Finally, ActiveMQ in Action is a nice book on ActiveMQ.

Recently lot of interviewers are asking

"What are the integration patterns are you familiar with?".

I know design patterns,what is integration pattern then?

We are going to develop a strong plugin-based c++ software. There is only few source-code examples for it. Due to the big size of the project, we want to do a simple software modeling, before we start to implement it. First of all class and sequence diagramms. We are newbies to UML, due to this reason, it would be nice, to have an UML-Example for a plugin-based project. We haven't found any examples for it on the web. Design patterns we want to use are i.e. plugins, plugins manager, factory method, singleton and some others. And now my question:

Do you know a (simple) example for a UML-Project for a similar software design?

Thank you.

Cheers Alex.

  • As for "factory method, singleton and some others", expressed in class diagrams, read the Gang of Four book.
  • As for plugins, you are looking in bad places. They are to be expressed in means of component and package diagrams, not class ones.
  • As for sequence diagrams, (also state machines, timing, activity...) , they are not intended for showing the problems of structure. And plugin is a structural term. Behavioral diagrams are useful, but they don't have any specifics when used for plugins.
  • For interactions of plugins, also look for UML communication and interactions overview diagrams. And maybe look at diagrams in the book "Enterprise Integration Patterns...". Their diagrams are not UML standard ones, but IMHO, are very convenient and useful for modelling of interactions on the level of components and packages (or plugins).

We have developed a WCF service which acts as an API to our system.

some of the methods expose functionality which eventually writes and updates data to DB.

Lets say that the client sent one of those requests, the service performed the needed operations while the client disconnects for some reason and did not get the response.

An example of problematic data being updated - "Add X to Member M".

just to make things clearer - client sends a request and receives a response which notifies about success/failure of the request.

our problem is - what if the client sends a request which is performed but the response is not received - how can we notify the client about the reponse?

One way would be to use an asynchronous messaging pattern, so instead of submitting a request and waiting on a response, you would submit a request and continue on, and have either a callback contract or an external-facing service contract that your service can connect back to (a callback contract is the best option when you're dealing with a typical client-server scenario). The downside to callbacks is that they're only supported using two-way bindings (WSDualHttpBinding, NetTcpBinding, etc.), which will limit interoperability with non-WCF technologies.

Another option would be to use an enterprise service bus. That's basically software that sits in the middle and coordinates everything between your client and service. That way if your client times out, the enterprise service bus can trigger a "compensation" activity that ensures both your client and service end up in a consistent state.

Update: Enterprise Service Bus is a very broad topic. Wikipedia is a good place to start, and Amazon offers several books covering various platforms in detail. As far as actual implementations, Microsoft offers BizTalk Server, IBM has WebSphere ESB, Oracle bought BEA Systems which offered AquaLogic (now Oracle Service Bus), and there are multiple open source systems as well including Apache ServiceMix, Apache Synapse, and Mule.

I have just been handed a huge project for an application that must run on a terminal that securely collects cash drops. It is a rewrite of an existing project, so requirements are pretty well defined, this is the product. Although my employer will supply me most application specific new requirements, I would like some general advice on architectural aspects and areas that I should pay attention to in order to ensure maximum quality for my rewrite. The application will run on XP Embedded.

Hopefully folk who have written such applications can maybe advise me on areas to focus on.

Unfortunately I have no experience with what you are going to implement, and I didn't take this book, but when I read your question I just remembered its title and idea that it can be useful pop up: "Real-Time Design Patterns: Robust Scalable Architecture for Real-Time Systems" (check also book advice from amazon customers under that book)
Here is google book link, so you can check its content

Btw, you will have client-server systems, maybe another integration needed, so there is another interesting book in that area which I read: Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions and alternative, similar catalog of SOA patterns http://www.soapatterns.org/ - just in case you will need integration and its security.

I'm starting the design of a new app that is primarily network oriented, and I'm looking for some advice from people who have come up with a good architectural design, or OOP class model.

Please describe the basic architecture and/or class structure. Did you abstract out the communication bits? What class entities did you come up with?

The app will have both listener and client classes. It's similar to a feed aggregator, but uses persistent connections rather HTTP. In other words, I connect to a socket and have a persistent connection in which data flows both ways. Then I also have clients that connect to me persistently and i send some (or all) of the data to them.

Also, I can't use WCF or anything that's in .NET 3.0 or 3.5 (though I can use C# 3 because i'm developing on VS2008). I have to be compatible with Windows 2000.

"Network application" is an unclear category covering quite a broad spectrum of possible requirements and designs. By definition, you will have to deal with concurrency and latencies. Robustness is always a challenge: it is all too easy to design a system whose availability is the lowest common denominator of all its hardware and software components. You will have to upgrade the O/S and components of your application: can you live with taking down all of the solution, or do you need some hot/cold standby or even failover capability? Networks fail in exotic ways (at least to us poor pure software guys): you are dependent on so many things you never consider (routers, switches, DNS servers, the temper of that bearded network engineer down the hallway!), and should design the interactions with a pessimistic frame of mind.

One warning: don't be too object-oriented when designing distributed applications. Objects have their place in managing software complexity, but the moment you start pushing them on the wire, a huge number of performance and version dependency problems raise their ugly heads. Service oriented architecture is a buzzword, but is useful as a design principle, to a point. The mainframe guys have known this for ages...

Two general book recommendations:

I was looking at the IT impacts for one of the changes that was required to my IT application stack. There are a number of systems and most of the systems using webservice for communication. After I had completed my assessment , one of my collegues asked me if there are any 'canonical changes' required. I was puzzled with the term and asked him what 'canonical change' means and he rephrased the question asking if there are any xml schema changes.

Does the term canonical change same as changes in xml schema? If it so , is there any specific reason for calling it canonical change?

Regards,

Both of the books below refer to an idea of a "canonical" XML, as your in-house proprietary format... it's assumed that you're taking multiple, third-party data sources (i.e., Web Services) and translating the XML into your "canonical" XML format.

For Example, let's say that you have a "canonical" XML format that looks something like the following:

<MyAPI>
  <OrderNo>123</OrderNo>
</MyAPI>

Web Service A may have similar data, but in a unique format:

<WebServiceA>
  <OrderNumber>123</OrderNumber>
</WebServiceA>

Web Service B may also have similar data, but in another distinct format:

<WebServiceB>
  <OrderID>123</OrderID>
</WebServiceB>

Yet, both will be translated into your "canonical" XML format:

<MyAPI>
  <OrderNo>123</OrderNo>
</MyAPI>

Optional assignment for one of my classes. 30-45 minute presentation/case study on either of these two topics:

  1. Examples of currently existing design patterns in real life projects: what problem they solve, why are they better than other techniques, etc
  2. New design patterns, what problems they solve that other design patterns can't, etc

Note that "new" and "existing" are with respect to the GoF book and the design patterns listed within.

For the first one, source code is not required, but it's probably a plus, so an open source project would be the best.

For the second, I'd basically need to be able to give a description like the ones in the GoF book for each pattern, with proper motivations, examples, and the like.

Anyone got some good ideas/pointers?

What are the best books/tutorials that you've read regarding back-end server programming ?

I'm not looking for a book written for a specific platform (windows or linux) or for a specific language (c, c++, java, etc) I'm looking for a book or a list of books that best describe the general patterns, algorithms and problems that might arise when developing in a server environment (multithreading issues, networking issues, etc).

Thanks :)

Where can I find a good high-level overview of Enterprise technology concepts and how they intermingle?

Such as, what is a:

  • Service Bus
  • Application Server
  • Messages
  • Middleware

The book Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Hohpe and Woolf, along with its supporting web site http://www.eaipatterns.com/ are terrific resources.

In the same series, Patterns of Enterprise Application Architecture by Martin Fowler is also very valuable. Martin Fowler's website contains a great amount of material, here is a good starting place: http://www.martinfowler.com/articles/enterprisePatterns.html.

SOA Design Patterns by Thomas Erl, and the companion web site, http://www.soapatterns.org/, is even more encyclopedic. I particularly like this treatment of the enterprise service bus pattern and its constituent patterns.

As with all books on patterns, once you've read through the introductory material, they can be used a reference books, allowing you to selectively read the topics that are of interest, and perhaps going back later for a more thorough, cover-to-cover reading.