+972 52-548-6969
Beware the common infrastructure
One of the common problems that I run into when consulting with clients, or just whenever I am talking to developers in general is the notion of common infrastructure. “We are going to spend some time building a common infrastructure which we can then use on all of our applications.”
I made that mistake myself with Rhino Commons, and again very recently with RaccoonBlog (look at the code, you see the Loci stuff, that is stuff that is used from another project).
Why is that a problem? Well, for the simplest reason of all. Different projects have different needs. A common infrastructure that tries to accommodate them all is going to be much more complex. Not only that, it is going to be much more brittle. If I am modifying it in the context of project A, can I really say that I didn’t break something for project B?
Let us take a simple example, executing tasks. In RaccoonBlog, we need tasks merely to handle comments and email (long running background tasks). In another application, we need to do retries, and we need to get notifications if after N retries, the task have failed. In a third project, we need a way to specify dependencies between tasks.
Sure, you can build something that satisfy all three projects, but it would be drastically more complex than having to modify the original task executer for each project needs. And yes, I do mean copying the code and modifying it.
And no, it is not a horrible sin against the Little Endianness. Even duplicated N times, the code is going to be simpler to read, perform faster, easier to maintain and modify over time.
Nitpicker note: I am not talking about 3rd party libraries here. If you can find something that fits your needs that already exists, wonderful. I am talking about infrastructure that you build, inside your organization.
Comments
Making common infrastructure would make sense, if your projects are huge and the common infrastructure is percieved as 3rd party library with its own development team, planning, requirements, testing and so on.
And only huge compannies can afford and and benefit that.
Main reason behind "We are going to spend some time building a common infrastructure which we can then use on all of our applications.” is the $$$ that someone has to throw on infrastructure and 90% of times it is a lack of those $$$ that puts 100 of different apps in same environment.
although virtualization eases this in a magnitude the rest 10% of cases fall back to "not enough brain to achieve this" cause it is complex even when separated
What if you have 2+ applications that vary by features, but not by concepts, or otherwise have very similar architectural needs?
A common infrastructure makes sense only when a company is doing a lot of similar projects, for different clients - ex.: an outsourcing company, doing a lot of similar web apps or sales applications for different clients - in such cases it's worth reusing what can be reused, but that is usually a hard choice to make.
I learned this the hard way several projects ago. You end up moving between two solutions, making sure the common infrastructure remains the same and makes sense for both applications. Then you realize you are wasting time.
The only time it makes sense to build common infrastructure is when you CAN create a 3rd party library out of what you are separating. These include HtmlHelpers for MVC, Serialization helpers, and other abstraction. At that point, it is worth putting the code in a Nuget package and sharing it with the world (if it is any good that is).
I could write a book about this problem.
@Jason Young, you don't know that up front, only after the projects have matured can you start to see definitive commonalities.
What about having a common infrastructure only for the most basic of things, e.g. if you had standard logging/auditing classes or if you were using a basic database abstraction layer with no detail about the project itself?
Awesome, I totally agree! Still, after almost 20 years in the business, I can't seem to win this argument. Every team fools ahead towards the common infrastructure. Even when it predictably goes haywire, everyone keeps defending it.
Please tolerate a brief foray into politics...
I've long thought that strong analogies can be drawn between legal code and computer code. For example, I believe your advice in this post suggests that it is wise to minimize the amount of laws written at the federal level, instead favoring laws written at the state level. If you modify a federal law to accommodate needs of state A, it's hard to know whether that change breaks something for state B.
There's probably lots of other computer programming concepts that could be beneficial if applied to the legislative world; Single Responsibility Principle comes to mind.
I can't count the number of times I've read that component reuse does not happen by accident. I'm not sure why someone would think that scaling that up would change that.
At the start of every new project, I find myself cherry picking code from my last project and improving it in some way or another. Then I always tell myself, "one day I'll refactor out a commons library". Maybe it's not even worth it...
I agree, copy & past does wonders for maintainability in this regard.
... which is why I've started to NetFX ( http://netfx.codeplex.com/ ) to distribute 'common' code in source form instead of the dlls.
There is a balance to just about everything in this world, and every project has its own unique equilibrium point between making things more complex with common code and duplication of simplified code. On one extreme you can start every piece of a project from scratch and lose the ability to propagate changes and maintenance becomes difficult; on the other, you could create a super complex common library that is difficult to use and extend.
It's not a big or small company thing. It's more of a big or small project thing (big projects obviously cost more). It all depends on the project and installing the hinge points for things that are likely to change using interfaces, compositions, and sparingly inheritance. I believe Design Patterns and The SOLID principles as well as a good plan, work together to guide the project to that equilibrium point.
Much easier said than done though...
I follow YAGNI when it comes to "common" peices of a software. When I build something I assume it's not going to be shared. When I'm working on another application and see I need the same thing I generally will just copy and paste it. At this point I can start to evaluate how likely it will be used again.
If I feel it's highly likely another project will use it, I will turn it into a nuget package for our private feed. The nuget packages we have my organization are gigantic time savers and never once has a package became unwieldy or any additional cost to maintain. Having explicit version numbers of the package and following SemVer.org I never have any concerns about changing my packages. Any consuming applications are already working, they don't NEED to update. As long as I follow breaking change = major version bump there is never any chaos. The cost of addressing the breaking changes is only ever paid IF and WHEN those apps truly need the new features to spend the effort to migrate forward.
Simply put it just comes down to doing your job as an architect well (a nontrivial task).
I've been in Ayende's mindset before, and I've been in the opposite mindset before, and I think I've recently achieved a balance.
When building projects at work, I kept identifying areas where I kept on using the same chunks of code across projects over and over again: a library for handling international mailing addresses, a library for handling payments with authorize/capture/refund support, etc.
I've refactored these into internal shared libraries where appropriate. If it takes a little bit of effort to make it reusable, then I go for it; if it doesn't, then I don't.
The key that I've found for me is that the shared libraries are EMERGENT, from identifying commonly-used code in existing projects, instead of the other way around.
If I try to prescribe shared libraries from the beginning, then I end up guessing wrong, not adequately foreseeing the unique subtleties that each project may require. But when I pull the abstraction from three similar implementations, then the abstraction is much more solid and usefully reusable. I used to think that this was simply immaturity and that I lacked foresight; as the years have gone on, I see that pretty much everyone else runs into this same thing, so I don't feel so bad now.
Been doing same thing recently - just picking required pieces from older codebase. However, I don't think this is a good approach rather than just the best with .NET which has underlying framework implemented either on too low or too high level. That causes you to produce snippets here and there which move from project to project. Lots of stuff is rather rudimentary but required elementary glue for common web apps .NET lacks.
Today I just told my product manager that I write code SO THAT I can just copy and paste.
@Daniel Schilling: the analogy with federal and state laws might not make sense for many readers here, because in most countries there is only one set of laws, the provinces does not have separate laws (I assume is just a particular stuff specific to US)..
Et tu brute?
First, I do not understand the reference to Little Endian. Little-endian is a hardware architecture which stores the least significant bit first. It is only relevant to when you are writing your own tcp/ip which has to account for such things.
Second, I think that in Oren’s nitpicker note, he has created an artificial boundary between internal and external components. Breaking your infrastructure into components provides both a wider code coverage leading to better tested code, as well as requires a better understanding of separation of concerns. Perhaps a better distinction would be between infrastructure and design patterns.
Wait - you write more frigging "common infrastructure" that you use across projects then almost any human I know. How did things like NServiceBus come about other than as "common infrastructure".
I absolutely 'Love your work', but I personally find it more than a little ironic and a bit hypocritical when I hear OSS folk trot out this oh-so-tired, high-mileage meme.
P.S. I waiting with baited breathe for a 'no infrastructure' CQRS implementation....
Well it is pretty common in the .net world to overengineer stuff and trying to build a swiss army knife in every project. I have seen this on several occasions at different companies the past 5 years as a .net developer.
This is where clear and well structured namespacing helps. I agree that hindsight is great for determining commonalities, and if you are rigerous with your namespacing, identifying code that does not depend on product specifics becomes a breeze.
Comment preview