+972 52-548-6969
Full speed ahead, and damn the benchmarks
A while ago I posted about upfront optimizations. Sasha Goldshtein commented (after an interesting email conversation that we had):
What truly boggles me is how the argument for correctness in software is not applied to performance in software. I don't understand how someone can write unit tests for their yet-unwritten code (as TDD teaches us) and disregard its performance implications, at the same time. No one in their right mind could possibly say to you, "Let's define and test for correctness later, first I'd like to write some code without thinking about functional requirements." But on the other hand, how can you say to someone, "Let's define and test for performance later, first I'd like to write some code without thinking about non-functional requirements?"
It fell into my blog backlog (which is a tame black hole) and just now came up.
The reason for this attitude, which I subscribe to, is very simple. A lot of developers will do a lot of really bad things in the name of performance. But I did see, more than once, whole teams dedicate inordinate amount of time to performance, contorting the design so it resembles a snake after a fit of the hiccups and leave an unholy mess in their wake.
Add to that that most of the time performance bottlenecks will not be where you think they would, you get into a really big problem with most developers (myself included, by the way). As a result, the push to "don't think about performance" has very real reasoning behind it. As I mentioned, you should think about performance when you are designing the system, but only in order to ensure that you aren't trying to do stupid things. (Like running a server off a single user embedded database, my latest blunder). Most of the time, and assuming this is not a brand new field, you already know that calling the DB in a tight loop is going to be a problem, so you don't even get into that situation.
Another issue is that I can refactor for performance, but (by definition), I can't refactor to be correct.
Comments
If there are measurable non-functional requirements, you should have tests for it. For example "Creating an invoice should take less than 2 seconds".
Pre-optimization would be to try to make the operation take much less time, say 0.5 seconds. In that way performance tests keep optimization just at the correct level.
Oren, thanks for resuming this interesting discussion. There is another point that has to be made. When designing code to be testable, you'd be looking for dependency injection, inversion of control, interface- and component-based approaches, and many other techniques of that ilk. Essentially, the primary obstacle to testability is coupling. On the other hand, the primary obstacle to getting good performance is decoupling!
For example, assume I was writing the .NET framework and my current assignment was to implement floating-point numbers addition. If I were to design a truly decoupled approach with no internal dependencies, I would have an IAddFloatingNumbers interface which would then be implemented by the framework, but could be mocked oh so easily. But I can't afford having an interface call for floating point addition, right? Quite vice versa - this addition is an intrinsic IL instruction which is recognized by the JIT and compiled to the most efficient code possible on the platform, using SSE, SSE2 and whatnot. Which is very coupled. Which is very NOT mockable. And guess what? It's the only way to implement addition unless you're writing a toy language.
And the exact same argument applies to a framework you or I are building. Right, if you're designing a PlaceOrder component, it's perfectly fine to hide it behind a mockable IPlaceOrder interface and forget about performance. But what if you're designing a messaging infrastructure for intra-application communication? It is no longer fine to hide it behind a mockable IQueue interface because that interface call is more expensive than the enqueue operation! And if you did couple everything to Queue<T>, then how exactly are you going to refactor your code for performance when you discover that you need a lock-free queue? I think we disagree on the meaning of "refactoring" if that's refactoring.
@Sasha - re: "And if you did couple everything to Queue<T>, then how exactly are you going to refactor your code for performance when you discover that you need a lock-free queue?" - I think that without realizing it you just accidentally made the case for decoupling. Using your own IPlaceOrder example, the decoupling allows one to optimize for performance once any bottlenecks have been concretely measured in a nearly-complete system.
There are hotspots in most applications that could benefit from optimization efforts. Those optimizations sometimes will lead you to remove abstractions in order to enhance performance.
Those hotspots tends to be rare, in my experience. Most often, perf can be had by changing the impl of the abstraction.
Replacing LockingQueue with lock free queue, replacing the impl of IGetData with CachingGetData, etc.
I am not arguing that there aren't places to do so, I am arguing that it is infrequent to need this, and when you do, in a good system, you'll have the ability to go into one place and reduce the level of abstraction that you have.
"Another issue is that I can refactor for performance, but (by definition), I can't refactor to be correct."
That depends on what you define under 'refactor' If 'refactor' means: change working code without changing its functionality how it works, then, yes you can't refactor to correctness. However, if 'refactor' means: changing the code so it's something different, you can refactor to become correct.
I think a lot of people will see 'refactoring' as the action 'change code'.
Frans,
My definition of refactor is changing without modifying external functionality.
Changing the code is not that. That is changing the code.
Refactor became somewhat of a buzzword
@Ayende - buzzword is right. You'd probably be surprised just how many people misunderstand the term, at least within the environments in which I usually find myself working. "But I can't change how it does anything! I can only make the code look a bit different!" and things like that.
They seem to latch onto the size of each change, which in reality is only intended to keep things safe and make individual refactorings readily applicable across a range of situations. They latch onto how little a single refactoring does, and forget that the entire point is to progressively make small refactoring after small refactoring. They latch onto the fact that existing functionality mustn't change, forgetting that it only mustn't change when seen from specific viewpoints (ie. the UI or an externally-exposed integation point) that have to be established on a situation by situation basis.
It usually takes a while to bring these people around, but even then some of them can't quite get their head around it. Luckily, most do, even if you have to pair with them and walk through a whole series of incremental refactorings resulting in an overall obvious improvement.
I think a lot of this stuff does boil down to problems with premature optmization, but there are things that don't count as premature optmization, the primary one is algorithm choice - which will be dictated by business/scale requirements.
Say you pick an algorithm that happens to have O(n^2) performance - unless you know that that is the performance driver, you might spend a lot of time tweaking the edges to improve it without realizing that it's a fundamentally approach.
My general approach is to work out/select algorithms that have the appropriate big O complexity, and then profile the rest.
Ayende,
This is one of the most common issues I run into when leading teams in the areas we both love to play in...
You hit all the salient points, however I would simply add that (adding a layer of abstraction - grin) IN GENERAL if you are predicting the future, you are likely in a place that is creating waste (typically almost complete waste).
The case of 'regression unit tests' that try to 'predict' performance issues to me is simply an instance of the more general 'predicting the future' problem that millions of people and BILLIONS of dollars are still wasted on. This is how I present is usually. Most people instinctively know that we cannot predict the future, then turn around and ask for a non-statistical project plan!
Systems are complex, and even 'predicting the future' and saying 'we understand the problem domain' before we are finished is usually silly (even when the domain seems fairly trivial, I am always astounded at how much the experts learn).
A core argument I make in the damn book I hope to finish someday is just how much software engineering is a 'wicked problem' in the formal sense as Rittel(?) around 1973 defined it.
A critical path through a plan somebody dreamed up (again predicting) is not a singular thing any more then one 'theoretical future' is.
Each managed task element should (at least if people really want to try to wrestle this monster) have (in my experience) a statistical confidence (to the week is about all u can do), a recording of the net impact for a 'bad occurance' where indeed say the 80% confidence has fallen in reality into the 20% fail category (2 tasks at 80% might have drastically different net 'problems' created by their failure even though they have the same duration -- work around the holidays is a common but perhaps overly simplistic example).
I almost hate to say the word 'Agile' as it is so misunderstood, but Agile is meant to provide the real-time feedback for a 'living project plan' that evolves as we evolve what the hell we are doing. Anything else is a plan to fail (as I know u know).
Anyway, keep up the awesome work.. I'd love to hear your take on these ideas as they have come to frame how I conceptualize and execute solutions. In fact all the design/technology decisions almost become self-evident when these truths are embraced in very compelling ways...
Thanks,
Damon Carr
Comment preview