My Descendents Thank You

I have 3,932,160 grandchildren.

At least I will in 600 years, if my children and theirs each produce two reproducing children per generation.  And that’s exactly when today’s performance optimization will pay for its own development time in CPU cycles.

I have written previously about intentionally leaving query results unordered to save some academic CPU cycles while burning money in end-user time – a poor “passive” optimization.  Now we’ll discuss actively making inane optimizations that will never pay for themselves – a poor “aggressive” optimization.

A developer modifies a query with the academic goal of saving several milliseconds per execution (perhaps as few as 6 ms).  If this was a query that ran thousands of times per hour during user time, we might have been able to quantify some savings to the business.  Unfortunately, the optimized queries were in some procedures that run once per day during a nightly batch cycle that had no time pressure.

I ran the numbers on the CPU savings and it will take more than 600 years for the saved CPU cycles to accumulate the same amount of time spent to “optimize” the query.  Because the CPU time for a batch process does not correspond to the real money being spent in user-time, this optimization will never be paid for… even in the 600 year break-even period for the CPU time.

These two types of performance optimization illustrate the danger of optimizing for purity of execution without giving any consideration to the practical implications of where our technical decisions force the business to spend its money.

If we keep wasting it like this, they certainly won’t be spending it on us.

Don’t waste my (users’) time

Tradeoffs are everywhere.  This is particularly true in the realm of software development.  As the saying goes, “speed costs.  How fast do you want to go?”  We should always keep these tradeoffs in mind as we balance our technology decisions with the needs of the business.

Here is a quick tale of a not-so-great performance optimization that really emphasizes the importance of making optimization decisions that actually have business value…

The case of a missed sort optimization
A business user and her colleagues currently sort and search a grid manually because "without some kind of 'sort' you have no idea where in the list your application will be found".  These results apparently get scrambled each time the grid refreshes.  The company is spending wage dollars (read “real money”) to do the work that the CPU could be doing for “free”.

In this case, the ten seconds worth of real money spent by the developer to type “ORDER BY ” would have been a huge win.  One ten-second real money line of code would have saved thousands of several-second real money live employee sorts. I don’t know about you, but I get excited about single order-of-magnitude performance increases.  This was a three-orders-of-magnitude oversight!

Because the developer didn't make this 10 second investment up front, the company now has two options (there may be others... conversations are good at flushing out options!):

1.    do nothing and continue to spend  real money to do the sorting that the CPU could be doing for free

2.    spend thousands of dollars of real money to add an ORDER BY (or an appropriate index to force a sort).

I can hear you scoffing now, “Sure, thousands of dollars!” Picture it with this nearly exaggerated process –

We start with our line-of-business employees spending their time (remember, that’s real money) adding the suggestion to the feature request system and voting it up (picture Microsoft Connect for your business applications).  We just spent more real money getting the bug typed in and voted up than we would have spent with the initial sort, but I digress.

Next, we have the bug validation team spending real money making the appropriate contacts and follow ups with a decision maker who determines (spending more real money) if this is a legitimate feature request.

Once we have determined that this is a real bug, worthy of company resources (more real money), we proceed to the team (think real money multiplied by Dilbert) that determines which features and applications get worked by which development teams and when.

Now that we’ve gotten all the way through the verification, validation, prioritization, and schedulization phase, it’s time for a software development team (there’s that real money multiplier again) to actually begin working the bug – remember that ORDER BY thing we were talking about?  Our software development team is going to go through its own verification, validation, prioritization and schedulization process.

They have to reproduce the bug, find the source code module where they can actually make the fix, fix it (our initial 10 second line of code), then run it though the test deployment process, user acceptance testing, the pre-prod deployment process, the pre-prod testing process, the change request for production process (think about another team here), the production deployment process and the post-production testing process.

Don't get me wrong, this isn't some process oriented bureaucracy, this is a kanbanagilescrum .NET development shop with all the modern efficiencies as their disposal.

You probably thought I forgot where I was going with this.  It was close there for a second, but it's still here.  The problem with this isn't that the developer made a simple oversight on a search screen and nobody noticed before production.  That’s understandable, and we all do similar things routinely.  The problem is that there really are developers who intentionally leave all of the sorting to the end-user because they want to save the CPU sort cost for the 400 rows.

These decisions not only erode our users’ confidence in both the development team and the application, they consciously make the tradeoff between CPU time and User time – and choose infinitesimally small bits of CPU time in favor of measurably large amounts of real money paid in the form of user time.

Stop wasting my users’ time!