Setting Benchmarks Starting to Tune

- 15 - The performance should be specified for as many aspects of the system as possible, including: • Multiuser response times depending on the number of users if applicable • Systemwide throughput e.g., number of transactions per minute for the system as a whole, or response times on a saturated network, again if applicable • The maximum number of users, data, files, file sizes, objects, etc., the application supports • Any acceptable and expected degradation in performance between minimal, average, and extreme values of supported resources Agree on target values and acceptable variances with the customer or potential users of the application or whoever is responsible for performance before starting to tune. Otherwise, you will not know where to target your effort, how far you need to go, whether particular performance targets are achievable at all, and how much tuning effort those targets may require. But most importantly, without agreed targets, whatever you achieve tends to become the starting point. The following scenario is not unusual: a manager sees horrendous performance, perhaps a function that was expected to be quick, but takes 100 seconds. His immediate response is, Good grief, I expected this to take no more than 10 seconds. Then, after a quick round of tuning that identifies and removes a huge bottleneck, function time is down to 10 seconds. The managers response is now, Ah, thats more reasonable, but of course I actually meant to specify 3 seconds—I just never believed you could get down so far after seeing it take 100 seconds. Now you can start tuning. You do not want your initial achievement to go unrecognized especially if money depends on it, and it is better to know at the outset what you need to reach. Agreeing on targets before tuning makes everything clear to everyone.

1.6.2 Setting Benchmarks

After establishing targets with the users, you need to set benchmarks. These are precise specifications stating what part of the code needs to run in what amount of time. Without first specifying benchmarks, your tuning effort is driven only by the target, Its gotta run faster, which is a recipe for a wasted return. You must ask, How much faster and in which parts, and for how much effort? Your benchmarks should target a number of specific functions of the application, preferably from the user perspective e.g., from the user pressing a button until the reply is returned, or the function being executed is completed. You must specify target times for each benchmark. You should specify ranges: for example, best times, acceptable times, etc. These times are often specified in frequencies of achieving the targets. For example, you might specify that function A takes not more than 3 seconds to execute from user click to response received for 80 of executions, with another 15 of response times allowed to fall in the 3- to 5-second range, and 5 allowed to fall in the 5- to 10-second range. Note that the earlier section on user perceptions indicates that the user will see this function as having a 5-second response time the 90th percentile value if you achieve the specified ranges. You should also have a range of benchmarks that reflect the contributions of different components of the application. If possible, it is better to start with simple tests so that the system can be understood at its basic levels, and then work up from these tests. In a complex application, this helps to determine the relative costs of subsystems and which components are most in need of performance-tuning. The following point is critical: Without clear performance objectives, tuning will never be completed. This is a common syndrome on single or small group projects, where code keeps on being tweaked as better implementations or cleverer code is thought up. - 16 - Your general benchmark suite should be based on real functions used in the end application, but at the same time should not rely on user input, as this can make measurements difficult. Any variability in input times or any other part of the application should either be eliminated from the benchmarks or precisely identified and specified within the performance targets. There may be variability, but it must be controlled and reproducible.

1.6.3 The Benchmark Harness