Dont Tune What You Dont Need to Tune

- 19 - You need to be careful when running tests that have small differences in timings. The first test is usually slightly slower than any other tests. Try doubling the test run so that each test is run twice within the VM e.g., rename main to maintest , and call maintest twice from a new main . There are almost always small variations between test runs, so always use averages to measure differences and consider whether those differences are relevant by calculating the variance in the results. For distributed applications , you need to break down measurements into times spent on each component, times spent preparing data for transfer and from transfer e.g., marshalling and unmarshalling objects and writing to and reading from a buffer, and times spent in network transfer. Each separate machine used on the networked system needs to be monitored during the test if any system parameters are to be included in the measurements. Timestamps must be synchronized across the system this can be done by measuring offsets from one reference machine at the beginning of tests. Taking measurements consistently from distributed systems can be challenging, and it is often easier to focus on one machine, or one communication layer, at a time. This is usually sufficient for most tuning.

1.8 Dont Tune What You Dont Need to Tune

The most efficient tuning you can do is not to alter what works well. As they say, If it aint broke, dont fix it. This may seem obvious, but the temptation to tweak something just because you have thought of an improvement has a tendency to override this obvious statement. The second most efficient tuning is to discard work that doesnt need doing. It is not at all uncommon for an application to be started with one set of specifications and to have some of the specifications change over time. Many times the initial specifications are much more generic than the final product. However, the earlier generic specifications often still have their stamps in the application. I frequently find routines, variables, objects, and subsystems that are still being maintained but are never used and never will be used, since some critical aspect of these resources is no longer supported. These redundant parts of the application can usually be chopped without any bad consequences, often resulting in a performance gain. In general, you need to ask yourself exactly what the application is doing and why. Then question whether it needs to do it in that way, or even if it needs to do it at all. If you have third-party products and tools being used by the application, consider exactly what they are doing. Try to be aware of the main resources they use from their documentation. For example, a zippy DLL shared library that is speeding up all your network transfers is using some resources to achieve that speedup. You should know that it is allocating larger and larger buffers before you start trying to hunt down the source of your mysteriously disappearing memory. Then you can realize that you need to use the more complicated interface to the DLL that restricts resource usage, rather than a simple and convenient interface. And you will have realized this before doing extensive and useless object profiling, because you would have been trying to determine why your application is being a memory hog. When benchmarking third-party components, you need to apply a good simulation of exactly how you will use those products. Determine characteristics from your benchmarks and put the numbers into your overall model to determine if performance can be reached. Be aware that vendor benchmarks are typically useless for a particular application. Break your application down into a hugely simplified version for a preliminary benchmark implementation to test third-party components. You should make a strong attempt to include all the scaling necessary so that you are benchmarking a fully scaled usage of the components, not some reduced version that will reveal little about the components in full use. - 20 -

1.9 Performance Checklist