Example: Best Practice Case Study

Managing System Processes 3-37

3.6.4 Optimizing Batch Loader Performance

This section provides some basic guidelines that you can use to improve Batch Loader performance. These suggestions can minimize potentially slow batch load performance when you are checking in a large number of content items. In many cases, proper tuning for batch loading can significantly speed up a slow server. To minimize batch loading slow downs, try implementing the following Batch Loader adjustments: ■ Temporarily disable other activities such as shutting down Inbound Refinery see the Oracle Fusion Middleware Administrators Guide for Conversion and suspending the automatic update cycle feature of the Repository Manager. See Section A.1.3.1, Repository Manager: Indexer Tab. ■ Analyze your database usage during a batch load to help the database query optimizer. Databases have built-in optimizer utilities that can help make database queries more efficient. However, to maximize the efficiency of optimizers, it is necessary to update or re-create the statistics about the physical characteristics of a table and the associated indexes. These characteristics include number of records, number of pages, and the average record length. The optimizers use these statistics to access data. Each database has a proprietary command that you can use to invoke the statistic update or recreation process. For example: – For Oracle, use the ANALYZE TABLE COMPUTE STATISTICS command – For SQL Server, use the CREATE STATISTICS statement – For DB2, use the RUNSTATS command

3.6.4.1 Example: Best Practice Case Study

This case study describes a very slow load batch performance and the steps taken to diagnose and correct the situation. This information can serve as a model for isolating underlying issues and resolving batch loading performance problems.

3.6.4.1.1 Background Information A user wanted to load 27,000 content items into the

Oracle Content Server instance that was running on an AIX server. The DB2 database was running on a separate AIX server. The content items included TIFs as the native files and corresponding PDFs as the Web-viewable files. Inbound Refinery generated thumbnails from the native files. Initially during the batch load, the performance was acceptable with sub-second insert times. However, after a few thousand content items were loaded, the performance began to degrade. Content items started to require a few seconds to load and, eventually, the load time was over 10 seconds per content item.

3.6.4.1.2 Preliminary Troubleshooting While the batch load was running, nothing seemed

to be wrong with the Oracle Content Server instance. It had sufficient memory, the CPU utilization was low less than 5, and there were no disk bottlenecks. The Inbound Refinery server was busy, but was processing thumbnails at an acceptable rate. Two issues were found with the database server: ■ Two processes were taking turns to update the database. While one process was executing, the second process waited for other process to release database locks. When the first process completed, the second process executed while the first process waited. The processes in this executewait cycle included: 3-38 Oracle Fusion Middleware System Administrators Guide for Oracle Content Server – The actual batch load process that was updating the database tables after inserting a content item. – The Oracle Content Server instance was updating the database tables; changing the status from GENWWW to DONE after receiving notification that a thumbnail had been completed. The two processes should not have been contending with each other because they were not updating the same content items. It seemed that the two processes were locking each other out because DB2 had performed lock escalation and was now locking entire database pages instead of single rows. ■ There were a large number of tablespace scans being performed by both processes.

3.6.4.1.3 Solution A two-step solution was used:

1. Inbound Refinery was shut down to prevent the status update process from competing with the batch loading process. The performance did improve because there was a 2000+ backlog of content items from the completed thumbnails. 2. A RUNSTATS command was issued on all the Oracle Content Server database tables to update the table statistics. This dramatically improved the performance of the batch load. The insert time returned to sub-second and the batch load completed within a short amount of time. It took 21 hours to insert the first 22,000 content items. After updating the table statistics, the remaining 5,000 content items were inserted in 13 minutes.

3.7 Finding Status and Error Information