Expert Oracle RAC Performance Diagnostics and Tuning
For your convenience Apress has placed some of the front
matter material after the index. Please use the Bookmarks
and Contents at a Glance links to access them.
Contents at a Glance
About the Author ...............................................................................................................
Acknowledgments ............................................................................................................
■Chapter 1: Methodology
■
Chapter 2: Capacity Planning and Architecture ..............................................................
■
Chapter 3: Testing for Availability ...................................................................................
■
Chapter 4: Testing for Scalability ....................................................................................
■
Chapter 5: Real Application Testing ..............................................................................
■
Chapter 6: Tools and Utilities .........................................................................................
■
Chapter 7: SQL Tuning ...................................................................................................
■
Chapter 8: Parallel Query Tuning ...................................................................................
■
Chapter 9: Tuning the Database .....................................................................................
■
Chapter 10: Tuning Recovery .........................................................................................
■
Chapter 11: Tuning Oracle Net
■
Chapter 12: Tuning the Storage Subsystem ..................................................................
■
Chapter 13: Tuning Global Cache
■
Chapter 14: Tuning the Cluster Interconnect ................................................................
■
Chapter 15: Optimize Distributed Workload .................................................................
■
Chapter 16: Oracle Clusterware Diagnosis ...................................................................
■
Chapter 17: Waits, Enqueues, and Latches
■
Chapter 18: Problem Diagnosis .....................................................................................
■ Appendix A: The SQL Scripts Used in This Book
Index
Introduction
Working for several years across several industries on various RAC projects, there have been several occasions to troubleshoot performance issues in a production environment. Applications and databases that where moved from a single-instance Oracle environment to a two or more node RAC environment to hide a performance problem. An example that comes to mind, which I have encountered in the field on several occasions, is when the database was moved to a RAC environment because the single instance was running at 100% CPU and it was hoped that by moving to a RAC configuration, the 100% CPU overload would be distributed between the various instances in the cluster. This really does not happen this way; RAC cannot do magic to fix poorly written structured query language (SQL) statements or SQL statements that have not been optimized. The general best practice or rule of thumb to follow is when an application can scale from one CPU to multiple CPUs on a single node/instance configuration, it could potentially scale well in a RAC environment. The outcome of migrating applications that perform poorly to a RAC environment is to roll back to a single-instance configuration (by disabling the RAC/cluster parameters), testing/ tuning the application and identifying problem SQL statements, and when the application is found to successfully scale (after SQL statement tuning), it is moved to a RAC environment.
Moving to a RAC environment for the right reasons (namely, availability and scalability) should be done only when the application and environments that have been tested and proven to meet the goals. Almost always the reason for a RAC environment to crash on the third, fourth or sixth day after it’s rolled into a production environment is lack of proper testing. This is either because testing was never considered as part of the project plan or because testing was not completed due to project schedule delays. Testing the application through the various phases discussed in this book helps identify the problem areas of the application; and tuning them helps eliminate the bottlenecks. Not always do we get an opportunity to migrate a single-instance Oracle database into a RAC environment. Not always is an existing RAC environment upgraded from one type of hardware configuration to another. What I am trying to share here is the luxury to test the application and the RAC environment to get the full potential happens only once before it’s deployed into production. After this point, its primary production calls for poor response time, node evictions, high CPU utilization, faulty networks, chasing behind run of processes, and so on and so forth.
During the testing phase, taking the time to understand the functional areas of the application, how these functional areas could be grouped into database services, or mapping an application service to a database service would help place a single service or a group of services on an instance in the cluster. This would help in the distribution of workload by prioritizing system resources such as CPU, I/O, and so forth. This mapping could also help in availability partially by disabling a specific database service during a maintenance window when application changes need to be deployed into production, thus avoiding shutting down the entire database.
About This Book
The book is primarily divided into two sections: testing and tuning. In the testing section of the book, various phases of testing grouped under a process called “RAP” (Recovery, Availability, & Performance) have been defined. The second section discusses troubleshooting and tuning the problems. The style followed in the book is to use workshops through performance case studies across various components of a RAC environment.
Almost always, when a performance engineer is asked a question such as why the query is performing badly, or why the RAC environment is so slow, or why did the RAC instance crash, the expected answer should be “it depends.” This is because there could be several pieces to the problem, and no one straight answer could be the reason. If the answers are all straight and if there is one reason to a problem, we would just need a Q&A book and we would not need the mind of a technical DBA to troubleshoot the issue. Maybe a parameter “ORACLE_GO_FASTER” (OGF) could be set and all the slow-performing queries and the database would run faster. Similar to the “it depends” answer, in this book I have tried to cover most of the common scenarios and problems that I have encountered in the field, there may or may not be a direct reference to the problem in your environment. However, it could give you a start in the right direction.
How to Use This Book
The chapters are written to follow one another in a logical fashion by introducing testing topics in the earlier chapters and building on them to performance and troubleshooting various components of the cluster. Thus, it is advised that you read the chapters in order. Even if you have worked with clustered databases, you will certainly find a nugget or two that could be an eye opener.
Throughout the book, examples in the form of workshops are provided with outputs, followed by discussions and analysis into problem solving. The book contains the following chapters:
Chapter 1—Methodology Performance tuning is considered an art and more recently a science. However, it is definitely never a gambling game
where guesswork and luck are the main methods of tuning. Rather, tuning should be backed by reasons, and scientific evidence should be the determining factor. In this chapter, we discuss methodologies to approach performance tuning of the Oracle database in a precise, controlled method to help obtain successful results.
Chapter 2—Capacity Planning and Architecture Before starting the testing phase, it is important to understand how RAC works, how the various components of the
architecture communicate with each other. How many users and workload can a clustered solution handle? What is the right capacity of a server? The cluster that is currently selected may be outgrown due to increased business or high data volume or other factors. In this chapter, we discuss how to measure and ascertain the capacity of the systems to plan for the future.
Chapter 3—Testing for Availability The primary reason for purchasing a RAC configuration is to provide availability. Whereas availability is the
immediate requirement of the organization, scalability is to satisfy future needs when the business grows. When one or more instances fail, the others should provide access to the data, providing continuous availability within a data center. Similarly, when access to the entire data center or the clustered environment is lost, availability should be provided by a standby location. When components fail, one needs to ensure that the redundant component is able to function without any hiccups. In this chapter, we cover just that:
- Testing for failover
- Simulating failover
- Test criteria to be adopted
- User failover
- Tuning the failover to achieve maximum availability
such scalability is not achievable unless the hardware and the application tiers are scalable. Meaning that unless the hardware itself is scalable, the application or database cannot scale. In this chapter, we discuss the methods to be used to test the hardware and application for scalability:
Testing for data warehouse
-
Testing for OLTP
-
Testing for mixed workload systems
-
Operating system parameter tuning
-
Diagnosis and problem solving at the hardware level
-
Verification of the RAC configuration at the hardware and O/S levels
-
Using tools such as SwingBench, Hammerora, and other utilities
-
Chapter 5—Real Application Testing Once the hardware has been tested and found to be scalable, the next step is to ensure that the application will scale
in this new RAC environment. Keeping the current production numbers and the hardware scalability numbers as a baseline, one should test the application using the database replay feature of the RAT tool to ensure that the application will also scale in this new RAC environment.
Chapter 6—Tools and Utilities There are several tools to help tune an Oracle database, tools that are bundled with the Oracle RDBMS product
and others that are provided by third party vendors. In this chapter, we discuss some of the key tools and utilities that would help the database administrators and the performance analysts. A few of the popular tools are Oracle Enterprise Cloud Control, SQLT (SQL Trace), AWR (Automatic Workload Repository), AWR Warehouse, ASH (Active Session History), and ADDM (Automatic Database Diagnostic Monitor).
Chapter 7—SQL Tuning The application communicates with the database using SQL statements. This includes both storage and retrieval. If the queries are not efficient and tuned to retrieve and or store data efficiently, it directly reflects on the performance
of the application. In this chapter, we go into detail on the principles of writing and tuning efficient SQL statements, usage of hints to improve performance, and selection and usage of the right indexes. Some of the areas that this chapter will cover are the following:
Reducing physical IO
-
Reducing logical IO
-
Tuning based on wait events
-
Capturing trace to analyze query performance
-
SQL tuning advisory
-
AWR and ADDM reports for query tuning in a RAC environment
-
data in a sequential manner. They could also be executed using multiple processes and using a parallel method to retrieve all the required data. Parallelism could be on a single instance in which multiple CPUs will be used to retrieve the required data or by taking advantage of more than one instance (in a RAC environment) to retrieve data. In this chapter, we cover the following:
Parallel queries on a single node
-
Parallel queries across multiple nodes
-
Optimizing parallelism
-
Tracing parallel operations
-
Parameters to tune for parallel operations
-
SQL tuning advisory
-
AWR and ADDM reports for parallel query tuning in a RAC environment
-
Chapter 9—Tuning the Database The database cache area is used by Oracle to store rows fetched from the database so that subsequent requests for
the same information is readily available. Data is retained in the cache based on the usage. In this chapter, we discuss efficient tuning of the shared cache, the pros and cons of logical I/O operations versus physical I/O operations, and how to tune the cache area to provide the best performance. In this chapter, we discuss some of the best practices to be used when designing databases for a clustered environment.
Chapter 10—Tuning Recovery No database is free from failures; RAC that supports multiple instances is a solution for high availability and
scalability. Every instance in a RAC environment is also prone for failures. When an instance fails in a RAC configuration, another instance that detects the failure performs the recovery operation. Similarly, the RAC database also can fail and have to be recovered. In this chapter, we discuss the tuning of the recovery operations.
Chapter 11—Tuning Oracle Net The application communicates with the database via SQL statements; these statements send and receive information
from the database using the Oracle Net interface provided by Oracle. Depending on the amount of information received and sent via the Oracle Net layer, there could be a potential performance hurdle. In this chapter, we discuss tuning the Oracle Net interface. This includes tuning the listener, TNS (transparent network substrate), and SQL Net layers.
Chapter 12—Tuning the Storage Subsystem RAC is an implementation of Oracle in which two or more instances share a single copy of the physical database. This means that the database and the storage devices that provide the infrastructure should be available for access
from all the instances participating in the configuration. Efficiency of the database to support multiple instances depends on a good storage subsystem and an appropriate partitioning strategy. In this chapter, we look into the performance measurements that could be applied in tuning the storage subsystem.
Chapter 13—Tuning Global Cache Whereas the interconnect provides the mechanism for the transfer of information between the instances, the sharing
of resources is managed by Oracle cache fusion technology. All instances participating in the clustered configuration share data resident in the local cache of one instance with other process on other instances. Locking, past images, current images, recovery, and so forth normally involved in a single-instance level can also present at a higher level across multiple instances. In this chapter, we discuss tuning of the global cache.
Chapter 14—Tuning the Cluster Interconnect The cluster interconnect provides the communication link between two or more nodes participating in a clustered
configuration. Oracle utilizes the cluster interconnect for interinstance communication and sharing of data in the respective caches of the instance. This means that this tier should perform to its maximum potential, providing efficient communication of data between instances. In this chapter, we discuss the tuning of the cluster interconnect.
Chapter 15—Optimization of Distributed Workload One of the greatest features introduced in Oracle 10g is the distributed workload functionality. With this databases can
be consolidated; and by using services options, several applications could share an existing database configuration utilizing resources when other services are not using them. Efficiency of the environment is obtained by automatically provisioning services when resources are in demand and automatically provisioning instances when an instance in a cluster or server pool is not functioning.
Chapter 16—Tuning the Oracle Clusterware Oracle’s RAC architecture places considerable dependency on the cluster manager of the operating system. In this
chapter, we discuss tuning the various Oracle clusterware components: Analysis activities performed by the clusterware
-
Performance diagnosis for the various clusterware components, including ONS (Oracle
- notification services), EVMD (Event Manager Daemon), and LISTENER Analysis of AWR, ADDM reports and OS-level tools to tune the Oracle clusterware
-
Debugging and tracing clusterware activity for troubleshooting clusterware issues
-
Chapter 17—Enqueues, Waits, and Latches When tuned and optimized SQL statements are executed, there are other types of issues such as contention,
concurrency, locking, and resource availability that could cause applications to run slow and provide slow response times to the users. Oracle provides instrumentation into the various categories of resource levels and provides methods of interpreting them. In this chapter, we look at some of these critical statistics that would help optimize the database. By discussing enqueues, latches, and waits specific to a RAC environment, in this chapter we drill into the contention, concurrency, and scalability tuning of the database. Chapter 18—Problem Diagnosis To help the DBA troubleshoot issues with the environment, Oracle provides utilities that help gather statistics across all instances. Most of the utilities that focus on database performance-related statistics were discussed in Chapter 5. There are other scripts and utilities that collect statistics and diagnostic information to help troubleshoot and get to
the root cause of problems. The data gathered through these utilities will help diagnose where the potential problem could be. In this chapter, we discuss the following: Health monitor
-
Automatic Diagnostic Repository
-
Appendix A—The SQL Scripts Used in This Book The appendix provides a quick reference to all the scripts used and referenced in the book.
CHAPTER 1 Methodology
Performance tuning is a wide subject, probably a misunderstood subject; so it has become a common practice among technologists and application vendors to regard performance as an issue that can be safely left for a tuning exercise performed at the end of a project or system implementation. This poses several challenges, such as delayed project deployment, performance issues unnoticed and compromised because of delayed delivery of applications for performance optimization, or even the entire phase of performance optimization omitted due to delays in the various stages of the development cycle. Most important, placing performance optimization at the end of a project life cycle basically reduces opportunities for identifying bad design and poor algorithms in implementation. Seldom do they realize that this could lead to potentially rewriting certain areas of the code that are poorly designed and lead to poor performance.
Irrespective of a new product development effort or an existing product being enhanced to add additional functionality, performance optimization should be considered from the very beginning of a project and should be part of the requirements definition and integrated into each stage of the development life cycle. As modules of code are developed, each unit should be iteratively tested for functionality and performance. Such considerations would make the development life cycle smooth, and performance optimization could follow standards that help consistency of application code and result in improved integration, providing efficiency and performance.
There are several approaches to tuning a system. Tuning could be approached artistically like a violinist who tightens the strings to get the required note, where every note is carefully tuned with the electronic tuner to ensure that every stroke matches. Similarly, the performance engineer or database administrator (DBA) could take a more scientific or methodical approach to tuning. A methodical approach based on empirical data and evidence is a most suitable method of problem solving, like a forensic method that a crime investigation officer would use. Analysis should be backed by evidence in the form of statistics collected at various levels and areas of the system:
From functional units of the application that are performing slowly
-
During various times (business prime time) of the day when there is a significant
- user workload
- The data collected would help to understand the reasons for the slowness or poor performance because there could be one or several reasons why a system is slow. Slow performance could be due to bad configuration, unoptimized or inappropriately designed code, undersized hardware, or several other reasons. Unless there is unequivocal evidence of why performance is slow, the scientific approach to finding the root cause of the problem should be adopted. The old saying that “tuning a computer system is an art” may be true when you initially configure a system using a standard set of required parameters suggested by Oracle from the installation guides; but as we go deeper into testing a more scientific approach of data collection, mathematical analysis and reasoning must be adopted because tuning should not be considered a hit-or-miss situation: it is to be approached in a rigorous scientific manner with supporting data.
procedure or set of procedures ) is the procedure or process followed from start to finish, from identification of the
problem to problem solving and documentation. A methodology developed and followed should be a procedure or process that is repeatable as a whole or in increments through iterations.
During all of these steps or iterations, the causes or reasons for a behavior or problem should be based on quantitative analysis and not on guesswork. Every system deployed into production has to grow in the process of a regression method of performance testing to determine poorly performing units of the application. During these tests, the test engineer would measure and obtain baselines and optimize the code to achieve the performance numbers or service-level agreements (SLA) requirements defined by the business analysts.
Performance Requirements
As with any functionality and business rule, performance needs are also (to be) defined as part of business requirements. In organizations that start small, such requirements may be minimal and may be defined by user response and feedback after implementation. However, as the business grows and when the business analyst defines changes or makes enhancements to the business requirements, items such as entities, cardinalities, and the expected response time requirements in use cases should also be defined. Performance requirements are every bit as important as functional requirements and should be explicitly identified at the earliest possible stage. However, too often, the system requirements will specify what the system must do, without specifying how fast it should do it.
When these business requirements are translated into entity models, business processes, and test cases, the cardinalities, that is, the expected instances (aka records) of a business object and required performance levels should be incorporated into the requirements analysis and the modelling of the business functions to ensure these numbers could be achieved. Tagh-level requirement of a direct-to-home broadcasting system that plans to expand its systems based on the growth patterns observed over the years.
Table 1-1.
Business Requirements
Entity Current Count Maximum Count Maximum Read Maximum Update Average Growth
Access (trans/sec) Access (trans/sec) Rate (per year)Smartcards 16,750,000 90,000,000
69 73 4,250,000 Products 43,750,000 150,000,000
65 65 21,250,000 Transmission logs 400,000 536,000,000 N/A 138 670,000,000 records/day Report back files 178,600 390,000,000 N/A N/A 550,000,000 records/day records processed/year Note: trans/sec. = transactions per second; N/A = not applicable.
1. It will store for 15 million subscriber accounts.
2. Four smart cards will be stored per subscriber account.
3. Average growth rate is based on the maximum number of active smart cards.
4. The peak time for report back transactions is from midnight to 2 AM.
5. Peak times for input transactions are Monday and Friday afternoons from 3 PM to 5 PM.
6. The number of smart cards is estimated to double in 3 years. Based on an 18-hour day (peak time = 5 times average rate), today 3.5 messages are processed per second. This is projected to increase over the next 2 years to 69 messages per second.
Table gives a few requirements that help in 1. sizing the database (Requirement 1 and 6); 2. planning on the layout of the application to database access (Requirement 5); and 3. allocation of resources (Requirements 4 and 5).
These requirements with the expected transaction rate per second helps the performance engineer to work toward a goal. It’s a truism that errors made during requirements definition are the most expensive to fix in production and that missing requirements are the hardest requirements errors to correct. That is, of all the quality defects that might make it into a production system, those that occur because a requirement was unspecified are the most critical. To avoid these surprises, the methodology should take into consideration testing the application code in iterations from complex code to the least complex code and step-by-step integration of modules when the code is optimal.
Missing detailed requirements lead to missing test cases: if we don’t identify a requirement, we are unlikely to create a performance test case for the requirement. Therefore, application problems caused by missing requirements are rarely discovered prior to the application being deployed.
During performance testing, we should create test cases to measure performance of every critical component and module interfacing with the database. If the existing requirements documents do not identify the performance requirements for a business-critical operation, they should be flagged as “missing requirement” and refuse to pass the operation until the performance requirement is fully identified and is helpful in creating a performance test case.
In many cases, we expect a computer system to produce the same outputs when confronted with the same inputs—this is the basis for most test automation. However, the inputs into a routine can rarely be completely controlled. The performance of a given transaction will be affected by
The number of rows of data in the database
-
Other activity on the host machine that might be consuming CPU, memory, or performing
- disk input/output (I/O) The contents of various memory caches—including both database and operating system (O/S)
- cache (and sometimes client-side cache) Other activity on the network, which might affect network round-trip time
- Unless there is complete isolation of the host that supports the database and the network between the application client (including the middle tier if appropriate), you are going to experience variation in application performance.
Therefore, it’s usually best to define and measure performance taking this variation into account. For instance, transaction response times maybe expressed in the following terms:
1. In 99% of cases, Transaction X should complete within 5 seconds.
2. In 95% of cases, Transaction X should complete within 1 second. The end result of every performance requirement is to provide throughput and response times to various user requests. Within the context of the business requirements the key terminologies used in these definitions should also be defined: for instance, 95% of cases; Transaction X should complete within 1 second. What’s a transaction in this context? Is it the time it takes to issue the update statement? Or is it the time it takes for the user to enter something and press the “update” or “commit” button? Or yet, is it the entire round-trip time between the user pressing the “OK” button and the database completing the operation saving or retrieving the data successfully and returning the final results back to the user?
Early understanding of the concepts and terminology along with the business requirements helps all stack holders of the project to have the same viewpoint, which helps in healthy discussions on the subject.
- Throughput: Number of requests processed by the database over a period of time normally measured by number of transactions per second.
- Response time: Responsiveness of the database or application to provide the requests results over a stipulated period of time, normally measured in seconds.
In database performance terms, the response time could be measured as database time or db time. This is the amount of time spent by the session at the database tier performing operations and in the process of completing its operation, waiting for resources such as CPU, disk I/O, and so forth.
Tuning the System Structured tuning starts by normalizing the application workload and then reducing any application contention.
After that is done, we try to reduce physical I/O requirements by optimizing memory caching. Only when all of that is done do we try to optimize physical I/O itself.
Step 1: Optimizing Workload
There are different types of workloads:
- Workloads that have small quick transactions returning one or few rows back to the requestor
- Workloads that return a large number of rows (sequential range scan of the database) back to the requestor
- A mixed workload where the users sometimes request for small random rows; however, they can also request a large number of rows
The expectations are for applications to provide good response to various types of workloads. Optimization of database servers should be in par with the workloads they can support. Overcomplicating the tuning effort to extract the most out of the servers may not give sufficient results. Therefore, before looking at resource utilization such as memory, disk I/O, or upgrading hardware, it’s important to ensure that the application is making optimal demands on the database server. This involves finding and tuning the persistence layer consuming excessive resources. Only after this layer is tuned should the database or O/S level tuning be considered.
Step 2: Finding and Eliminating Contention
Most applications making requests to the database will perform database I/O or network requests, and in the process of doing this consumes CPU resources. However, if there is contention for resources within the database, the database and its resources may not scale well. Most database contention could be determined using Oracle’s wait interface by
V$SESSION, V$SESSION_WAIT, V$SYSTEM_WAIT, V$EVENT_NAME, and V$STATNAME. High wait events related to
querying latches and buffers should be minimized. Most wait events in a single instance (non-Real Application Clusters [RAC])
gc buffer
configuration represent contention issues that will be visible in RAC as global events, such as global cache
busy. Such issues should be treated as single instance issues and should be fixed before moving the application to a RAC configuration.
■ Note Oracle wait interface is discussed in Chapters 6, 8, and 17.
Step 3: Reduce Physical I/O
Most database operations involve disk I/Os, and it can be an expensive operation relative to the speed of the disk and other I/O components used on the server. Processing architectures have three major areas that would require or demand a disk I/O operation:
1. A logical read by a query or session does not find data in the cache and hence has to perform a disk I/O because the buffer cache is smaller than the working set.
2. SORT and JOIN operations cannot be performed in memory and need to spill to the TEMP table space on disk.
3. Sufficient memory is not found in the buffer cache, resulting in the buffers being prematurely written to disk; it is not able to take advantage of the lazy writing operation. Optimizing physical I/O (PIO) or disk I/O operations is critical to achieve good response times. For disk
I/O intensive operations, high-speed storage or using storage management solutions such as Automatic Storage Management (ASM) will help optimize PIO.
Step 4: Optimize Logical I/O
Reading from a buffer cache is faster compared to reading from a physical disk or a PIO operation. However, in Oracle’s architecture, high logical I/O (LIOs) is not so inexpensive that it can be ignored. When Oracle needs to read a row from buffer, it needs to place a lock on the row in buffer. To obtain a lock, Oracle has to request a latch; for instance, in the case of a consistent read (CR) request, a latch on buffer chains has to be obtained. To obtain a latch, Oracle has to depend on the O/S. The O/S has limitations on how many latches can be made available at a given point in time. The limited number of latches are shared by a number of processes. When the requested latch is not available, the process will go into a sleep state and after a few nanoseconds will try for the latch again. Every time a latch is requested there is no grantee that the requesting process may be successful in getting the latch and may have to go into a sleep state again. The frequent trying to obtain a latch leads to high CPU utilization on the host server and cache buffer chains latch contention as sessions try to access the same blocks. When Oracle has to scan a large number of rows in the buffer to retrieve only a few rows that meet the search criteria, this can prove costly. LIO should be reduced as much as possible for efficient use of CPU and other resources. In a RAC environment this becomes even more critical because there are multiple instances in the cluster, and each instance may perform a similar kind of operation. For example, another user maybe executing the very same statement retrieving the same set of rows and may experience the same kind of contention. In the overall performance of the RAC, environment will indicate high CPU usage across the cluster.
■ Note LIO is discussed in Chapter 7 and Latches are discussed in Chapter 17. Methodology
Problem-solving tasks of any nature need to be approached in a systematic and methodical manner. A detailed procedure needs to be developed and followed from end to end. During every step of the process, data should be collected and analyzed. Results from these steps should be considered as inputs into the next step, which in turn is performed in a similar systematic approach. Hence, methodology is the procedure or process followed from start to finish, from identification of the problem to problem solving and documentation.
During all this analysis, the cause or reasons for a behavior or problem should be based on quantitative analysis and not on guesswork or trial and error.
USING DBMS_ APPLICATION INFO
A feature that could help during all the phases of testing, troubleshooting, and debugging of the application is
the use of the DBMS_APPLICATION_INFO package in the application code. The DBMS_APPLICATION_INFO package
has procedures that will allow modularizing performance data collection based on specific modules or areas within modules.
Incorporating the DBMS_APPLICATION_INFO package into the application code helps the administrators to easily
track the sections of the code (module/action) that are high resource consumers. When the user/applicationsession registers a database session, the information is recorded in V$SESSION and V$SQLAREA . This helps in easy
identification of the problem areas of the application.
The application should set the name of the module and name of the action automatically each time a user enters
that module. The name given to the module could be the name of the code segment in an Oracle pre-compiler
application or service within the Java application. The action name should usually be the name or description of
the current transaction within a module.Procedures Procedure Description SET_CLIENT_INFO Sets the CLIENT_INFO field of the session.
SET_MODULE Sets the name of the module that is currently running. SET ACTION Sets the name of the current action within the current module. SET_SESSION_LONGOPS Sets a row in the GV$SESSION_LONGOPS table.
When the application connects to the database using a database service name (either using a Type 4 client or a
Type 2 client) then even a granular level of resource utilization for a given service, module, and/or action could be
collected. Database service names are also recorded in GV$SESSION .One of the great benefits of enabling the DBMS_APPLICATION_INFO package call in the application code is that the
database performance engineer can enable statistics collection or enable tracing when he/she feels it’s needed
and at what level it’s needed.Methodologies could be different depending on the work involved. There could be methodologies for
- Development life cycle
- Migration • Testing • Performance tuning
Performance Tuning Methodology The performance tuning methodology can be broadly categorized into seven steps.
Problem Statement
Identify or state the specific problem in hand. This could be different based on the type of application and the phase of the development life cycle. When a new code is being deployed into production, the problem statement is to meet the requirements for response time and transaction per second and the recovery time. The business analysts, as we have discussed earlier, define these requirements. Furthermore, based on the type of requirement being validated, the scope may require some additional infrastructure such as data guard configuration for disaster recovery.
On the other hand, if the code is already in production, then the problem statement could be made in terms of slow response time that the users have been complaining about; a dead lock situation that has been encountered in your production environment; an instance in a RAC configuration that crashes frequently, and so forth.
A clear definition of the tuning objective is a very important step in the methodology because it basically defines what is going to be achieved in the testing phase or test plan that is being prepared.
Information Gathering Gather all information relating to the problem identified in step one. This depends on the problem being addressed.
If this is a new development rollout, the information gathering will be centered on the business requirements, the development design, entity model of the database, the database sizing, the cardinality of the entities, the SLA requirements, and so forth. If this is an existing application that is already in production, the information-gathering phase may be around collecting statistics, trace, log, or other information. It is important to understand the environment, the configuration, and the circumstances around the performance problem. For instance, when a user complains of poor performance, it may be a good idea to interview the user. The interview can consist of several levels to understanding the issue.
What kind of functional area of the application was used, and at what time of the day was the operation performed? Was this consistently occurring every time during the same period in the same part of the application (it is possible that there was another contending application at that time, which may be the cause of the slow performance)? This information will help in collecting data pertaining to that period of the day and will also help in analyzing data from different areas of the applications, other applications that access the database, or even applications that run on the same servers.
Once user-level information is gathered, it may be useful to understand the configuration and environment in general:
SINGLETON on one instance
- Does the application use database services? Is the service running as
UNIFORM)? What other services are running on these servers?
or more than one instance (
- Is the cluster configured to use server pools?
- What resource plans have been implemented to prioritize the application (if any)?
/var/log/messages from the system administrators
- Collecting data from the
- Adding additional debug flags to the cluster services to gather additional information in the various GRID (Cluster Ready Services [CRS]) infrastructure log files and so forth
In Oracle Database 11g Release 2, and recently in Oracle Database 12c Release 1, there are several additional components added to the clusterware, which means several more log files (illustrated in Figureo look into when trying to identify reasons for problems.
acfs ohasd orarootagent_root crsd oragent_oracle cssd oracssdmonitor_root ohasd evmd oracssdagent_root agent
diag
gpnpd orarootagent_root gnsd oragent_oracle
<nodename>
client crsd
GRID HOME log e.g.. ssky1l1p1 scriptagent_oracle gipcd diskmon srvm
crs
admin ctssd mdnsd racgmain racg racgevtf cvu racgeut crflogd crfmond alert<nodename>.log
Figure 1-1. Oracle 11g R2 grid component log files Area Identification
Once the information concerning the performance issue is gathered, the next step is to identify the area of the application system that is reported to have a performance issue. Most of the time, the information gathered during the previous step of the methodology is sufficient. However, this may require a fine-grained look at the data and statistics collected.
If the issue was with the instance or a server crashing in the RAC environment, data related to specific modules, such as the interconnect, data related to the heartbeat verification via the interconnect, and the heartbeat verification against the voting disks have to be collected. For example, a detailed look at the data in the GRID infrastructure log
crsctl debug log css "CSSD:9") to get the clusterware to write
files may have to be analyzed after enabling debug ( more data into these log files. If this is a performance-related concern, then collecting data using a trace from the user t the minimum collecting trace using event 10046, would be really helpful.