Virtualizing your Oracle database infrastructure without benchmarking and following best practices is a bit like trying to find your way in a strange big city without a street map. You may get where you need to go eventually, but it will be a long and difficult journey.
Fortunately, we at EMC are in the midst of that virtualization journey and have put together some benchmarks and best practices to help guide your organization’s path to running Oracle database on virtualized infrastructure. It is no more an uncharted territory.
In this blog (the second in a series on virtualizing Oracle), I will give you some insights into EMC’s journey thus far and the benchmarking process we used to set the stage for our virtual deployment best practices. I will get into those best practices in detail in the next blog in this series. [Read the first blog in the series—Running Oracle on Virtual Infrastructure Really Pays Off.]
We began virtualizing our databases about four years ago, following several years of virtualizing most of our application servers. EMC has hundreds of Oracle databases supported by IT. Since many of them were running on legacy Sun Solaris platforms, we first focused on migrating them to an open, scalable x86 platform and virtualizing them at the same time.
When we began virtualizing the database tier, we started with our development and test databases for our less-critical applications to test our theories on actual workloads and determine what worked and what didn’t. We then continued with the production environments, applying what we had learned from the test and development databases. This method of continual learning has been very successful for us, driving constant improvement of our knowledge and results.
At the time of this posting, we have virtualized about 84 percent of our database infrastructure and are now in the process of virtualizing our most mission-critical databases. This includes converting one of our mission-critical Oracle RAC environments (a clustered database) to a non-RAC (or single instance database), as well as virtualizing one of our high-performance Oracle RAC databases in production.
A crucial part of our learning process is testing the virtualization framework in our IT Innovation Lab, using real-world transactions and measuring any latency in those transactions because that would adversely impact end-user productivity. This is critical for ensuring that virtualizing databases won’t significantly reduce their performance. Our innovation lab consists of the exact same components that we have in production, from storage though the actual x86 blade servers used in the production VCE Vblock architecture. In order to measure the impact of these changes, EMC IT utilizes a custom benchmark that was developed utilizing a copy of our production CRM system.
For the benchmarking process, there are three components. The first, and most important, consists of executing the actual end-user transactions. This is performed using a tool that automates the execution of the application. We have created several scripts to perform our most critical transactions and measure the exact response time of each transaction, to the hundredths of a millisecond.
The second benchmark component measures the impact of batch processing on the system. These tests execute the exact same jobs that run in production during our peak processing times at our end of quarter. They are timed to begin at the same exact time that they would run in production and the completion time for each of them is tracked and easily queried for changes in run times.
The last component of this benchmark is running a set of 48 ad-hoc queries— those often-unpredictable requests from users that can put unexpected stress on resources and slow down the system. These are queries that were run in production mixed with queries specifically designed to drive I/O, add contention to our heavily hit tables and to exercise the RAC interconnect. All of the queries are timed to execute at the same frequency and are provided with a large set of variable values, which are picked at random.
The above benchmark has proven to closely resemble the production environment that the impacts we see in the lab exactly match the results we see in production. We have used this process to test everything from database and hardware upgrades to a pre-beta version of VMware vSphere 5.0. We also use it to determine the impact of our database best practices against all different database configuration, operating systems, and underlying file systems. More recently, we found it effective in determining how to achieve a balance between performance and efficiency for database workloads on virtualized infrastructure.
The best practices we have developed based on this process cover memory access, networking and CPU configurations. They also define how to configure storage layouts for maximum performance as well as how to balance these best practices to achieve high levels of efficiency.
Stay tuned for my next blog detailing all of the best practices EMC uses for virtualizing Oracle databases in order to gain maximum database performance and efficiency.