VMworld Europe 2019 Virtualizing Applications Track Highlights – Oracle

Oracle Workshop

 

Architecting and Implementing Oracle Workloads on VMware Technologies [BCA2171TE]

 

Thursday November 7th

 

Lead Instructor – Sudhir Balasubramanian @vRacDba

 

On a mission to arm yourself with the latest knowledge and skills needed to master Oracle workloads virtualization? This VMworld workshop will prepare you to lead the virtualization effort in your organization, with instructor-led demos and in-depth course work designed to put you in the ranks of the IT elite. The class will provide the attendee the opportunity to learn the essential skills necessary to run Oracle implementations on VMware vSphere. The best practices and optimal approaches to deployment, operation and management of Oracle database and application software will be presented by VMware vExpert and Oracle ACE Sudhir Balasubramanian who will be joined by other VMware and industry experts. This technical workshop will exceed the standard breakout session format by delivering “real-life,” instructor-led, live training and incorporating the recommended design and configuration practices for architecting Business Critical Databases on VMware vSphere infrastructure. Subjects such as Real Applications Clusters, Automatic Storage Management, vSAN and NSX , Oracle with vSphere Hardware Accelerators (PVRDMA and Persistent Memory) as well as Oracle running on the VMware Cloud on AWS will be covered in depth.

VMworld Europe this year has a lot to offer for everyone interested in running Oracle workloads efficiently and effectively on VMware vSphere – check out these events and ensure to add them to your schedule.

See you at VMworld 2019 Europe

Posted in Uncategorized

A Lean, Mean, Fighting Machine – Oracle RAC on VMware Cloud on AWS (SDDC 1.8v2)

Introducing …. the new lean, mean, fighting machine!! Oracle RAC on VMware Cloud on AWS SDDC 1.8v2 , all muscles, no fat!!

 

 

No more need to reserve all the storage upfront for installing and running Oracle RAC on VMware Cloud on AWS starting 1.8v2 !!

 

 

“To be precise” – No more Eager Zero Thick disk(s) needed for Oracle RAC shared vmdk’s on VMware Cloud on AWS starting 1.8v2. Read on.

 

 

 

 

 

Oracle RAC on VMware vSAN

 

By default, in a VM, the simultaneous multi-writer “protection” is enabled for all. vmdk files ie all VM’s have exclusive access to their vmdk files. So, in order for all of the VM’s to access the shared vmdk’s simultaneously, the multi-writer protection needs to be disabled. The use case here is Oracle RAC.

KB 2121181 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s on VMware vSAN .

Requirement for shared disks with the multi-writer flag setting for a RAC environment is that the shared disk is has to be set to Eager Zero Thick provisioned.

In vSAN , the Eager Zero thick attribute can be controlled by a rule in the VM Storage Policy called ‘Object Space Reservation’ which need to be set to 100% which pre-allocates all object’s components on disk.

More details can be found in KB 2121181.

 

 

 

 

Oracle RAC on VMware Cloud on AWS

 

VMware Cloud on AWS is an on-demand service that enables customers to run applications across vSphere-based cloud environments with access to a broad range of AWS services. Powered by VMware Cloud Foundation, this service integrates vSphere, vSAN and NSX along with VMware vCenter management, and is optimized to run on dedicated, elastic, bare-metal AWS infrastructure. ESXi hosts in VMware Cloud on AWS reside in an AWS availability Zone (AZ) and are protected by vSphere HA.

The ‘Oracle Workloads and VMware Cloud on AWS: Deployment, Migration, and Configuration ‘ can be found here.

 

 

 

 

What’s New Sept 24th, 2019 (SDDC Version 1.8v2)

 

As per the updated release notes, starting VMware Cloud on AWS SDDC version v1.8v2 , vSAN datastores in VMware Cloud on AWS now support multi-writer mode on Thin-Provisioned VMDKs.

Previously, prior SDDC Version 1.8v2, vSAN required VMDKs to be Eager Zero Thick provisioned for multi-writer mode to be enabled. This change enables deployment of workloads such as Oracle RAC with Thin-Provisioned, multi-writer shared VMDKs.

More details about this change can be found here.

What this means is customers NO longer have to provision storage up-front when provisioning space for an Oracle RAC shared storage on VMware Cloud on AWS SDDC version 1.8 v2.

The RAC shared vmdk’s can be thin provisioned to start with and can slowly consume space as and when needed.

 

 

 

 

Key points to take away from this blog

 

This blog focuses on validating the various test cases which was run on

  • 6 node Stretched Clusters for VMware Cloud on AWS SDDC Version 1.8v2
  • 2 Node 18c Oracle RAC cluster on Oracle Linux Server release 7.6
  • Database Shared storage on Oracle ASM using Oracle ASMLIB

 

 

Key points to take away

  • Prior SDDC Version 1.8v2, Oracle RAC on VMware Cloud on AWS requires the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled
  • Starting SDDC Version 1.8v2, Oracle RAC on VMware Cloud on AWS does NOT require the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled
    • shared VMDKs can be thin provisioned (OSR=0) for multi-writer mode to be enabled
  • For existing Oracle RAC deployments
    • Storage Policy change of a shared vmdk from OSR=100 to OSR=0 (and vice-versa if needed) can be performed ONLINE without any Oracle RAC downtime

 

 

 

Setup – Oracle RAC on VMware Cloud on AWS SDDC Version 1.8v2

 

The VMware Cloud on AWS SDDC Version is 1.8v2 as shown below.

 

 

 

 

Setup is a 6 node i3 VMware Cloud on AWS SDDC cluster stretched across 2 Availability Zones (AZ’s) , ‘us-west-2a’ and ‘us-west-2b’.

 

 

 

Each ESXi server in the i3 VMware Cloud on AWS SDDC has 2 sockets, 18 cores per socket with 512GB RAM. Details of the i3 SDDC cluster type can be found here.

 

 

 

An existing 2 Node 18c Oracle RAC cluster with RAC VM’s ‘rac18c1’ and ‘rac18c2’ , operating system Oracle Linux Server release 7.6, was chosen for the test cases.

This RAC cluster had shared storage on Oracle ASM using Oracle ASMLIB. The shared disks for the RAC shared vmdk’s were provisioned with OSR=100 as per KB 2121181.

 

 

 

 

As part of creating any Oracle RAC cluster on vSAN, the first step is to create a VM Storage Policy that will be applied to the virtual disks  which will be used as the cluster’s shared storage.

In this example we had 2 VM Storage Policies

  • ‘Oracle RAC VMC AWS Policy – OSR 100 RAID 1’
  • ‘Oracle RAC VMC AWS Policy – OSR 0 RAID 1’

 

The VM storage policy ‘Oracle RAC VMC AWS Policy – OSR 100 RAID 1’ is shown as below. The rules include

  • Failures to tolerate (FTT) set to 1 failure – RAID-1 (Mirroring)
  • Number of disk stripes per object set to 1
  • Object Space Reservation (OSR) set to Thick provisioning

 

 

 

The VM storage policy ‘Oracle RAC VMC AWS Policy – OSR 0 RAID 1’ is shown as below. The rules include

  • Failures to tolerate (FTT) set to 1 failure – RAID-1 (Mirroring)
  • Number of disk stripes per object set to 1
  • Object Space Reservation (OSR) set to Thin provisioning

 

 

 

Currently both RAC VM’s have shared vmdk’s with VM storage policy ‘Oracle RAC VMC AWS Policy – OSR 100 RAID 1’ and are complaint with the storage policy.

 

 

 

 

Details of RAC VM ‘rac18c1’ is shown as below.

 

 

 

RAC VM ‘rac18c1’ has 8 vCPU’s, 32GB RAM with 3 vmdk’s.

  • Non-shared vmdk’s
    • SCSI 0:0 rac18c1.vmdk of size 60G [ Operating System ]
    • SCSI 0:0 rac18c1_1.vmdk of size 60G [ Oracle Grid and RDBMS Binaries]
  • Shared vmdk
    • SCSI 2:0 rac18c1_3.vmdk of size 3TB [ Database ]

 

 

Details of the 3TB shared vmdk can be seen below. Notice that the shared vmdk is provisioned with VM storage policy ‘Oracle RAC VMC AWS Policy – OSR 100 RAID 1’.

 

 

 

Details of RAC VM ‘rac18c2’ is shown as below. Both VM’s are the same from CPU and Memory perspective,

 

 

RAC VM ‘rac18c2’ has 8 vCPU’s, 32GB RAM with 3 vmdk’s.

  • Non-shared vmdk’s
    • SCSI 0:0 rac18c2.vmdk of size 60G [ Operating System ]
    • SCSI 0:0 rac18c2_1.vmdk of size 60G [ Oracle Grid and RDBMS Binaries]
  • Shared vmdk
    • SCSI 2:0 rac18c1_3.vmdk of size 3TB [ Database ]

 

Details of the 3TB shared vmdk can be seen below. Notice that RAC VM ‘rac18c2’ points to RAC VM ‘rac18c1’ shared vmdk which has been provisioned with VM storage policy ‘Oracle RAC VMC AWS Policy – OSR 100 RAID 1’.

 

 

 

Operating System ‘fdisk’ command reveals the OS sees a 3TB disk on both VM’s

 

 

[root@rac18c1 ~]# fdisk -lu /dev/sdd
Disk /dev/sdd: 3298.5 GB, 3298534883328 bytes, 6442450944 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4f8c7d76
Device Boot Start End Blocks Id System
/dev/sdd1 2048 4294967294 2147482623+ 83 Linux
[root@rac18c1 ~]#

 

[root@rac18c2 ~]# fdisk -lu /dev/sdd
Disk /dev/sdd: 3298.5 GB, 3298534883328 bytes, 6442450944 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4f8c7d76
Device Boot Start End Blocks Id System
/dev/sdd1 2048 4294967294 2147482623+ 83 Linux
[root@rac18c2 ~]#

 

 

Looking at the actual space consumed for RAC VM ‘rac18c1’ , the shared vmdk ‘rac18c1_3.vmdk’ shows up as 6TB , reason the VM storage policy had

  • OSR=100 , so Eager Zero Thick provisioned
  • Failures to tolerate (FTT) set to ‘1 failure – RAID-1 (Mirroring)’ , so 2 copies

 

 

 

 

The actual space consumed for RAC VM ‘rac18c2’ is shown as below.

 

 

 

All Cluster Services & Resource are up.

 

[root@rac18c1 ~]# /u01/app/18.0.0/grid/bin/crsctl stat res -t
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Local Resources
——————————————————————————–
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE rac18c1 STABLE
ONLINE ONLINE rac18c2 STABLE
ora.DATA_DG.dg
ONLINE ONLINE rac18c1 STABLE
ONLINE ONLINE rac18c2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rac18c1 STABLE
ONLINE ONLINE rac18c2 STABLE
ora.chad
ONLINE ONLINE rac18c1 STABLE
ONLINE ONLINE rac18c2 STABLE
ora.net1.network
ONLINE ONLINE rac18c1 STABLE
ONLINE ONLINE rac18c2 STABLE
ora.ons
ONLINE ONLINE rac18c1 STABLE
ONLINE ONLINE rac18c2 STABLE
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac18c2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac18c1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac18c1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac18c1 169.254.10.195 192.1
68.140.207,STABLE
ora.asm
1 ONLINE ONLINE rac18c1 Started,STABLE
2 ONLINE ONLINE rac18c2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE rac18c1 STABLE
ora.mgmtdb
1 ONLINE ONLINE rac18c1 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE rac18c1 STABLE
ora.rac18c.db
1 ONLINE ONLINE rac18c1 Open,HOME=/u01/app/o
racle/product/18.0.0
/dbhome_1,STABLE
2 ONLINE ONLINE rac18c2 Open,HOME=/u01/app/o
racle/product/18.0.0
/dbhome_1,STABLE
ora.rac18c1.vip
1 ONLINE ONLINE rac18c1 STABLE
ora.rac18c2.vip
1 ONLINE ONLINE rac18c2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac18c2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac18c1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac18c1 STABLE
——————————————————————————–
[root@rac18c1 ~]#

 

 

Database Details

  • Recommendation is to have multiple ASM disks group on multiple PVSCSI controllers as per the Oracle Best Practices Guide
  • RAC database has 1 ASM disk group ‘DATA_DG’ with 1 ASM Disk 3TB created for sake of simplicity to show all the test cases
    • Actual ASM Disk provisioned size = 3TB
    • Database Space used < 3TB
  • Tablespace ‘SLOB’ was created on the ASM Disk Group ‘DATA_DG’ of size 1TB and loaded with SLOB data.
  • ASM Disk Group ‘DATA_DG’ also houses all the other components of the Database and the RAC Cluster including OCR, VOTE, Redo Log file, Data Files etc.

 

 

  

 

Test Suite Setup

 

Key points to be kept in mind as part of the test cases

  • Prior to SDDC Version 1.8v2, Oracle RAC on VMware Cloud on AWS required the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled
  • Starting SDDC Version 1.8v2, Oracle RAC on VMware Cloud on AWS does NOT require the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled
    • shared VMDKs can be thin provisioned (OSR=0) for multi-writer mode to be enabled
  • Existing / Current RAC vmdk’s were provisioned as EZT using OSR=100
  • Online change of Storage Policy of a shared vmdk from OSR=100 to OSR=0 and vice-versa can be performed without any Oracle RAC downtime

 

 

 

As part of the test suite, 5 test cases were performed against the RAC cluster

  • Current shared vmdk Storage Policy
    • ‘Oracle RAC VMC AWS Policy – OSR 100 RAID 1’
  • Create a new VM Storage Policy
    • ‘Oracle RAC VMC AWS Policy – OSR 0 RAID 1’
  • Test Case 1-4
    • Using existing ASM Disk Group (DATA_DG)
      • Change Storage Policy of the shared vmdk’s OSR=100 to OSR=0 online without any RAC downtime
      • Change Storage Policy of the shared vmdk’s OSR=0 to OSR=100 online without any RAC downtime
      • Repeat the above 2 tests again just for consistency sake and check DB and OS error logs
  • Test Case 5
    • Using existing ASM Disk Group (DATA_DG)
      • ASM Disk Group (DATA_DG) has 1 ASM disk with VM Storage Policy OSR=100
      • Add a new vmdk with VM Storage Policy OSR=0
      • Guest Operating System only goes off the vmdk provisioned size
        • no idea what the actual allocated vmdk size is on the underlying storage
      • Drop old ASM disk with VM Storage Policy OSR=100 with ASM rebalance operation
      • Check DB and OS error logs

 

SLOB Test Suite 2.4.2 was used with run time of 15 minutes with 128 threads

  • UPDATE_PCT=30
  • SCALE=10G
  • THINK_TM_FREQUENCY=0
  • RUN_TIME=900

 

All test cases were performed and the results were captured . Only Test Case 1 is published as part of this blog. The other test cases had the same exact outcome and hence is not included in this blog.

 

 

Test Case 1: Online change of Storage Policy for shared vmdk with no RAC downtime

 

As part of this test case, while the Oracle RAC was running, the Storage Policy of the shared vmdk was changed from ‘Oracle RAC VMC AWS Policy – OSR 100 RAID 1’ to ‘Oracle RAC VMC AWS Policy – OSR 0 RAID 1’.

 

1) Start SLOB load generator for 15 minutes against the RAC database with 128 threads against the SCAN to load balance between the 2 RAC instances

 

2) Edit settings for RAC VM ‘rac18c1’.

 

 

 

 

3) Change the shared vmdk Storage Policy from OSR=100 to OSR=0

 

 

 

 

 

4) Edit settings for RAC VM ‘rac18c2’ and change shared vmdk Storage Policy from OSR=100 to OSR=0

 

 

 

 

5) Check both RAC VM’s ‘rac18c1’ and ‘rac18c2’ Storage Policy Compliance

 

 

 

6) In the Datastore view,  Check RAC VM ‘rac18c1’ shared vmdk ‘rac18c1_3.vmdk’ size.

 

  • As per VM settings, the Allocated Size is 3 TB but
  • Actual size on vSAN datastore is < 3TB

 

This indicates that the shared vmdk is indeed thin-provisioned , not thick provisioned.

 

 

 

 

7) The O/S still reports the ASM disks with original size i.e 3 TB

 

[root@rac18c1 ~]# fdisk -lu /dev/sdd
Disk /dev/sdd: 3298.5 GB, 3298534883328 bytes, 6442450944 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4f8c7d76
Device Boot Start End Blocks Id System
/dev/sdd1 2048 4294967294 2147482623+ 83 Linux
[root@rac18c1 ~]#

 

[root@rac18c2 ~]# fdisk -lu /dev/sdd
Disk /dev/sdd: 3298.5 GB, 3298534883328 bytes, 6442450944 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4f8c7d76
Device Boot Start End Blocks Id System
/dev/sdd1 2048 4294967294 2147482623+ 83 Linux
[root@rac18c2 ~]#

 

 

8) No errors were observed in the database Alert log files and OS /var/log/messages file for both RAC instances

 

 

Summary of the test matrix along with the outcome for the various test cases are as below.

 

 

 

 

Summary

 

  • Prior to SDDC Version 1.8v2, Oracle RAC on VMware Cloud on AWS required the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled
  • Starting SDDC Version 1.8v2, Oracle RAC on VMware Cloud on AWS does NOT require the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled
    • shared VMDKs can be thin provisioned (OSR=0) for multi-writer mode to be enabled
  • Online change of Storage Policy of a shared vmdk from OSR=100 to OSR=0 and vice-versa can be performed without any Oracle RAC downtime
  • All above test cases were performed and the results were captured . Only Test Case 1 is published as part of this blog. The other test cases had the same exact outcome and hence is not included in this blog.

 

All Oracle on vSphere white papers including Oracle on VMware vSphere / VMware vSAN / VMware Cloud on AWS , Best practices, Deployment guides, Workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop
http://www.vmw.re/oracle-one-stop-shop

Posted in Uncategorized

Oracle on VMware Support Policy changes Oct 9, 2019 – Metalink Note 249212.1

At the 2019 Oracle Open World conference, the week of Sept 16, 2019 , Oracle and VMware announced a new strategic alliance which resulted in Oracle becoming a member of the VMware Cloud Partner Program (VCPP) by offering the VMware Cloud Foundation based product names “Oracle Cloud VMware Solution” (OCVS) , sold by Oracle and its partners

 

More details on the VMware Oracle partnership can be found here.

 

The 2nd part of this strategic alliance is a new agreement for enhanced support for Oracle products running on VMware Environments running on Oracle Supported Computing Environments.

 

Oracle Corporation has now changed its landmark Metalink/MyOracleSupport.com note 249212.1 describing that mutual support.  

The new text from the Metalink/MyOracleSupport.com note 249212.1 is listed below.

Oracle customers with an active support contract and running supported versions of Oracle products will receive assistance from Oracle when running those products on VMware virtualized environments.

If Oracle identifies the underlying issue is not caused by Oracle’s products or is being run in a computing environment not supported by Oracle, Oracle will refer customers to VMware for further assistance and Oracle will provide assistance to VMware as applicable in resolving the issue.

This support policy does not affect Oracle or VMware licensing policies.

NOTE: For Oracle Real Application Clusters (RAC), Oracle will only accept Service Requests as described in this note on Oracle RAC 11.2.0.2 and later release.

 

 

Screenshot of updated Metalink Note 249212.1 , as of October 10 , 2019 can be found below

 

 

https://support.oracle.com/epmos/faces/DocumentDisplay?id=249212.1

 

 

The previous language had been considered somewhat vague by some customers who should now be confident that Oracle will, as stated in the note support their Oracle products running on VMware virtualized environments.

 

Previous version of the Metalink note 249212.1, as of Sept 23, 2019 can be found below.

 

 

 

Congratulations to all Oracle and VMware personnel who worked for many months to ensure that Oracle on VMware customers could continue to run these world leading products in tandem.

Posted in Uncategorized

Oracle Cloud VMware Solution – A New beginning of a Strategic relationship

“One small step for VMware & Oracle , one giant leap for VMware & Oracle mutual customers”

 

 

 

VMware is thrilled to announce a new strategic alliance with Oracle Corp to support our mutual customers’ hybrid cloud strategies.

The centerpiece of this relationship is a new service that will be offered by Oracle called Oracle Cloud VMware Solution.

Full text of the Oracle Cloud VMware Solution announcement and press release can be found here :

http://www.vmware.com/go/vmware-oracle-cloud

 

 

This service will enable customers to run VMware Cloud Foundation on Oracle’s Generation 2 Cloud infrastructure. But equally important for the thousands of customers running Oracle software on VMware today in their data centers, Oracle and VMware have entered into a new mutual support agreement for currently supported Oracle products running on VMware environments.

Quote from the mutual support agreement.  “Oracle shall, using reasonable efforts, support Oracle customers with active support contracts running supported versions of Oracle products in Oracle supported computing environments who encounter problems when running those products on VMware virtualized environments.”

 

 

Oracle on VMware customers can now run and implement with enhanced confidence that two of their most strategic vendors are fully aligned and are committed to addressing support issues when running Oracle software in VMware environments.

This announcement was made by Larry Ellison, CEO Oracle Corporation at the keynote on Monday September 16th, 2019.

 

 

Video snippet of Larry Ellison announcing the VMware Oracle Partnership Announcement at OOW Sept 16th, 2019 can be found here .

Full video of Larry Ellison talk about VMware at the Day 1 Keynotes at Oracle Open World can be found here.

 

John Gilmartin, GM and SVP , ISBU VMware re-iterated the value of the strategic alliance between VMware and Oracle in the keynote on Tuesday September 17th, 2019.

 

   

 

Full video of John Gilmartin on stage with Oracle execs on Day 2 keynote at Oracle Open World can be found here.

 

 

 

 

Customer Support for Oracle on VMware

Additionally, customers will have access to Oracle technical support for Oracle products running on VMware environments. When the Oracle support statement is updated, it will enable customers to achieve quicker time to resolution of issues.

Oracle will support customers with active support contracts running supported versions of Oracle products in Oracle supported computing environments.

https://www.oracle.com/corporate/pressrelease/oow19-oracle-and-vmware-091619.html

 

 

From Reuters :

Oracle Corp and VMware Inc on Monday announced a deal designed to resolve years of tension over how Oracle handles technical support for VMware users and make it easier for them to move to Oracle’s cloud computing service.  The two companies said that Oracle would offer technical support to customers who run its applications on top of VMware.

Oracle and VMware have clashed in the past over the issue. Oracle said on Monday it would now provide support for those situations.

 

 

“Customers don’t want to deploy two products unless it’s supported by both vendors. This was a stumbling block for the past two decades,” said Sanjay Poonen, chief operating officer for customer operations at VMware, said in an interview. “Our relationship with Oracle is significantly better than it was 20 years ago. It’s a new day.”

 

 

In addition to the above , VMware & Oracle had 3 booth theater sessions , 2 sessions at the The Exchange Lobby (Moscone South) – Cloud Infrastructure Theater , September 18th ,  10.15 am – 10.35 am and 1.15 am – 1.35 pm to reiterate the same message.

 

 

Video recording of the VMware & Oracle Team booth theater presentation at the Dell Booth on Sept 18, 2019 can be found here .

Video snippet of VMware & Oracle Team answering customer questions at the Dell Booth on Sept 18, 2019 can be found here .

 

 

 

Please go to the VMware and Applications Blog site and specifically to the “Oracle on VMware One-Stop-Shop” blog for an FAQ on the enhanced support agreement and the strategic alliance between Oracle and VMware.

 

 

https://blogs.vmware.com/apps/oracle

https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html

https://www.vmware.com/solutions/business-critical-apps/oracle-virtualization.html

 

https://www.oracle.com/cloud/VMware/

https://blogs.oracle.com/cloud-infrastructure/announcing-a-certified-vmware-solution-on-oracle-cloud

Oracle and VMware Partner to Support Customers’ Hybrid Cloud Strategies – Press Release

 

Posted in Uncategorized

Drive Oracle Performance and Scalability with vSphere, vSAN, and VMware Cloud on AWS

Oracle on VMware

 

 

 

 

 

 

Running business-critical Oracle applications on VMware on premises or in the cloud helps organizations maximize resources without compromising application performance and scale dynamically to meet service level commitments. Virtualization also helps IT teams improve manageability, increase agility, and shed costs—and all while serving as a foundation for cloud computing. With a streamlined VMware foundation, IT and application administrators can better leverage storage, network and computing resources to control costs, deliver higher quality of service, accelerate time to market and respond faster to changing business needs.  VMware customers also have the assurance of full support from Oracle when running these applications in any VMware virtualization environment, as detailed in Oracle Cloud VMware Solution – A New beginning of a Strategic relationship.

 

Oracle on vSphere

Databases are typically among the most provisioned applications in the data center. With vSphere virtualization, Oracle database infrastructure can be significantly consolidated and its performance significantly increased.  VMware vSphere 6.7 introduced key technologies that make it the best performing platform for Oracle applications. New features such as support for Persistent Memory (PMEM) and enhanced support for Remote Directory Memory Access (RDMA) significantly accelerate the performance of enterprise applications. Read more about these capabilities in Introducing vSphere 6.7 for Enterprise Applications.

 

 

Hyperconverged Infrastructure and vSAN

In vSAN 6.7, VMware offers an all-flash hyperconverged infrastructure for scalable, resilient, high-performance storage that meets the demands of online transaction processing (OLTP) environments, while simplifying administration with policy-based management. Read Oracle Database on All-Flash vSAN 6.7 to learn about a reference architecture that helps customers design and implement optimal configurations specifically for Oracle Database on all-flash vSAN 6.7.

 

 

VMware Cloud on AWS

With VMware Cloud on AWS, VMware provides an on-demand service that integrates vSphere, vSAN, and NSX along with VMware vCenter management allowing customers to run applications across vSphere-based cloud environments with access to a broad range of AWS services. It is an ideal solution for cloud use cases such as disaster recovery, cloud migration, and data center extension. Read more in Oracle Workloads and VMware Cloud on AWS: Deployment, Migration, and Configuration.

 

To learn more about how VMware Cloud on AWS works, try the VMware Cloud on AWS Hands-on Lab.  And to see the latest and greatest vSphere features in action, try the vSphere Hands on Lab.

 

 

 

Posted in Uncategorized

Using Jumbo Frames on VMware NSX for Oracle Workloads

 

 

This blog is not a deep dive on VMware NSX or VXLAN concepts , this blog will focus on Oracle Real Application Cluster (RAC) Interconnect MTU sizing using Jumbo Frames on VMware NSX.

 

VMware NSX

 

VMware NSX Data Center is the network virtualization and security platform that enables the virtual cloud network, a software-defined approach to networking that extends across data centers, clouds, and application frameworks.

With NSX Data Center, networking and security are brought closer to the application wherever it’s running, from virtual machines (VMs) to containers to bare metal. Like the operational model of VMs, networks can be provisioned and managed independent of underlying hardware.

More details on VMware NSX can be found here.

 

 

Virtual eXtensible Local Area Network (VXLAN)

 

A blog by Vyenkatesh Deshpande, describe the different components of the VMware’s VXLAN implementation.

Important concepts about Unicast, Broadcast and Multicast can be found here.

VXLAN is an overlay network technology. Overlay network can be defined as any logical network that is created on top of the existing physical networks. VXLAN creates Layer 2 logical networks on top of the IP network.

Using VXLAN adds another 50 bytes of additional overhead for the Protocol.

On top the ICMP/ping implementation doesn’t encapsulate the 28-byte ICMP (8) + IP (20) header, so we must account for 28 Bytes.

 

 

VMware Cloud on AWS

 

VMware Cloud on AWS is an on-demand service that enables customers to run applications across vSphere-based cloud environments with access to a broad range of AWS services.

Powered by VMware Cloud Foundation, this service integrates vSphere, vSAN and NSX along with VMware vCenter management, and is optimized to run on dedicated, elastic, bare-metal AWS infrastructure. ESXi hosts in VMware Cloud on AWS reside in an AWS availability Zone (AZ) and are protected by vSphere HA.

The use case for deploying VMware Cloud on AWS are multi-fold namely

  • Data Center Extension & DR
  • Cloud Migration
  • Application modernization & Next-Generation Apps build out

More detail on VMware Cloud on AWS can be found here.

 

 

 

Oracle Net Services

 

Oracle Net, a component of Oracle Net Services, enables a network session from a client application to an Oracle Database server. When a network session is established, Oracle Net acts as the data courier for both the client application and the database.

Oracle Net communicates with TCP/IP to enable computer-level connectivity and data transfer between the client and the database.

More details on Oracle Net Services can be found here.

  

 

 

Oracle Real Application Cluster (RAC)

 

Non-cluster Oracle databases have a one-to-one relationship between the Oracle database and the instance. Oracle RAC environments, however, have a one-to-many relationship between the database and instances. An Oracle RAC database can have several instances, all which access one database. All database instances must use the same interconnect, which can also be used by Oracle Clusterware.

Oracle Clusterware is a portable cluster management solution that is integrated with Oracle Database. Oracle Clusterware is a required component for using Oracle RAC that provides the infrastructure necessary to run Oracle RAC.

More details on Oracle RAC can be found here.

 

 

Oracle Real Application Cluster (RAC) Interconnect

 

All nodes in an Oracle RAC environment must connect to at least one Local Area Network (LAN) (commonly referred to as the public network) to enable users and applications to access the database.

In addition to the public network, Oracle RAC requires private network connectivity used exclusively for communication between the nodes and database instances running on those nodes. This network is commonly referred to as the interconnect. The interconnect network is a private network that connects all the servers in the cluster.

More details on Oracle Real Application Cluster (RAC) Interconnect can be found here.

 

 

Oracle Real Application Cluster (RAC) Networking requirements

 

Oracle documentation for RAC –There are 2 main requirements , among others , with respect to Broadcast and Multicast traffic

  • Broadcast Requirements
    • Broadcast communications (ARP and UDP) must work properly across all the public and private interfaces configured for use by Oracle Grid Infrastructure.
    • The broadcast must work across any configured VLANs as used by the public or private interfaces
    • When configuring public and private network interfaces for Oracle RAC, you must enable Address Resolution Protocol (ARP). Highly Available IP (HAIP) addresses do not require ARP on the public network, but for VIP failover, you need to enable ARP. Do not configure NOARP.

 

  • Multicast Requirements
    • For each cluster member node, the Oracle mDNS daemon uses multicasting on all interfaces to communicate with other nodes in the cluster.
    • Multicasting is required on the private interconnect. For this reason, at a minimum, you must enable multicasting for the cluster
      • Across the broadcast domain as defined for the private interconnect
      • On the IP address subnet ranges 224.0.0.0/24 and optionally 230.0.1.0/24
    • You do not need to enable multicast communications across routers.

More information on Oracle RAC networking requirements can be found here.

 

 

 

Oracle Real Application Cluster using VMware NSX

 

Oracle Workloads , both Single Instance and RAC can seamless and transparently run on top of VMware NSX without any issues.

With Extended Oracle RAC , both Storage and Network virtualization needs to be deployed to provided high availability, workload Mobility, workload balancing and effective Site Maintenance between sites.

NSX supports multi-datacenter deployments to allow L2 adjacency in software, to put it in simple words stretching the network to allow VM too utilize the same subnets in multiple sites.

The blog post here showcases the ability to stretch an Oracle RAC solution in an Extended Oracle RAC deployment between multi-datacenter and using VMware NSX for L2 Adjacency.

This topic and the related demo also featured in VMworld 2016.

VIRT7575 – Architecting NSX with Business Critical Applications for Security, Automation and Business Continuity

 

 

 

Oracle Real Application Cluster using Jumbo Frames on VMware NSX

 

The standard Maximum Transmission Unit (MTU) for IP frames is 1500 Bytes. Jumbo Frames are MTU’s larger than 1500 Bytes , we usually refer to a frame with an MTU of 9000 Bytes.

Jumbo Frames can be implemented for private Cluster Interconnects and requires very careful configuration and testing to realize its benefits.

In many cases, failures or inconsistencies can occur due to incorrect setup, bugs in the driver or switch software, which can result in sub-optimal performance and network errors.

In order to make Jumbo Frames work properly for a Cluster Interconnect network, the host’s private network adapter must be configured with a persistent MTU size of ( 9000 bytes – 50 bytes of VXLAN overhead – 28 bytes of ICMP/ping ) = 8922 Bytes.

 

 

Setting MTU to 9000 Bytes to enable Jumbo Frames on VMware SDDC

 

Jumbo frames let ESXi hosts send larger frames out onto the physical network. The network must support jumbo frames end-to-end that includes physical network adapters, physical switches, and storage devices. Before enabling jumbo frames, check with your hardware vendor to ensure that your physical network adapter supports jumbo frames.

You can enable jumbo frames on a vSphere distributed switch or vSphere standard switch by changing the maximum transmission unit (MTU) to a value greater than 1500 bytes. 9000 bytes is the maximum frame size that you can configure.

More details on Jumbo Frames on VMware vSphere can be found here.

Refer to the blog ‘What’s the Big Deal with Jumbo Frames?’ about Jumbo Frames and VMware SDDC.

For on-premises setup, using VMware Web Client , Edit Settings on the distributed switch to set the MTU size.

 

 

Distributed switch with MTU set to 9000 bytes

 

 

 

On VMware Cloud on AWS, since it’s a managed service , customer will not have to set MTU to 9000 bytes as it is set when the SDDC cluster is provisioned in the first place.

 

 

 

 In our lab setup, the standard i3 ESXi servers had a 25GB Elastic Network Adapter (PF) attached .

 

 

 

  

 

Oracle Metalink on changing RAC private network MTU size

 

Refer to Oracle Metalink document ‘Recommendation for the Real Application Cluster Interconnect and Jumbo Frames (Doc ID 341788.1)’ for more information on Jumbo frames for Oracle RAC interconnect.

Refer to Metalink document ‘How to Modify Private Network Information in Oracle Clusterware (Doc ID 283684.1)’ describes how to change private network MTU only.

For example, private network MTU is changed from 1500 to 8922 [9000 – 50 bytes of VXLAN overhead – 28 bytes of ICMP/ping = 8922 Bytes ] , network interface name and subnet remain the same.

  1. Shutdown Oracle Clusterware stack on all nodes
  2. Make the required network change of MTU size at OS network layer, ensure private network is available with the desired MTU size, in this case 8922 bytes , ping with the desired MTU size works on all cluster nodes
  3. Restart Oracle Clusterware stack on all nodes

 

 

 

Conclusion

 

VMware NSX Data Center is the network virtualization and security platform that enables the virtual cloud network, a software-defined approach to networking that extends across data centers, clouds, and application frameworks. With NSX Data Center, networking and security are brought closer to the application wherever it’s running, from virtual machines (VMs) to containers to bare metal

In addition to the public network, Oracle RAC requires private network connectivity used exclusively for communication between the nodes and database instances running on those nodes. This network is commonly referred to as the interconnect.

In case of Oracle Real Application Cluster using VMware NSX and Jumbo Frames, Jumbo Frames can be implemented for private Cluster Interconnects and requires very careful configuration and testing to realize its benefits.

In order to make Jumbo Frames work properly for a Cluster Interconnect network, the host’s private network adapter must be configured with a persistent MTU size of  ( 9000 – 50 bytes of VXLAN overhead – 28 bytes of ICMP/ping ) = 8922 Bytes

All Oracle on vSphere white papers including Oracle on VMware vSphere / VMware vSAN /  VMware Cloud on AWS , Best practices, Deployment guides, Workload characterization guide can be found at  Oracle on VMware Collateral – One Stop Shop

 

Posted in Uncategorized

Oracle and vSphere Persistent Memory (PMEM) – Oracle Instance Recovery – An Investigation

Introduction to VMware Persistent Memory (PMEM)

 

Persistent Memory (PMEM) resides between DRAM and disk storage in the data storage hierarchy. This  technology enables byte-addressable updates and does not lose data if power is lost.

Instead of having nonvolatile storage at the bottom with the largest capacity but the slowest performance, nonvolatile storage is now very close to DRAM in terms of performance.

PMEM is a byte-addressable form of computer memory that has the following characteristics:

  • DRAM-like latency and bandwidth
  • Regular load/store CPU instructions
  • Paged/mapped by operating system just like DRAM
  • Data is persistent across reboots

More information about Persistent Memory (PMEM)  and how vSphere 6.7 can take advantage of PMEM technology to accelerate IO-intensive Oracle workloads can be found here.

The Accelerating Oracle Performance using vSphere Persistent Memory (PMEM) paper examines the performance of Oracle databases using VMware vSphere 6.7 Persistent Memory feature in different modes for below uses cases for

  • Improved performance of Oracle Redo Log using vPMEM Disk-backed vmdks/vPMEM disks in DAX mode
  • Accelerating Performance using Oracle Smart Flash Cache
  • Potential reduction in Oracle Licensing

In the blog article Oracle and vSphere Persistent Memory (PMEM) – vPMEM v/s vPMEMDisk ,  we demonstrate the performance improvement in Redo log activity when redo log files are placed on vPMEM Disk-backed vmdks/vPMEM disks in DAX mode over redo logs on vPMEMDisk backed vmdks.

 

 

Accelerating Oracle Instance / Crash Recovery using PMEM – An Investigation

 

This blog addresses the instance or crash recovery aspect of Oracle database and investigates if Persistent Memory can help with speeding up the Oracle crash recovery process.

 

 

‘It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.’ –  Sherlock Holmes Quote (A Scandal in Bohemia).

 

 

Tryst with Redis (Re-architecting an Application to Leverage PMEM)

 

To illustrate the benefits of byte-addressable persistent memory, we took a popular open source application Redis and modified it to take full advantage of persistent memory.

Redis, by default, only commits key-value transactions to memory. Persistence across reboot (e.g., due to power loss) is provided by periodically writing transaction logs to storage. By default, transaction log is written to storage every second. Therefore, Redis can lose a second’s worth of transactions in the default configuration.

Initially, we experimented by storing Redis transaction logs in persistent memory. That provided noticeable but marginal (around 12%) improvement in performance. Then, we decided to modify Redis to take full advantage of byte-addressable persistent memory by placing the entire key-value database in persistent memory. We call the modified version PMem-aware Redis. We added very low-overhead journaling to make Redis transactions on persistent memory crash consistent. With these changes, we got around 45% improvement in transaction throughput.

In addition, PMem-aware Redis can recover from crash almost instantaneously because the database is already in persistent memory and need to be read from disk. For comparison, unmodified Redis using NVMe SSD can take minutes to recover after crash, even for modest (around 1 GB) database sizes. Finally, since persistent memory is local to a host, we experimented with replicating Redis transactions to a standby host. Even with replication, PMem-aware Redis performed around 28% better than unmodified Redis using a fast and host-local NVMe SSD without any replication.

 

 

More details on this can be found in the article ‘Persistent Memory Initiative’ and white paper ‘Persistent Memory Performance on vSphere 6.7 Performance Study’ .

 

 

Oracle VM setup

 

As mentioned above, this blog will investigate if Persistent Memory can help with speeding up the crash recovery process of an Oracle database.

The Oracle VM  ‘Oracle122-RHEL-PMEM-udev’ has 6 vCPUS with vRAM 32GB.

 

 

 

The 4 hard disks attached to the VM are as below –

  • Hard Disk 1 – 60G – OS
  • Hard Disk 2 – 100G – Oracle + Grid Infrastructure binaries
  • Hard Disk 3 – 2TB – Oracle Database datafiles
  • Hard Disk 4 – 32GB – Oracle Redo logs

 

Below is the operating system view of the vmdk’s –

[root@oracle122-rhel ~]# fdisk -lu

Disk /dev/sdb: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7783b7a1

Device Boot Start End Blocks Id System
/dev/sdb1 2048 209715199 104856576 83 Linux

Disk /dev/sdc: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7d28e830

Device Boot Start End Blocks Id System
/dev/sdc1 2048 4294967294 2147482623+ 83 Linux

Disk /dev/sdd: 34.4 GB, 34359738368 bytes, 67108864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x952fca2a

Device Boot Start End Blocks Id System
/dev/sdd1 2048 67108863 33553408 83 Linux

Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0004908d

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 125829119 61864960 8e Linux LVM

….
[root@oracle122-rhel ~]#

 

 

 

Components and Granules in the SGA

 

All SGA components allocate and deallocate space in units of granules. Oracle Database tracks SGA memory use in internal numbers of granules for each SGA component.

The granule size is determined by the amount of SGA memory requested when the instance starts. Specifically, the granule size is based on the value of the SGA_MAX_SIZE initialization parameter.

The table below shows the SGA granule size.

 

https://docs.oracle.com/database/121/ADMIN/memory.htm#ADMIN11203

 

 

 

Oracle Memory Management

 

The memory structures that must be managed are the system global area (SGA) and the instance program global area (instance PGA).

The 2 main memory management methods are :

  • Automatic Memory Management (AMM)
    • manage the SGA memory and instance PGA memory completely automatically. You designate only the total memory size to be used by the instance, and Oracle Database dynamically exchanges memory between the SGA and the instance PGA as needed to meet processing demands. This capability is referred to as automatic memory management. With this memory management method, the database also dynamically tunes the sizes of the individual SGA components and the sizes of the individual PGAs

 

  • Manual Memory Management – These methods are
    • Automatic shared memory management – for the SGA
    • Manual shared memory management – for the SGA
    • Automatic PGA memory management – for the instance PGA
    • Manual PGA memory management – for the instance PGA

 

To use the MEMORY_TARGET or MEMORY_MAX_TARGET feature, the  /dev/shm mount point should be equal in size or larger than the value of MEMORY_TARGET or MEMORY_MAX_TARGET, whichever is larger.

For Automatic Memory Management (AMM) , the initialization parameters memory_target and memory_max_size are  set to 16G.

Contents of the /etc/fstab which shows the /dev/shm mount point –

oracle@oracle122-rhel:DBPROD:/home/oracle> cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Jan 26 19:27:50 2018
#
# Accessible filesystems, by reference, are maintained under ‘/dev/disk’
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/rhel-root / xfs defaults 0 0
UUID=d200eecc-1db9-4c9f-8ae4-8b95f20d70c9 /boot xfs defaults 0 0
/dev/mapper/rhel-home /home xfs defaults 0 0
/dev/mapper/rhel-swap swap swap defaults 0 0
/dev/vg2_oracle/LogVol_u01 /u01 ext4 defaults 1 2
#/dev/pmem1 /redolog ext4 dax,defaults 1 2
#/dev/pmem2 /redolog_dax ext4 dax,defaults 1 2

shmfs /dev/shm tmpfs size=24g 0
oracle@oracle122-rhel:DBPROD:/home/oracle>

 

 

 

Test Scenario and Steps

 

We have 2 Test scenarios –

  • Oracle Crash Recovery using AMM Memory Management without Persistent Memory
  • Oracle Crash Recovery using AMM Memory Management with Persistent Memory

 

The Test steps are to  –

  • Run a SLOB workload , which is an Oracle I/O workload generation tool kit, against the database for 5 minutes
  • Within 3 minutes into the load, shutdown the database via a ‘shutdown abort’ command, mimicking an abnormal database failure or a database crash
  • Restarting the database will result in a crash recovery of the database thread and the recovery time would be recorded for each of the above test scenarios
  • Compare the recovery time for each of the above scenario

 

 

Oracle Crash Recovery using AMM Memory Management without Persistent Memory

 

Output of /proc/meminfo to show the VM memory total :

oracle@oracle122-rhel:DBPROD:/home/oracle> cat /proc/meminfo
MemTotal: 32781052 kB
MemFree: 19212424 kB
MemAvailable: 20514972 kB
Buffers: 108464 kB
Cached: 12110000 kB
SwapCached: 0 kB
Active: 11939808 kB
Inactive: 1126292 kB
Active(anon): 11539540 kB
Inactive(anon): 65204 kB
Active(file): 400268 kB
Inactive(file): 1061088 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 6291452 kB
SwapFree: 6291452 kB
Dirty: 100 kB
Writeback: 0 kB
AnonPages: 847652 kB
Mapped: 1327056 kB
Shmem: 10757112 kB
Slab: 123280 kB
SReclaimable: 65332 kB
SUnreclaim: 57948 kB
KernelStack: 8000 kB
PageTables: 94676 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 22681976 kB
Committed_AS: 12997260 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 33760788 kB
VmallocChunk: 34325131260 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 173952 kB
DirectMap2M: 5068800 kB
DirectMap1G: 30408704 kB
oracle@oracle122-rhel:DBPROD:/home/oracle>

 

output of df command to show /dev/shm filesystem size

oracle@oracle122-rhel:DBPROD:/home/oracle> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 36G 2.3G 34G 7% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 24G 11G 14G 43% /dev/shm
tmpfs 16G 9.0M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 1014M 223M 792M 22% /boot
/dev/mapper/rhel-home 18G 4.0G 14G 23% /home
/dev/mapper/vg2_oracle-LogVol_u01 99G 31G 64G 33% /u01
tmpfs 3.2G 0 3.2G 0% /run/user/501
tmpfs 3.2G 0 3.2G 0% /run/user/0
oracle@oracle122-rhel:DBPROD:/home/oracle>

 

List the contents of the /dev/shm file system. As we can see from the above Granules table, the SGA & PGA for database was set to 16GB and hence the SGA granule size is 32MB.

The ASM instance memory footprint is close to 32MB and hence the granule size is MB for the ASM instance.

oracle@oracle122-rhel:DBPROD:/home/oracle> ls -l /dev/shm
total 10747904
-rw——- 1 grid oinstall 4194304 Nov 23 08:39 ora_+ASM_32769_0
-rw——- 1 grid oinstall 4194304 Nov 22 18:52 ora_+ASM_32769_1
-rw——- 1 grid oinstall 4194304 Nov 22 18:52 ora_+ASM_32769_2
…..
-rw——- 1 grid oinstall 4194304 Nov 23 08:39 ora_+ASM_65536_114
-rw——- 1 grid oinstall 4194304 Nov 22 18:52 ora_+ASM_65536_115
….
-rw——- 1 grid oinstall 0 Nov 22 18:52 ora_+ASM_65536_157
-rw——- 1 grid oinstall 0 Nov 22 18:52 ora_+ASM_65536_158
-rw——- 1 grid oinstall 0 Nov 22 18:52 ora_+ASM_65536_159
-rw——- 1 grid oinstall 4194304 Nov 22 18:52 ora_+ASM_65536_16
-rw——- 1 grid oinstall 0 Nov 22 18:52 ora_+ASM_65536_160
-rw——- 1 grid oinstall 0 Nov 22 18:52 ora_+ASM_65536_161
-rw——- 1 grid oinstall 0 Nov 22 18:52 ora_+ASM_65536_162

-rw——- 1 oracle asmadmin 33554432 Nov 23 08:33 ora_DBPROD_294916_50
-rw——- 1 oracle asmadmin 0 Nov 23 08:32 ora_DBPROD_294916_500
-rw——- 1 oracle asmadmin 0 Nov 23 08:32 ora_DBPROD_294916_501
-rw——- 1 oracle asmadmin 0 Nov 23 08:32 ora_DBPROD_294916_502
-rw——- 1 oracle asmadmin 0 Nov 23 08:32 ora_DBPROD_294916_503
…..
-rw——- 1 oracle asmadmin 33554432 Nov 23 08:33 ora_DBPROD_294916_56
….
-rw——- 1 oracle asmadmin 33554432 Nov 23 08:33 ora_DBPROD_294916_59
oracle@oracle122-rhel:DBPROD:/home/oracle>

 

Start the SLOB workload against the Oracle database using command ‘/u01/software/SLOB/SLOB/runit.sh -s 1 -t 100

 

oracle@oracle122-rhel:DBPROD:/home/oracle> /u01/software/SLOB/SLOB/runit.sh -s 1 -t 100
Before AWR
SQL*Plus: Release 12.2.0.1.0 Production on Fri Nov 23 08:49:06 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
SQL> Connected.
SQL>
PL/SQL procedure successfully completed.
SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 – 64bit Production
SLOB started with 1 users
/u01/software/SLOB/SLOB/runit.sh: line 2: !/bin/bash: No such file or directory
NOTIFY : 2018.11.23-08:49:08 : For security purposes all file and directory creation and deletions
NOTIFY : 2018.11.23-08:49:08 : performed by /u01/software/SLOB/SLOB/runit.sh are logged in: /u01/software/SLOB/SLOB/.file_operations_audit_trail.out.
NOTIFY : 2018.11.23-08:49:08 : SLOB TEMPDIR is /tmp/.SLOB.2018.11.23.084908. SLOB will delete this directory at the end of this execution.
NOTIFY : 2018.11.23-08:49:08 : Sourcing in slob.conf
NOTIFY : 2018.11.23-08:49:08 : Performing initial slob.conf sanity check…
NOTIFY : 2018.11.23-08:49:08 :
NOTIFY : 2018.11.23-08:49:08 : All SLOB sessions will connect to DBPROD_PMEM_RHEL_PDB1 via SQL*Net.
NOTIFY : 2018.11.23-08:49:08 : Connecting to the instance to validate slob.conf->SCALE setting.

UPDATE_PCT: 100
SCAN_PCT: 0
RUN_TIME: 300
WORK_LOOP: 0
SCALE: 28G (3670016 blocks)
WORK_UNIT: 64
REDO_STRESS: HEAVY
HOT_SCHEMA_FREQUENCY: 0
HOTSPOT_MB: 8
HOTSPOT_OFFSET_MB: 16
HOTSPOT_FREQUENCY: 3
THINK_TM_FREQUENCY: 0
THINK_TM_MIN: .1
THINK_TM_MAX: .5
DATABASE_STATISTICS_TYPE: awr
SYSDBA_PASSWD: “vmware123”
DBA_PRIV_USER: “sys”
ADMIN_SQLNET_SERVICE: “DBPROD_PMEM_RHEL_PDB1”
SQLNET_SERVICE_BASE: “DBPROD_PMEM_RHEL_PDB1”
SQLNET_SERVICE_MAX: “”

EXTERNAL_SCRIPT: “”
THREADS_PER_SCHEMA: 100 (-t option)

Note: runit.sh will use the following connect strings as per slob.conf settings:
Admin Connect String: “sys/vmware123@DBPROD_PMEM_RHEL_PDB1 as sysdba”

NOTIFY : 2018.11.23-08:49:09 : Clearing temporary SLOB output files from previous SLOB testing.
NOTIFY : 2018.11.23-08:49:09 : Testing admin connectivity to the instance to validate slob.conf settings.
NOTIFY : 2018.11.23-08:49:09 : Testing connectivity. Command: “sqlplus -L sys/vmware123@DBPROD_PMEM_RHEL_PDB1 as sysdba”.
NOTIFY : 2018.11.23-08:49:09 : Next, testing 1 user (non-admin) connections…
NOTIFY : 2018.11.23-08:49:09 : Testing connectivity. Command: “sqlplus -L user1/user1@DBPROD_PMEM_RHEL_PDB1”.
NOTIFY : 2018.11.23-08:49:09 : Performing redo log switch.
NOTIFY : 2018.11.23-08:49:09 : Redo log switch complete. Setting up trigger mechanism.
NOTIFY : 2018.11.23-08:49:19 : Running iostat, vmstat and mpstat on current host–in background.
NOTIFY : 2018.11.23-08:49:19 : Connecting 100 (THREADS_PER_SCHEMA) session(s) to 1 schema(s) …
NOTIFY : 2018.11.23-08:49:22 :
NOTIFY : 2018.11.23-08:49:22 : Executing awr “before snap” procedure. Command: “sqlplus -S -L sys/vmware123@DBPROD_PMEM_RHEL_PDB1 as sysdba”.
NOTIFY : 2018.11.23-08:49:23 : Before awr snap ID is 1218
NOTIFY : 2018.11.23-08:49:23 :
NOTIFY : 2018.11.23-08:49:23 : Test has been triggered. Processes are executing.
NOTIFY : 2018.11.23-08:49:23 : List of monitored sqlplus PIDs written to /tmp/.SLOB.2018.11.23.084908/8583.f_wait_pids.out.
NOTIFY : 2018.11.23-08:49:34 : Waiting for 287 seconds before monitoring running processes (for exit).
^C
oracle@oracle122-rhel:DBPROD:/home/oracle>

 

Run ‘shutdown abort’ against database to simulate a crash at 3 minutes into the load.

 

oracle@oracle122-rhel:DBPROD:/home/oracle> date
Fri Nov 23 08:52:21 PST 2018

oracle@oracle122-rhel:DBPROD:/home/oracle> ./stop_db_abort
SQL*Plus: Release 12.2.0.1.0 Production on Fri Nov 23 08:52:27 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
SQL> Connected.
SQL> ORACLE instance shut down.
SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 – 64bit Production
oracle@oracle122-rhel:DBPROD:/home/oracle>

 

Restart the Oracle database which prompts Oracle to perform crash recovery of the thread.

 

Observe the database alert log file to see the crash recovery progress.

 

oracle@oracle122-rhel:DBPROD:/u01/admin/DBPROD/diag/rdbms/dbprod/DBPROD/trace> tail -f alert_DBPROD.log
2018-11-23T08:52:27.264745-08:00
Shutting down instance (abort) (OS id: 994)
License high water mark = 105
2018-11-23T08:52:27.265146-08:00
USER (ospid: 994): terminating the instance
2018-11-23T08:52:28.276572-08:00
Instance terminated by USER, pid = 994
2018-11-23T08:52:29.094544-08:00
Instance shutdown complete (OS id: 994)

2018-11-23T08:56:01.276626-08:00
Completed redo scan
read 3122163 KB redo, 683839 data blocks need recovery
2018-11-23T08:56:06.017712-08:00

2018-11-23T08:56:08.955533-08:00
Started redo application at
Thread 1: logseq 10505, block 48777, offset 0
2018-11-23T08:56:08.960895-08:00
Recovery of Online Redo Log: Thread 1 Group 4 Seq 10505 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group04_redo01.log
Mem# 1: +REDO_DG/DBPROD/group04_redo02.log
2018-11-23T08:56:09.522910-08:00
Recovery of Online Redo Log: Thread 1 Group 5 Seq 10506 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group05_redo01.log
Mem# 1: +REDO_DG/DBPROD/group05_redo02.log
2018-11-23T08:56:10.128741-08:00
Recovery of Online Redo Log: Thread 1 Group 6 Seq 10507 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group06_redo01.log
Mem# 1: +REDO_DG/DBPROD/group06_redo02.log
2018-11-23T08:56:10.736250-08:00
Recovery of Online Redo Log: Thread 1 Group 7 Seq 10508 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group07_redo01.log
Mem# 1: +REDO_DG/DBPROD/group07_redo02.log
2018-11-23T08:56:11.354438-08:00
Recovery of Online Redo Log: Thread 1 Group 8 Seq 10509 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group08_redo01.log
Mem# 1: +REDO_DG/DBPROD/group08_redo02.log
2018-11-23T08:56:11.963102-08:00
Recovery of Online Redo Log: Thread 1 Group 9 Seq 10510 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group09_redo01.log
Mem# 1: +REDO_DG/DBPROD/group09_redo02.log
2018-11-23T08:56:12.575632-08:00
Recovery of Online Redo Log: Thread 1 Group 10 Seq 10511 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group10_redo01.log
Mem# 1: +REDO_DG/DBPROD/group10_redo02.log
2018-11-23T08:56:13.196423-08:00
Recovery of Online Redo Log: Thread 1 Group 11 Seq 10512 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group11_redo01.log
Mem# 1: +REDO_DG/DBPROD/group11_redo02.log
2018-11-23T08:56:13.855567-08:00
Recovery of Online Redo Log: Thread 1 Group 12 Seq 10513 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group12_redo01.log
Mem# 1: +REDO_DG/DBPROD/group12_redo02.log
2018-11-23T08:56:14.533012-08:00
Recovery of Online Redo Log: Thread 1 Group 13 Seq 10514 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group13_redo01.log
Mem# 1: +REDO_DG/DBPROD/group13_redo02.log
2018-11-23T08:56:15.210285-08:00
Recovery of Online Redo Log: Thread 1 Group 14 Seq 10515 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group14_redo01.log
Mem# 1: +REDO_DG/DBPROD/group14_redo02.log
2018-11-23T08:56:15.899655-08:00
Recovery of Online Redo Log: Thread 1 Group 15 Seq 10516 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group15_redo01.log
Mem# 1: +REDO_DG/DBPROD/group15_redo02.log
2018-11-23T08:56:16.605822-08:00
Recovery of Online Redo Log: Thread 1 Group 16 Seq 10517 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group16_redo01.log
Mem# 1: +REDO_DG/DBPROD/group16_redo02.log
2018-11-23T08:56:17.319616-08:00
Completed redo application of 377.92MB
2018-11-23T08:56:25.061718-08:00
Completed crash recovery at
Thread 1: RBA 10517.323909.16, nab 323909, scn 0x00000000087264b7
683839 data blocks read, 679844 data blocks written, 3122163 redo k-bytes read
Endian type of dictionary set to little

 

Start of recovery process – 2018-11-23T08:56:08.955533-08:00
Stop of recovery process – 2018-11-23T08:56:25.061718-08:00
Time taken for crash recovery ~ 17 seconds

 

 

 

Oracle Crash Recovery using AMM Memory Management with Persistent Memory

 

Stop Oracle database and ASM instance . Add a NVDIMM device to the VM of size 32GB as shown below.

 

 

Running linux ‘fdisk’ command shows the nvdimm device as below.

 

[root@oracle122-rhel ~]# fdisk -lu
….

Disk /dev/pmem0: 34.4 GB, 34359738368 bytes, 67108864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


[root@oracle122-rhel ~]#

 

Mount the ‘/dev/shm’ file system on the PMEM device as shown below

 

[root@oracle122-rhel ~]# umount /dev/shm ; mount -t tmpfs -o size=24g /dev/pmem0 /dev/shm ; df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 36G 2.3G 34G 7% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 9.1M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 1014M 223M 792M 22% /boot
/dev/mapper/rhel-home 18G 4.0G 14G 23% /home
/dev/mapper/vg2_oracle-LogVol_u01 99G 30G 65G 32% /u01
tmpfs 3.2G 0 3.2G 0% /run/user/501
tmpfs 3.2G 0 3.2G 0% /run/user/0
/dev/pmem0 24G 0 24G 0% /dev/shm
[root@oracle122-rhel ~]#

 

Restart the Oracle ASM and Database instance.

 

Rerun the same SLOB workload against the oracle database.

 

oracle@oracle122-rhel:DBPROD:/home/oracle> /u01/software/SLOB/SLOB/runit.sh -s 1 -t 100
Before AWR
SQL*Plus: Release 12.2.0.1.0 Production on Fri Nov 23 09:16:56 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
SQL> Connected.
SQL>
PL/SQL procedure successfully completed.

SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 – 64bit Production

SLOB started with 1 users

/u01/software/SLOB/SLOB/runit.sh: line 2: !/bin/bash: No such file or directory
NOTIFY : 2018.11.23-09:16:58 : For security purposes all file and directory creation and deletions
NOTIFY : 2018.11.23-09:16:58 : performed by /u01/software/SLOB/SLOB/runit.sh are logged in: /u01/software/SLOB/SLOB/.file_operations_audit_trail.out.
NOTIFY : 2018.11.23-09:16:58 : SLOB TEMPDIR is /tmp/.SLOB.2018.11.23.091658. SLOB will delete this directory at the end of this execution.
NOTIFY : 2018.11.23-09:16:58 : Sourcing in slob.conf
NOTIFY : 2018.11.23-09:16:58 : Performing initial slob.conf sanity check…
NOTIFY : 2018.11.23-09:16:58 :
NOTIFY : 2018.11.23-09:16:58 : All SLOB sessions will connect to DBPROD_PMEM_RHEL_PDB1 via SQL*Net.
NOTIFY : 2018.11.23-09:16:58 : Connecting to the instance to validate slob.conf->SCALE setting.

UPDATE_PCT: 100
SCAN_PCT: 0
RUN_TIME: 300
WORK_LOOP: 0
SCALE: 28G (3670016 blocks)
WORK_UNIT: 64
REDO_STRESS: HEAVY
HOT_SCHEMA_FREQUENCY: 0
HOTSPOT_MB: 8
HOTSPOT_OFFSET_MB: 16
HOTSPOT_FREQUENCY: 3
THINK_TM_FREQUENCY: 0
THINK_TM_MIN: .1
THINK_TM_MAX: .5
DATABASE_STATISTICS_TYPE: awr
SYSDBA_PASSWD: “vmware123”
DBA_PRIV_USER: “sys”
ADMIN_SQLNET_SERVICE: “DBPROD_PMEM_RHEL_PDB1”
SQLNET_SERVICE_BASE: “DBPROD_PMEM_RHEL_PDB1”
SQLNET_SERVICE_MAX: “”

EXTERNAL_SCRIPT: “”
THREADS_PER_SCHEMA: 100 (-t option)

Note: runit.sh will use the following connect strings as per slob.conf settings:
Admin Connect String: “sys/vmware123@DBPROD_PMEM_RHEL_PDB1 as sysdba”

NOTIFY : 2018.11.23-09:16:59 :
NOTIFY : 2018.11.23-09:16:59 :
WARNING : 2018.11.23-09:16:59 : *****************************************************************************
WARNING : 2018.11.23-09:16:59 : SLOB has found possible zombie processes from a prior SLOB test.
WARNING : 2018.11.23-09:16:59 : It is possible that a prior SLOB test aborted.
WARNING : 2018.11.23-09:16:59 : Please investigate the following processes:
WARNING : 2018.11.23-09:16:59 : *****************************************************************************
UID PID PPID C STIME TTY STAT TIME CMD
oracle 30594 1 0 08:49 pts/0 S 0:00 iostat -xm 3
oracle 30595 1 0 08:49 pts/0 S 0:00 vmstat 3
WARNING : 2018.11.23-09:16:59 : *****************************************************************************
NOTIFY : 2018.11.23-09:16:59 : Checking for unlinked output files for processes: 30594 30595
NOTIFY : 2018.11.23-09:16:59 : Unlinked files for process pid 30594 (ls -l /proc/30594/fd):
NOTIFY : 2018.11.23-09:16:59 : Unlinked files for process pid 30595 (ls -l /proc/30595/fd):
WARNING : 2018.11.23-09:16:59 : *****************************************************************************
NOTIFY : 2018.11.23-09:16:59 : Clearing temporary SLOB output files from previous SLOB testing.
NOTIFY : 2018.11.23-09:16:59 : Testing admin connectivity to the instance to validate slob.conf settings.
NOTIFY : 2018.11.23-09:16:59 : Testing connectivity. Command: “sqlplus -L sys/vmware123@DBPROD_PMEM_RHEL_PDB1 as sysdba”.
NOTIFY : 2018.11.23-09:16:59 : Next, testing 1 user (non-admin) connections…
NOTIFY : 2018.11.23-09:16:59 : Testing connectivity. Command: “sqlplus -L user1/user1@DBPROD_PMEM_RHEL_PDB1”.
NOTIFY : 2018.11.23-09:16:59 : Performing redo log switch.
NOTIFY : 2018.11.23-09:16:59 : Redo log switch complete. Setting up trigger mechanism.
NOTIFY : 2018.11.23-09:17:09 : Running iostat, vmstat and mpstat on current host–in background.
NOTIFY : 2018.11.23-09:17:09 : Connecting 100 (THREADS_PER_SCHEMA) session(s) to 1 schema(s) …
NOTIFY : 2018.11.23-09:17:12 :
NOTIFY : 2018.11.23-09:17:12 : Executing awr “before snap” procedure. Command: “sqlplus -S -L sys/vmware123@DBPROD_PMEM_RHEL_PDB1 as sysdba”.
NOTIFY : 2018.11.23-09:17:13 : Before awr snap ID is 1219
NOTIFY : 2018.11.23-09:17:13 :
NOTIFY : 2018.11.23-09:17:13 : Test has been triggered. Processes are executing.
NOTIFY : 2018.11.23-09:17:13 : List of monitored sqlplus PIDs written to /tmp/.SLOB.2018.11.23.091658/17594.f_wait_pids.out.
NOTIFY : 2018.11.23-09:17:23 : Waiting for 287 seconds before monitoring running processes (for exit).
^C
oracle@oracle122-rhel:DBPROD:/home/oracle>

 

Run ‘shutdown abort’ against database to simulate a crash at 3 minutes into the load.

 

oracle@oracle122-rhel:DBPROD:/home/oracle> date
Fri Nov 23 09:20:23 PST 2018

oracle@oracle122-rhel:DBPROD:/home/oracle> ./stop_db_abort
SQL*Plus: Release 12.2.0.1.0 Production on Fri Nov 23 09:20:28 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
SQL> Connected.
SQL> ORACLE instance shut down.
SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 – 64bit Production
oracle@oracle122-rhel:DBPROD:/home/oracle>

 

Restart the Oracle database which prompts Oracle to perform crash recovery of the thread.

 

Observe the database alert log file to see the crash recovery progress.

 

oracle@oracle122-rhel:DBPROD:/u01/admin/DBPROD/diag/rdbms/dbprod/DBPROD/trace> tail -f alert_DBPROD.log
….
2018-11-23T09:20:28.524404-08:00
Shutting down instance (abort) (OS id: 18455)
License high water mark = 105
2018-11-23T09:20:28.524825-08:00
USER (ospid: 18455): terminating the instance
2018-11-23T09:20:29.724996-08:00
Instance terminated by USER, pid = 18455
2018-11-23T09:20:30.532675-08:00
Instance shutdown complete (OS id: 18455)
…..
2018-11-23T09:22:47.704602-08:00
Completed redo scan
read 3233975 KB redo, 725845 data blocks need recovery
2018-11-23T09:22:51.953683-08:00

2018-11-23T09:22:56.044295-08:00
Started redo application at
Thread 1: logseq 10518, block 197286, offset 0
2018-11-23T09:22:56.050171-08:00
Recovery of Online Redo Log: Thread 1 Group 1 Seq 10518 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group01_redo01.log
Mem# 1: +REDO_DG/DBPROD/group01_redo02.log
2018-11-23T09:22:56.430511-08:00
Recovery of Online Redo Log: Thread 1 Group 2 Seq 10519 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group02_redo01.log
Mem# 1: +REDO_DG/DBPROD/group02_redo02.log
2018-11-23T09:22:57.038002-08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 10520 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group03_redo01.log
Mem# 1: +REDO_DG/DBPROD/group03_redo02.log
2018-11-23T09:22:57.661509-08:00
Recovery of Online Redo Log: Thread 1 Group 4 Seq 10521 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group04_redo01.log
Mem# 1: +REDO_DG/DBPROD/group04_redo02.log
2018-11-23T09:22:58.285863-08:00
Recovery of Online Redo Log: Thread 1 Group 5 Seq 10522 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group05_redo01.log
Mem# 1: +REDO_DG/DBPROD/group05_redo02.log
2018-11-23T09:22:58.899482-08:00
Recovery of Online Redo Log: Thread 1 Group 6 Seq 10523 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group06_redo01.log
Mem# 1: +REDO_DG/DBPROD/group06_redo02.log
2018-11-23T09:22:59.525095-08:00
Recovery of Online Redo Log: Thread 1 Group 7 Seq 10524 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group07_redo01.log
Mem# 1: +REDO_DG/DBPROD/group07_redo02.log
2018-11-23T09:23:00.153196-08:00
Recovery of Online Redo Log: Thread 1 Group 8 Seq 10525 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group08_redo01.log
Mem# 1: +REDO_DG/DBPROD/group08_redo02.log
2018-11-23T09:23:00.802933-08:00
Recovery of Online Redo Log: Thread 1 Group 9 Seq 10526 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group09_redo01.log
Mem# 1: +REDO_DG/DBPROD/group09_redo02.log
2018-11-23T09:23:01.497673-08:00
Recovery of Online Redo Log: Thread 1 Group 10 Seq 10527 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group10_redo01.log
Mem# 1: +REDO_DG/DBPROD/group10_redo02.log
2018-11-23T09:23:02.152860-08:00
Recovery of Online Redo Log: Thread 1 Group 11 Seq 10528 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group11_redo01.log
Mem# 1: +REDO_DG/DBPROD/group11_redo02.log
2018-11-23T09:23:02.861497-08:00
Recovery of Online Redo Log: Thread 1 Group 12 Seq 10529 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group12_redo01.log
Mem# 1: +REDO_DG/DBPROD/group12_redo02.log
2018-11-23T09:23:03.583704-08:00
Recovery of Online Redo Log: Thread 1 Group 13 Seq 10530 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group13_redo01.log
Mem# 1: +REDO_DG/DBPROD/group13_redo02.log
2018-11-23T09:23:04.323907-08:00
Recovery of Online Redo Log: Thread 1 Group 14 Seq 10531 Reading mem 0
Mem# 0: +REDO_DG/DBPROD/group14_redo01.log
Mem# 1: +REDO_DG/DBPROD/group14_redo02.log
2018-11-23T09:23:05.024650-08:00
Completed redo application of 375.40MB
2018-11-23T09:23:13.419001-08:00
Completed crash recovery at
Thread 1: RBA 10531.255760.16, nab 255760, scn 0x000000000878ad36
725845 data blocks read, 721385 data blocks written, 3233975 redo k-bytes read
Endian type of dictionary set to little
….

 

Start of recovery process – 2018-11-23T09:22:56.044295-08:00
Stop of recovery process – 2018-11-23T09:23:13.419001-08:00
Time taken for crash recovery ~ 17 secs

 

 

Test Results

 

Oracle Crash Recovery using AMM Memory Management without Persistent Memory

Start of recovery process – 2018-11-23T08:56:08.955533-08:00
Stop of recovery process – 2018-11-23T08:56:25.061718-08:00
Time taken for crash recovery ~ 17 seconds

 

Oracle Crash Recovery using AMM Memory Management with Persistent Memory

Start of recovery process – 2018-11-23T09:22:56.044295-08:00
Stop of recovery process – 2018-11-23T09:23:13.419001-08:00
Time taken for crash recovery ~ 17 seconds

 

The above 2 results are the same and we wonder why we are not able to accelerate the Oracle crash recovery process in the same way as we were able to achieve with the Redis IMBD. Keep in mind, with PMem-aware Redis. we added very low-overhead journaling to make Redis transactions on persistent memory crash consistent .

That’s not the case with the Oracle software out of the box. Oracle will still use its proprietary crash recovery mechanism and hence the reason.

Oracle recreates the SGA granules on every instance startup , in /dev/shm , and hence even if the old SGA granules are present in PMEM , Oracle disregards it for crash recovery purposes.

The best way to find out is to trace the Oracle startup process which shows the SGA granules under /dev/shm are created with O_RDWR, O_CREAT & O_SYNC flags.

The man page for Linux open describes the flags in the open() command below.

A call to open() creates a new open file description, an entry in the system-wide table of open files. The open file description records the file offset and the file status flags (see below). A file descriptor is a reference to an open file description; this reference is unaffected if pathname is subsequently removed or modified to refer to a different file. For further details on open file descriptions, see NOTES.

The argument flags must include one of the following access modes: O_RDONLY, O_WRONLY, or O_RDWR. These request opening the file read-only, write-only, or read/write, respectively.

O_CREAT – If pathname does not exist, create it as a regular file. The owner (user ID) of the new file is set to the effective user ID of the process.

O_SYNC –    Write operations on the file will complete according to the requirements of synchronized I/O file integrity completion (by contrast with the synchronized I/O data integrity completion provided by O_DSYNC.)

 

http://man7.org/linux/man-pages/man2/open.2.html

 

The complete output of the strace command for Instance startup is below.

strace –o /tmp/foo –f sh ./start_db

 

Output :

24932 open(“/dev/shm”, O_RDONLY|O_NOCTTY) = 3
24932 fstat(3, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=5480, …}) = 0
24932 close(3) = 0
24932 open(“/etc/mtab”, O_RDONLY|O_CLOEXEC) = 3
24932 fstat(3, {st_mode=S_IFREG|0444, st_size=0, …}) = 0
24932 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba5c338000
24932 read(3, “rootfs / rootfs rw 0 0\nsysfs /sy”…, 1024) = 1024
24932 read(3, “eezer cgroup rw,nosuid,nodev,noe”…, 1024) = 1024
24932 read(3, “ogVol_u01 /u01 ext4 rw,relatime,”…, 1024) = 401
24932 read(3, “”, 1024) = 0
24932 close(3) = 0
24932 munmap(0x7fba5c338000, 4096) = 0
24932 lstat(“/dev”, {st_mode=S_IFDIR|0755, st_size=3540, …}) = 0
24932 lstat(“/dev/shm”, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=5480, …}) = 0
24932 stat(“/dev/shm”, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=5480, …}) = 0
24932 uname({sysname=”Linux”, nodename=”oracle122-rhel.vslab.local”, …}) = 0
24932 statfs(“/dev/shm”, {f_type=TMPFS_MAGIC, f_bsize=4096, f_blocks=6291456, f_bfree=6127616, f_bavail=6127616, f_files=4097625, f_ffree=4097352, f_fsid={0, 0}, f_namelen=255, f_frsize=409
6, f_flags=ST_VALID|ST_RELATIME}) = 0
24932 stat(“/dev/shm”, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=5480, …}) = 0
24932 open(“/usr/lib64/gconv/gconv-modules.cache”, O_RDONLY) = 3
24932 fstat(3, {st_mode=S_IFREG|0644, st_size=26254, …}) = 0
24932 mmap(NULL, 26254, PROT_READ, MAP_SHARED, 3, 0) = 0x7fba5c332000
24932 close(3) = 0
24932 fstat(1, {st_mode=S_IFIFO|0600, st_size=0, …}) = 0
24932 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba5c331000
24932 write(1, “Filesystem 1K-blocks Used “…, 114) = 114
24932 close(1 <unfinished …>
24917 <… read resumed> “Filesystem 1K-blocks Used “…, 4096) = 114
24932 <… close resumed> ) = 0
24932 munmap(0x7fba5c331000, 4096) = 0
24917 close(13 <unfinished …>
24932 close(2) = 0
24917 <… close resumed> ) = 0
24932 exit_group(0) = ?
24917 wait4(24932, <unfinished …>
24932 +++ exited with 0 +++
24917 <… wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 24932
24917 — SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=24932, si_uid=54321, si_status=0, si_utime=0, si_stime=0} —
24917 munmap(0x7fc6387df000, 4096) = 0
24917 get_mempolicy(NULL, NULL, 0, NULL, 0) = 0
24917 get_mempolicy(NULL, NULL, 0, NULL, 0) = 0
24917 shmget(IPC_PRIVATE, 4096, IPC_CREAT|IPC_EXCL|0600) = 983043
24917 shmat(983043, NULL, 0) = 0x7fc6387df000
24917 open(“/dev/shm/ora_DBPROD_983043_0”, O_RDWR|O_CREAT|O_SYNC, 0600) = 13
24917 close(13) = 0
24917 shmdt(0x7fc6387df000) = 0
24917 getrlimit(RLIMIT_STACK, {rlim_cur=32768*1024, rlim_max=32768*1024}) = 0
24917 brk(NULL) = 0x1603e000
24917 open(“/proc/self/maps”, O_RDONLY) = 13
24917 read(13, “00400000-15293000 r-xp 00000000 “…, 299) = 299
24917 read(13, ” /u01/app/o”…, 299) = 299
24917 read(13, ” /u01/app/oracle”…, 299) = 299
24917 read(13, “t/12.2.0/dbhome_1/lib/libnque12.”…, 299) = 299
24917 read(13, “377e000-7fc63377f000 r–p 0000b0″…, 299) = 299
24917 read(13, ” 33556075 /usr”…, 299) = 299
24917 read(13, “5 /usr/lib64/l”…, 299) = 299
24917 read(13, “p 00014000 fd:00 33554500 “…, 299) = 299
24917 read(13, “3d6b000-7fc633f6a000 —p 001c30″…, 299) = 299
24917 read(13, “7fc633f75000 rw-p 00000000 00:00″…, 299) = 299
24917 read(13, ” /usr/lib64/libreso”…, 299) = 299
24917 read(13, “1a4000-7fc6343a4000 —p 0001600″…, 299) = 299
24917 read(13, “6000-7fc6343a8000 rw-p 00000000 “…, 299) = 299
24917 read(13, “18 /usr/lib64/”…, 299) = 299
24917 read(13, “\n7fc6346c5000-7fc6348c4000 —p “…, 299) = 299
24917 read(13, “c6000-7fc6348c8000 r-xp 00000000″…, 299) = 299
24917 read(13, “-7fc634aca000 rw-p 00003000 fd:0″…, 299) = 299
24917 read(13, “le/product/12.2.0/dbhome_1/lib/l”…, 299) = 299
24917 read(13, “00001000 fd:00 33557989 “…, 299) = 299
24917 read(13, ” 00000000 fd:02 2885530 “…, 299) = 299
24917 read(13, ” /u01/app/oracle/product/”…, 299) = 299
24917 read(13, “0/dbhome_1/lib/libocrb12.so\n7fc6″…, 299) = 299
24917 read(13, “-7fc63578b000 —p 00109000 fd:0″…, 299) = 299
24917 read(13, ” /u01/app/ora”…, 299) = 299
24917 read(13, “duct/12.2.0/dbhome_1/lib/libskgx”…, 299) = 299
24917 read(13, “so\n7fc636c31000-7fc636c78000 rw-“…, 299) = 299
24917 read(13, “b/libdbcfg12.so\n7fc636ca1000-7fc”…, 299) = 299
24917 read(13, “d000 r-xp 00000000 fd:02 2889021″…, 299) = 299
24917 read(13, ” /u01/app/oracle/p”…, 299) = 299
24917 read(13, “2.2.0/dbhome_1/lib/libipc1.so\n7f”…, 299) = 299
24917 read(13, “duct/12.2.0/dbhome_1/lib/libmql1″…, 299) = 299
24917 read(13, “96000-7fc637798000 rw-p 00000000″…, 299) = 299
24917 read(13, ” /usr/lib64/librt-2.1″…, 299) = 299
24917 read(13, “041000 fd:02 2884686 “…, 299) = 299
24917 read(13, ” /u01/app/oracle/product/12.2.”…, 299) = 299
24917 read(13, “ome_1/lib/libskgxp12.so\n7fc637ef”…, 299) = 299
24917 read(13, “c6381bf000 rw-p 000b5000 fd:02 2″…, 299) = 299
24917 read(13, ” /u01/app/oracle/”…, 299) = 299
24917 read(13, “0/dbhome_1/lib/libodmd12.so\n7fc6″…, 299) = 299
24917 read(13, “0-7fc6385e4000 r-xp 00000000 fd:”…, 299) = 299
24917 read(13, “000 rw-p 00022000 fd:00 33555224″…, 299) = 299
24917 read(13, “-ffffffffff601000 r-xp 00000000 “…, 299) = 68
24917 close(13) = 0
24917 shmat(983043, NULL, SHM_RDONLY) = 0x7fc6387df000
24917 shmdt(0x7fc6387df000) = 0
24917 shmat(983043, NULL, 0) = 0x7fc6387df000
24917 shmdt(0x7fc6387df000) = 0
24917 mmap(0x60000000, 33554432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x60000000
24917 munmap(0x60000000, 33554432) = 0
24917 open(“/dev/shm/ora_DBPROD_983043_0”, O_RDWR|O_SYNC) = 13
24917 mmap(0x60000000, 33554432, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, 13, 0) = 0x60000000
24917 close(13) = 0
24917 lseek(8, 0, SEEK_CUR) = 851
24917 write(8, “\n*** 2018-10-11T16:06:11.497151-“…, 38) = 38
24917 write(11, “00+3}0Pc\n”, 9) = 9
24917 write(8, ” Shared memory segment allocated”…, 105) = 105
24917 write(8, “\n”, 1) = 1
24917 shmget(IPC_PRIVATE, 4096, IPC_CREAT|IPC_EXCL|0600) = 1015812
24917 shmat(1015812, NULL, 0) = 0x7fc6387df000
24917 open(“/dev/shm/ora_DBPROD_1015812_0”, O_RDWR|O_CREAT|O_SYNC, 0600) = 13
24917 close(13) = 0
24917 open(“/dev/shm/ora_DBPROD_1015812_1”, O_RDWR|O_CREAT|O_SYNC, 0600) = 13
24917 close(13) = 0
24917 open(“/dev/shm/ora_DBPROD_1015812_2”, O_RDWR|O_CREAT|O_SYNC, 0600) = 13
24917 close(13) = 0
24917 open(“/dev/shm/ora_DBPROD_1015812_3”, O_RDWR|O_CREAT|O_SYNC, 0600) = 13
24917 close(13) = 0
24917 open(“/dev/shm/ora_DBPROD_1015812_4”, O_RDWR|O_CREAT|O_SYNC, 0600) = 13
24917 close(13) = 0
24917 open(“/dev/shm/ora_DBPROD_1015812_5”, O_RDWR|O_CREAT|O_SYNC, 0600) = 13
24917 close(13) = 0
24917 open(“/dev/shm/ora_DBPROD_1015812_6”, O_RDWR|O_CREAT|O_SYNC, 0600) = 13
24917 close(13) = 0
24917 open(“/dev/shm/ora_DBPROD_1015812_7”, O_RDWR|O_CREAT|O_SYNC, 0600) = 13
……….

 

 

Conclusion

 

Persistent Memory (PMEM) resides between DRAM and disk storage in the data storage hierarchy. This technology enables byte-addressable updates and does not lose data if power is lost.

The Accelerating Oracle Performance using vSphere Persistent Memory (PMEM) paper examines the performance of Oracle databases using VMware vSphere 6.7 Persistent Memory feature in different modes for below uses cases for

  • Improved performance of Oracle Redo Log using vPMEM Disk-backed vmdks/vPMEM disks in DAX mode
  • Accelerating Performance using Oracle Smart Flash Cache
  • Potential reduction in Oracle Licensing

In the blog article Oracle and vSphere Persistent Memory (PMEM) – vPMEM v/s vPMEMDisk , we demonstrate the performance improvement in Redo log activity when redo log files are placed on vPMEM Disk-backed vmdks/vPMEM disks in DAX mode over redo logs on vPMEMDisk backed vmdks.

With PMem-aware Redis. By adding very low-overhead journaling to make Redis transactions on persistent memory crash consistent, we were able to reduce the crash recovery time,

We are not able to accelerate the Oracle crash recovery process in the same way as Oracle will still use its proprietary crash recovery mechanism. Oracle recreates the SGA granules on every instance startup , in /dev/shm , and hence even if the old SGA granules are present in PMEM , Oracle disregards it for crash recovery purposes.

Unless the Oracle software code is cognizant of the PMEM device and take advantage of the memory persistence , Oracle crash recovery will continue to function as is without any PMEM advantages.

All Oracle on vSphere white papers including Oracle on VMware vSphere / VMware vSAN / VMware Cloud on AWS , Best practices, Deployment guides, Workload characterization guide can be found at Oracle on VMware Collateral – One Stop Shop.

 

 

Posted in Uncategorized