Oracle 11g R2 RAC学习测试环境搭建——安装环境准备

大家好！通过Virtual Box搭建Oracle 11g R2 RAC学习测试环境，如需要Oracle 10g RAC搭建教程请查看上一篇教程——《Oracle 10g R2 RAC学习环境搭建教程》，文章中如有不足之处或者有任何疑问，请指出。

由于时间仓促，今天只写了“安装前的环境准备”，软件安装过程将在明天更新上去，期待各位的指点……

分享知识，只为共同进步^_^

1. 相关软件

OS:Oracle Linux 6.6 64bit

SoftWare(oracle 11g R2): Grid(Grid_linux_x86-64 11G R2 11.2.0.4)

Database(database_linux_x86-64 11G R2 11.2.0.4)

相关工具：CRT、Xmanager、rlwrap

注：安装环境为VirtualBox虚拟机

节点物理内存：4G

共享存储（ASM）：DATAS（50G）、BACKUP（50G）、OCR（1G*3）

AD&DNS&NTP：Win Server 2012 R2（192.168.31.31/24）

2. 安装OS注意事项【RAC1&RAC2】

分区：简单分区两个/和swap 注：swap分区为物理内存的两倍

软件包安装：

Base System > Base
Base System > Client management tools
Base System > Compatibility libraries
Base System > Hardware monitoring utilities
Base System > Large Systems Performance
Base System > Network file system client
Base System > Performance Tools
Base System > Perl Support
Servers > Server Platform
Servers > System administration tools
Desktops > Desktop
Desktops > Desktop Platform
Desktops > Fonts
Desktops > General Purpose Desktop
Desktops > Graphical Administration Tools
Desktops > Input Methods
Desktops > X Window System
Applications > Internet Browser
Development > Additional Development
Development > Development Tools

字符界面设置【RAC1$RAC2】

[root@RAC1 ~]# vi /etc/inittab

[root@RAC2 ~]# vi /etc/inittab

id:3:initdefault:

关闭防火墙

service iptables stop
chkconfig iptables off
chkconfig --list iptables
setenforce 0
getenforce
vi /etc/sysconfig/selinux
SELINUX=disabled

关闭sendmail服务

service sendmail off 
chkconfig sendmail off 
chkconfig –list sendmail

3. 安装rlwrap软件【RAC1&RAC2】

[root@rac1 ~]# tar -zxvf rlwrap-0.42.tar.gz
[root@rac1 ~]# ll
[root@rac1 ~]# cd rlwrap-0.42
[root@rac1 rlwrap-0.42]# ./configure
[root@rac1 rlwrap-0.42]# make
[root@rac1 rlwrap-0.42]# make install

注：rlwrap软件安装后，就可以在sqlplus或者rman中使用上下箭头翻页查找执行过的语句会在oracle用户下的.bash_profile后面加alias sqlplus='rlwrap sqlplus’、alias rman='rlwrap rman'等等

4. 安装oracle软件需要的RPM包【RAC1&RAC2】

[root@RAC1 Packages]# mount /dev/cdrom /tmp/
[root@RAC1 Packages]# cd /tmp/Server/Packages
rpm -Uvh binutils-2.*
rpm -Uvh compat-libstdc++-33*
rpm -Uvh elfutils-libelf-0.*
rpm -Uvh libaio-0.*
rpm -Uvh libaio-devel-0.*
rpm -Uvh sysstat-9.*
rpm -Uvh glibc-2.*
rpm -Uvh glibc-common-2.*
rpm -Uvh glibc-devel-2.* glibc-headers-2.*
rpm -Uvh ksh-2*
rpm -Uvh make-3.*
rpm -Uvh libgcc-4.*
rpm -Uvh libstdc++-4.*
rpm -Uvh libstdc++-4.*.i686*
rpm -Uvh libstdc++-devel-4.*
rpm -Uvh gcc-4.*x86_64*
rpm -Uvh gcc-c++-4.*x86_64*
rpm -Uvh --allfiles elfutils-libelf-0*x86_64* elfutils-libelf-devel-0*x86_64*
rpm -Uvh elfutils-libelf-0*i686* elfutils-libelf-devel-0*i686*
rpm -Uvh libtool-ltdl*i686*
rpm -Uvh ncurses*i686*
rpm -Uvh readline*i686*
rpm -Uvh unixODBC*

5. 修改hosts文件【RAC1&RAC2】

vi /etc/hosts

192.168.31.11 rac1 rac1.cwucao.com
192.168.31.12 rac2 rac2.cwucao.com
 
192.168.31.13 rac1-vip rac1-vip.cwucao.com
192.168.31.14 rac2-vip rac2-vip.cwucao.com
 
10.0.0.1 rac1-priv rac1-priv.cwucao.com
10.0.0.2 rac2-priv rac2-priv.cwucao.com

测试【RAC1&RAC2】：

ping rac1 ping rac1.cwucao.com 
ping rac2 ping rac2.cwucao.com
ping rac1-priv ping rac1-priv.cwucao.com
ping rac2-priv ping rac2-priv.cwucao.com

在DNS服务器中添加三个主机记录

192.168.31.15/24 scan-cluster.cwucao.com
192.168.31.16/24 scan-cluster.cwucao.com
192.168.31.17/24 scan-cluster.cwucao.com

测试【RAC1&RAC2】：

nslookup scan-cluster

6. 设置内核参数和其它参数【RAC1&RAC2】

[root@RAC1 Packages]# vi /etc/sysctl.conf

# Disable netfilter on bridges.
#net.bridge.bridge-nf-call-ip6tables = 0
#net.bridge.bridge-nf-call-iptables = 0
#net.bridge.bridge-nf-call-arptables = 0
 
# Controls the maximum number of shared memory segments, in pages
#kernel.shmall = 4294967296
 
# oracle-rdbms-server-11gR2-preinstall setting for fs.file-max is 6815744
fs.file-max = 6815744
 
# oracle-rdbms-server-11gR2-preinstall setting for kernel.sem is '250 32000 100 128'
kernel.sem = 250 32000 100 128
 
# oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmni is 4096
kernel.shmmni = 4096
 
# oracle-rdbms-server-11gR2-preinstall setting for kernel.shmall is 1073741824 on x86_64
# oracle-rdbms-server-11gR2-preinstall setting for kernel.shmall is 2097152 on i386
kernel.shmall = 1073741824
 
# oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmax is 4398046511104 on x86_64
# oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmax is 4294967295 on i386
kernel.shmmax = 4398046511104
 
# oracle-rdbms-server-11gR2-preinstall setting for net.core.rmem_default is 262144
net.core.rmem_default = 262144
 
# oracle-rdbms-server-11gR2-preinstall setting for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304
 
# oracle-rdbms-server-11gR2-preinstall setting for net.core.wmem_default is 262144
net.core.wmem_default = 262144
 
# oracle-rdbms-server-11gR2-preinstall setting for net.core.wmem_max is 1048576
net.core.wmem_max = 1048576
 
# oracle-rdbms-server-11gR2-preinstall setting for fs.aio-max-nr is 1048576
fs.aio-max-nr = 1048576
 
# oracle-rdbms-server-11gR2-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500

[root@RAC1 Packages]# sysctl -p

使上面修改的配置生效

[root@RAC1 Packages]# vi /etc/security/limits.conf

# oracle-rdbms-server-11gR2-preinstall setting for nofile soft limit is 1024
#oracle soft nofile 1024
 
# oracle-rdbms-server-11gR2-preinstall setting for nofile hard limit is 65536
#oracle hard nofile 65536
 
# oracle-rdbms-server-11gR2-preinstall setting for nproc soft limit is 16384
# refer orabug15971421 for more info.
#oracle soft nproc 16384
 
# oracle-rdbms-server-11gR2-preinstall setting for nproc hard limit is 16384
#oracle hard nproc 16384
 
# oracle-rdbms-server-11gR2-preinstall setting for stack soft limit is 10240KB
oracle soft stack 10240
grid soft stack 10240
 
# oracle-rdbms-server-11gR2-preinstall setting for stack hard limit is 32768KB
oracle hard stack 32768
grid hard stack 32768
 
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

[root@RAC1 Packages]# vi /etc/pam.d/login

session required /lib64/security/pam_limits.so
session required pam_limits.so

注意：因为是64位的系统，所以一定要写/lib64/security/pam_limits.so

如果写成/lib/security/pam_limits.so（这是32位系统的设置）

[root@RAC1 Packages]# vi /etc/profile

if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi

执行source /etc/profile命令使配置生效

[root@RAC1 Packages]# vi /etc/csh.login

if ( $USER = "oracle" || $USER = "grid" ) then
limit maxproc 16384
limit descriptors 65536
endif
EOFCSH

[root@RAC1 Packages]# vi /etc/security/limits.d/90-nproc.conf

#* soft nproc 1024
* - nproc 16384

7. 配置NTP服务，实现网络时间同步

1）配置Win server 2012 R2为NTP服务器

1.修改以下选项的键值HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpServer内的[Enabled]设定为1。

打开NTP服务器功能（默认是不开启NTP Server服务，除非电脑升级成为域控制站）。

2.修改以下键值HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config\AnnounceFlags设定为5．

该设定强制主机将它自身宣布为可靠的时间源，从而使用内置的互补金属氧化物半导体 (CMOS) 时钟。如果要采用外面的时间服务器就用默认的a值即可．

3.修改以下键值HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters\Type内的[Type]设定为NTP。

4.重启Win32Time服务：先关闭 windows time 服务，再开启该服务。可以在“管理工具”的“服务”界面下完成，也可以以 DOS 方式输入“net stop w32time”、“net start w32time”。

至此，已完成服务器端设定．

2）配置节点【RAC1&RAC2】

[root@rac1 ~]# service ntpd status
ntpd is stopped
chkconfig ntpd off
mv /etc/ntp.conf /etc/ntp.conf.original
[root@rac1 ~]# ntpdate 192.168.31.31
30 Aug 19:40:31 ntpdate[12102]: step time server 192.168.75.1 offset -28799.377040 sec
[root@rac1 ~]# crontab –e
* * * * * ntpdate 192.168.31.31

这个脚本表示每分钟同步一次，crontab脚本间隔最短时间只能设置为1分钟

[root@rac1 ~]# crontab -l
* * * * * ntpdate 192.168.31.31
[root@rac1 ~]# service crond restart
Stopping crond: [ OK ]
Starting crond: [ OK ]

查看crontab的运行日志

[root@rac1 ~]# tail -10f /var/log/cron

8. 确认物理内存和交换内存信息

[root@RAC1 ~]# grep MemTotal /proc/meminfo
MemTotal: 3785084 kB
[root@RAC1 ~]# grep SwapTotal /proc/meminfo 
SwapTotal: 8388604 kB

9. 创建用户和组【RAC1&RAC2】

userdel -r oracle
groupdel dba
groupdel oinstall
 
groupadd -g 501 oinstall
groupadd -g 502 dba
groupadd -g 503 oper
groupadd -g 504 asmadmin
groupadd -g 506 asmdba
groupadd -g 507 asmoper
useradd -u 501 -g oinstall -G dba,asmadmin,asmdba,asmoper grid
useradd -u 502 -g oinstall -G dba,asmdba,oper oracle
 
passwd grid
passwd oracle

10. 创建软件目录【RAC1&RAC2】

创建Oracle RAC 所使用的目录，并修改相应的权限，在所有节点执行；

mkdir -p /u01/app/grid/
mkdir -p /u01/app/11.2.0/grid/product/db_1
mkdir -p /u01/app/oraInventory
chown -R grid:oinstall /u01/app
mkdir -p /u01/app/oracle/product/11.2.0/db_1
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01

注：grid用户的ORACLE_BASE不能包含ORACLE_HOME,比如 /u01/grid,/u01/grid/11.2.0就不行

11. 修改grid用户环境变量【RAC1&RAC2】

[root@RAC1 ~]# su - grid

[grid@RAC1 ~]$ cd /home/grid/

[grid@RAC1 ~]$ vi .bash_profile

TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
 
ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_BASE=/u01/app/grid; export ORACLE_BASE
ORACLE_HOME=/u01/app/11.2.0/grid/product/db_1; export ORACLE_HOME
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
THREADS_FLAG=native; export THREADS_FLAG
PATH=$ORACLE_HOME/bin:$PATH; export PATH
 
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
 
export DISPLAY=192.168.31.104:0.0

[grid@RAC1 ~]$ . .bash_profile --使上面修改的配置生效

12. 修改oracle用户环境变量【RAC1&RAC2】

[root@RAC1 ~]# su - oracle

[oracle@RAC1 ~]$ cd /home/oracle/

[oracle@RAC1 ~]$ vi .bash_profile

TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
 
ORACLE_HOSTNAME=RAC1.cwucao.com; export ORACLE_HOSTNAME
ORACLE_UNQNAME=racdb; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1; export ORACLE_HOME
ORACLE_SID=racdb1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
 
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH
 
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
NLS_LANG=AMERICAN_AMERICA.ZHS16GBK; export NLS_LANG
 
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
 
export DISPLAY=192.168.31.104:0.0
alias sqlplus='rlwrap sqlplus'
alias rman='rlwrap rman'

[oracle@RAC1 ~]$ . .bash_profile --使上面修改的配置生效

查看环境变量已经生效

env | grep ORA

13. 配置用户等效性

在rac1上

[root@rac1 ~]# su - grid
[grid@rac1 ~]$ mkdir ~/.ssh
[grid@rac1 ~]$ chmod 755 ~/.ssh
[grid@rac1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
e4:a4:16:17:ce:66:9e:ac:53:0f:92:02:9e:a3:23:89 grid@rac1.cwucao.com
[grid@rac1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is:
5a:8a:b5:7b:6e:f9:bd:17:2c:a9:f2:76:16:d5:03:54 grid@rac1.cwucao.com

在rac2上

[root@rac2 ~]# su - grid
[grid@rac2 ~]$ mkdir ~/.ssh
[grid@rac2 ~]$ chmod 755 ~/.ssh
[grid@rac2 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
ec:73:d6:01:68:7e:be:92:dc:08:ec:c0:26:dd:b9:4e grid@rac2.cwucao.com
[grid@rac2 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is:
09:5f:88:8b:0f:cf:df:39:93:dc:c7:e3:fe:2e:92:d7 grid@rac2.cwucao.com

在rac1上

[grid@rac1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[grid@rac1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[grid@rac1 ~]$ ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'rac2 (192.168.20.32)' can't be established.
RSA key fingerprint is 6e:5b:7d:2c:d7:d2:e3:ca:59:79:59:73:22:da:8b:77.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac2,192.168.20.32' (RSA) to the list of known hosts.
grid@rac2's password: 
[grid@rac1 ~]$ ssh rac2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
grid@rac2's password: 
[grid@rac1 ~]$ scp ~/.ssh/authorized_keys rac2:~/.ssh/authorized_keys
grid@rac2's password: 
authorized_keys 100% 2032 2.0KB/s 00:00 

在两个节点中，验证ssh

注：第一次執行，需輸入yes.再次執行後，即可

[grid@rac1 ~]$ ssh rac1 date
[grid@rac1 ~]$ ssh rac2 date
[grid@rac1 ~]$ ssh rac1-priv date
[grid@rac1 ~]$ ssh rac2-priv date
[grid@rac2 ~]$ ssh rac1 date
[grid@rac2 ~]$ ssh rac2 date
[grid@rac2 ~]$ ssh rac1-priv date
[grid@rac2 ~]$ ssh rac2-priv date

注：oracle用户也需要进行用户等效性，执行上面的操作即可

14. 安装并配置ASM 驱动

查询系统信息

uname -a
lsb_release -a

确保电脑安装以下软件包

[root@RAC1 ~]# rpm -qa *oracleasm*
oracleasmlib-2.0.12-1.el6.x86_64
kmod-oracleasm-2.0.8-4.el6_6.x86_64
oracleasm-support-2.1.8-1.el6.x86_64

初始化asmlib【RAC1&RAC2】

[root@rac1 oracle]# /etc/init.d/oracleasm configure -i
Configuring the Oracle ASM library driver.
 
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
 
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: 
Writing Oracle ASM library driver configuration: [ OK ]
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]

创建asm磁盘【RAC1】

[root@rac1 oracle]# /etc/init.d/oracleasm createdisk Vol1_OCR /dev/sdb1
Marking disk "/dev/sdb1" as an ASM disk: [ OK ]
[root@rac1 oracle]# /etc/init.d/oracleasm createdisk Vol2_OCR /dev/sdc1
Marking disk "/dev/sdc1" as an ASM disk: [ OK ]
[root@rac1 oracle]# /etc/init.d/oracleasm createdisk Vol3_OCR /dev/sdd1
Marking disk "/dev/sdd1" as an ASM disk: [ OK ]
[root@rac1 oracle]# /etc/init.d/oracleasm createdisk DATA_Vol1 /dev/sde1
Marking disk "/dev/sde1" as an ASM disk: [ OK ]
[root@rac1 oracle]# /etc/init.d/oracleasm createdisk BACKUP_Vol1 /dev/sdf1
Marking disk "/dev/sdf1" as an ASM disk: [ OK ]
 
[root@rac1 oracle]# /etc/init.d/oracleasm listdisks
BACKUP_VOL1
DATA_VOL1
VOL1_OCR
VOL2_OCR
VOL3_OCR

在【RAC2】上执行扫描，识别asm磁盘

[root@rac2 ~]# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks: [ OK ]
[root@rac2 ~]# /etc/init.d/oracleasm listdisks
BACKUP_VOL1
DATA_VOL1
VOL1_OCR
VOL2_OCR
VOL3_OCR

15. 安装grid前检查

安装cvuqdisk软件包

[root@rac1 software]# export CVUQDISK_GRP=oinstall
[root@rac1 software]# cd grid/rpm/
[root@rac1 rpm]# rpm -ivh cvuqdisk-1.0.7-1.rpm 
Preparing... ########################################### [100%]
 1:cvuqdisk ########################################### [100%]

注：在rac2上也需要执行安装

进行检查grid安装环境

[grid@RAC1 ~]$ /soft/grid/runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose

至此，安装环境准备好。。。。。。。。。。