使用oracle 11g 的direct nfs 存放数据文件

Oracle11gR2中已经集成了direct nfs driver，而在此之前是没有改驱动的

也就是说在11gR2中，经过简单设置之后，就能直接利用其他NAS存储设备了。

和传统的nfs比,好处是提高了性能和配置的简便性.

下面简要描述一下direct nfs 的配置过程。

环境如下：两台linux5.5

其中：一台安装oracle11gR2，也是nfs client （IP：192.168.133.11,NAME:edbj2p1）；

一台nfs server （IP：192.168.133.12,NAME:edbj2p2） ；

服务器端共享的目录: /vol/oradata

客户端挂载点:/u02/oradata/prod1

创建的表空间:shared_data

1.配置服务端edbj2p2

[root@edbj2p2 ~]# echo '/vol/oradata 192.168.133.11(rw,no_root_squash,insecure)' >> /etc/exports

注意:这里一定要加 insecure,否则,无法正常使用dnfs

[root@edbj2p2 ~]# exportfs -arv

[root@edbj2p2 ~]# service nfs restart

--确定服务端的nfs共享

[root@edbj2p2 ~]# showmount -e edbj2p2

Export list for edbj2p2:

/vol/oradata 192.168.133.11

2.配置客户端edbj2p1

2.1 确定portmap服务运行

[root@edbj2p2 ~]# service portmap status

portmap (pid 3540) is running...

2.2先确认在客户端机器可以看到服务器端共享出来的nfs

[root@edbj2p1 ~]# showmount -e edbj2p2

Export list for edbj2p2:

/vol/oradata 192.168.133.11

2.3调整libodm11.so文件

[oracle@edbj2p1 ~]$ cd $ORACLE_HOME/lib

[oracle@edbj2p1 lib]$ pwd

/u01/app/oracle/product/11.2.0/dbhome_1/lib

--修改libodm11.so,使用libnfsodm11.so

[oracle@edbj2p1 lib]$ mv libodm11.so libodm11.so.bak

[oracle@edbj2p1 lib]$ ln -s libnfsodm11.so libodm11.so

或者 一条命令搞定:

make -f ins_rdbms.mk dnfs_on

确认一下

[oracle@edbj2p1 lib]$ ls -ltr libodm*

-rw-r--r-- 1 oracle oinstall 5266 Sep 18 2011 libodm11.a

-rw-r--r-- 1 oracle oinstall 8957 Sep 18 2011 libodmd11.so

lrwxrwxrwx 1 oracle oinstall 12 Feb 5 16:05 libodm11.so.bak -> libodmd11.so

lrwxrwxrwx 1 oracle oinstall 14 Feb 6 11:39 libodm11.so -> libnfsodm11.so

2.4 创建oranfstab文件

--进入$ORACLE_HOME/dbs目录

[oracle@edbj2p1 lib]$ cd $ORACLE_HOME/dbs

[oracle@edbj2p1 dbs]$ pwd

/u01/app/oracle/product/11.2.0/dbhome_1/dbs

--创建 oranfstab

[oracle@edbj2p1 dbs]$ touch oranfstab

[oracle@edbj2p1 dbs]$ echo 'server:edbj2p2' >> oranfstab

[oracle@edbj2p1 dbs]$ echo 'path:192.168.133.12' >> oranfstab

[oracle@edbj2p1 dbs]$ echo 'export: /vol/oradata mount: /u02/oradata/prod1' >> oranfstab

其中：server:edbj2p2 代表nfs server的hostname

path:192.168.133.12 代表nfs server的IP

export: /vol/oradata 代表服务器端的目录

mount: /u02/oradata/prod1 代表nfs client上的目录

--在客户端事先建好目录

[root@edbj2p1 ~]# mkdir -p /u02/oradata/prod1

[root@edbj2p1 ~]# chown -R oracle:oinstall /u02/oradata/prod1

[root@edbj2p1 ~]# chmod -R 775 /u02/oradata/prod1

3.加载nfs (在客户端操作)

3.1 直接加载nfs

mount -t nfs edbj2p2:/vol/oradata /u02/oradata/prod1

3.2 永久生效,需要修改/etc/fstab

192.168.133.12:/vol/oradata /u02/oradata/prod1 nfs defaults 0 0

3.3 检查挂载结果

[root@edbj2p1 ~]# df -h

Filesystem Size Used Avail Use% Mounted on

...

192.168.133.12:/vol/oradata

16G 2.8G 12G 20% /u02/oradata/prod1

4.dnfs生效 (在客户端操作)

只有重起数据库,才能生效

sqlplus /nolog

sql>startup force

检查数据库日志,有以下记录:

...

diagnostic_dest = "/u01/app/oracle/product/11.2.0/dbhome_1"

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0

Mon Feb 06 14:35:54 2017

...

Direct NFS: attempting to mount /vol/oradata on filer edbj2p2 defined in oranfstab

Direct NFS: channel config is:

channel id [0] local [] path [192.168.133.12]

Direct NFS: mount complete dir /vol/oradata on edbj2p2 mntport 854 nfsport 2049

Direct NFS: channel id [0] path [192.168.133.12] to filer [edbj2p2] via local [] is UP

Direct NFS: channel id [1] path [192.168.133.12] to filer [edbj2p2] via local [] is UP

5.在dnfs上建立表空间 (在客户端操作)

create tablespace SHARED_DATA datafile '/u02/oradata/prod1/shared_data01.dbf' size 10m;

题外话：

create tablespace SHARED_DATA datafile '/u02/oradata/prod1/shared_data01.dbf' size 10m

Direct NFS: please check that oradism is setuid

如果后端报这个错,需要设置setuid

chown root oradism

chmod u+s oradism

6.验证:

6.1 v$dnfs_servers中,svrname应该是oranfstab中的server后面的项目

col dirname for a13

col SVRNAME for a10

select * from v$dnfs_servers;

1* select * from v$dnfs_servers

ID SVRNAME DIRNAME MNTPORT NFSPORT WTMAX RTMAX

---------- ---------- ------------- ---------- ---------- ---------- ----------

1 edbj2p2 /vol/oradata 854 2049 32768 32768

6.2 数据文件使用情况

col filename for a40

select * from V$DNFS_FILES

FILENAME FILESIZE PNUM SVR_ID

---------------------------------------- ---------- ---------- ----------

/u02/oradata/prod1/shared_data01.dbf 10493952 10 1

6.3 dnfs通道情况

col path for a20

col local for a10

set line 200

select * from V$DNFS_CHANNELS

PNUM SVRNAME PATH LOCAL CH_ID SVR_ID SENDS RECVS PINGS

---------- ---------- -------------------- ---------- ---------- ---------- ---------- ---------- ----------

10 edbj2p2 192.168.133.12 0 1 0 0 0

10 edbj2p2 192.168.133.12 1 1 0 0 0

11 edbj2p2 192.168.133.12 0 1 0 0 0

11 edbj2p2 192.168.133.12 1 1 0 0 0

30 edbj2p2 192.168.133.12 0 1 0 0 0

6.4 服务器端端口连接情况:

[root@edbj2p2 oradata]# netstat -an |grep 2049

tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN

tcp 0 0 192.168.133.12:2049 192.168.133.11:20295 ESTABLISHED

tcp 0 0 192.168.133.12:2049 192.168.133.11:51581 ESTABLISHED

tcp 0 0 192.168.133.12:2049 192.168.133.11:690 ESTABLISHED

tcp 0 0 192.168.133.12:2049 192.168.133.11:36340 ESTABLISHED

tcp 0 0 192.168.133.12:2049 192.168.133.11:15344 ESTABLISHED

udp 0 0 0.0.0.0:2049 0.0.0.0:*