DNS and NTP involves the changes in the four components on Exadata.
- DB nodes
- Cell nodes
- IB Switches
- Ethernet Switches
---IB Switches DNS--
If your switch is using firmware 2.0.4 or later:
ssh ilom-admin@99.99.99.12/13
-> set /SP/clients/dns nameserver=99.99.99.10,99.99.99.11
---IB Switches NTP--
ssh ilom-admin@99.99.99.12/13
-> show -d properties /SP/clock
/SP/clock
Properties:
datetime = Mon Jul 26 21:44:16 2021
timezone = +04 (Asia/Baku)
uptime = 464 days, 06:34:49
usentpserver = enabled
-> show -d properties /SP/clients/ntp/server/1
/SP/clients/ntp/server/1
Properties:
address = 88.88.88.250
-> show -d properties /SP/clients/ntp/server/2
/SP/clients/ntp/server/2
Properties:
address = 0.0.0.0
-> set /SP/clients/ntp/server/1 address=99.99.99.250
Set 'address' to '99.99.99.250'
-> set /SP/clients/ntp/server/2 address=99.99.99.251
Set 'address' to '99.99.99.251'
-> show -d properties /SP/clock
/SP/clock
Properties:
datetime = Mon Jul 26 21:50:48 2021
timezone = +04 (Asia/Baku)
uptime = 464 days, 06:41:20
usentpserver = enabled
-> show -d properties /SP/clients/ntp/server/1
/SP/clients/ntp/server/1
Properties:
address = 99.99.99.250
-> show -d properties /SP/clients/ntp/server/2
/SP/clients/ntp/server/2
Properties:
address = 99.99.99.251
--Cisco (Ethernet) switch DNS---
ssh admin@99.99.99.11
dm1sw-adm0# show running-config
!Command: show running-config
!Running configuration last done at: Mon Jul 26 21:00:03 2021
!Time: Mon Jul 26 21:01:42 2021
version 7.0(3)I7(6) Bios:version 05.34
......
no password strength-check
username admin password 5 $5$g6qgjbgv$hhEgdfgdfgdfgdfGhDIIc678GrOa82L6Lj. role network-admin
username ciscosnmp password 5 $5$ZWkcwADg$JgUzO9DCsdvsdvf65yMIJCcdD role network-operator
ip domain-lookup
ip domain-name mlspp.gov.az
ip name-server 88.88.88.10 88.88.88.11
system default switchport
copp profile lenient
.....
dm1sw-adm0# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
dm1sw-adm0(config)# no ip name-server 88.88.88.10
dm1sw-adm0(config)# no ip name-server 88.88.88.11
dm1sw-adm0(config)# end
dm1sw-adm0# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
dm1sw-adm0(config)# ip name-server 99.99.99.10
dm1sw-adm0(config)# ip name-server 99.99.99.11
dm1sw-adm0(config)# end
Finally Verify the changes
dm1sw-adm0# show running-config
Save the configuration.
dm1sw-adm0# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete.
Exit the session
dm1sw-adm0# exit
-- Cisco (ethernet) switch NTP ---
ssh admin@99.99.99.11
dm1sw-adm0# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
dm1sw-adm0(config)# no ntp server 88.88.88.250
dm1sw-adm0(config)# end
dm1sw-adm0# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
dm1sw-adm0(config)# ntp server 99.99.99.250 prefer
dm1sw-adm0(config)# ntp server 99.99.99.251
dm1sw-adm0(config)# end
Finally Verify the changes
dm1sw-adm0# show running-config
Save the configuration.
dm1sw-adm0# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete.
Exit the session
dm1sw-adm0# exit
--- DB nodes DNS ---
1. Log in to the database server as the root user.
Edit the /etc/resolv.conf file.
2. Set the DNS server and domain name using an editor such as vi. There should be a name server line for each DNS server.
search example.com
nameserver 99.99.99.10
nameserver 99.99.99.11
3.Set the DNS server in the server ILOM.
ssh root@99.99.99.6/7
-> show /SP/clients/dns
/SP/clients/dns
Targets:
Properties:
auto_dns = enabled
nameserver = 88.88.88.10
retries = 1
searchpath = mlspp.gov.az
timeout = 5
Commands:
cd
set
show
-> set /SP/clients/dns nameserver=99.99.99.10,99.99.99.11
Set 'nameserver' to '99.99.99.10,99.99.99.11' [99.99.99.10, 99.99.99.11]
--- DB nodes NTP ---
1. Stop the NTP/Chrony services on each database server.
[root@dm1db1 ~]# systemctl stop chronyd.service
2. Update the ntp.conf/chrony.conf file with the IP address of the new NTP server.
Start the NTP/Chrony services on the database server.
[root@dm1db1 ~]# vi /etc/chrony.conf
[root@dm1db1 ~]# systemctl start chronyd.service
3.Set the DNS server in the server ILOM for each node.
ssh root@99.99.99.6/7
-> set /SP/clients/ntp/server/1 address=99.99.99.250
Set 'address' to '99.99.99.250'
-> set /SP/clients/ntp/server/2 address=99.99.99.251
Set 'address' to '99.99.99.251'
-> show /SP/clients/ntp/server/1
/SP/clients/ntp/server/1
Targets:
Properties:
address = 99.99.99.250
-> show /SP/clock
/SP/clock
Targets:
Properties:
datetime = Mon Jul 26 23:03:43 2021
timezone = +04 (Asia/Baku)
uptime = 15 days, 06:27:05
usentpserver = enabled
--- Cell nodes DNS and NTP---
1. Log in to the Oracle Exadata Storage Server as the root user. (each cell separately and respectively)
2. Specify a time interval to repair the disk and bring it back online.
The default DISK_REPAIR_TIME attribute value of 3.6 hours should be long enough for most environments.
a. Check the repair time for all mounted disk groups.
Log in to the Oracle ASM instance and run the following query:
SQL> SELECT dg.name,a.value FROM v$asm_diskgroup dg, v$asm_attribute a WHERE dg.group_number=a.group_number AND a.name='disk_repair_time';
b. Adjust the DISK_REPAIR_TIME parameter, if needed.
In the following command, h.n is the amount of time in hours, such as 4.6.
SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'DISK_REPAIR_TIME'='h.nH';
3. Check that putting the grid disks offline will not cause a problem for Oracle ASM.
[root@db1cell2 ~]# cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome
DATAC1_CD_00_db1cell2 ONLINE Yes
DATAC1_CD_01_db1cell2 ONLINE Yes
DATAC1_CD_02_db1cell2 ONLINE Yes
DATAC1_CD_03_db1cell2 ONLINE Yes
DATAC1_CD_04_db1cell2 ONLINE Yes
DATAC1_CD_05_db1cell2 ONLINE Yes
RECOC1_CD_00_db1cell2 ONLINE Yes
RECOC1_CD_01_db1cell2 ONLINE Yes
RECOC1_CD_02_db1cell2 ONLINE Yes
RECOC1_CD_03_db1cell2 ONLINE Yes
RECOC1_CD_04_db1cell2 ONLINE Yes
RECOC1_CD_05_db1cell2 ONLINE Yes
The value Yes should be returned for the grid disks. If one or more disks does not return a Yes value, then restore data redundancy for the disk group, and repeat the command until all disk groups return a Yes value.
4. Inactivate all grid disks on the cell.
[root@db1cell2 ~]# cellcli -e alter griddisk all inactive
GridDisk DATAC1_CD_00_db1cell2 successfully altered
GridDisk DATAC1_CD_01_db1cell2 successfully altered
GridDisk DATAC1_CD_02_db1cell2 successfully altered
GridDisk DATAC1_CD_03_db1cell2 successfully altered
GridDisk DATAC1_CD_04_db1cell2 successfully altered
GridDisk DATAC1_CD_05_db1cell2 successfully altered
GridDisk RECOC1_CD_00_db1cell2 successfully altered
GridDisk RECOC1_CD_01_db1cell2 successfully altered
GridDisk RECOC1_CD_02_db1cell2 successfully altered
GridDisk RECOC1_CD_03_db1cell2 successfully altered
GridDisk RECOC1_CD_04_db1cell2 successfully altered
GridDisk RECOC1_CD_05_db1cell2 successfully altered
This command may take more than 10 minutes to complete. Inactivating the grid disks automatically sets the disk to offline in the Oracle ASM instance.
5. Confirm the grid disks are offline.
a. Check the status of the grid disks.
[root@db1cell2 ~]# cellcli -e list griddisk attributes name, asmmodestatus,asmdeactivationoutcome
DATAC1_CD_00_db1cell2 OFFLINE Yes
DATAC1_CD_01_db1cell2 OFFLINE Yes
DATAC1_CD_02_db1cell2 OFFLINE Yes
DATAC1_CD_03_db1cell2 OFFLINE Yes
DATAC1_CD_04_db1cell2 OFFLINE Yes
DATAC1_CD_05_db1cell2 OFFLINE Yes
RECOC1_CD_00_db1cell2 OFFLINE Yes
RECOC1_CD_01_db1cell2 OFFLINE Yes
RECOC1_CD_02_db1cell2 OFFLINE Yes
RECOC1_CD_03_db1cell2 OFFLINE Yes
RECOC1_CD_04_db1cell2 OFFLINE Yes
RECOC1_CD_05_db1cell2 OFFLINE Yes
The output should show asmmodestatus=OFFLINE or asmmodestatus=UNUSED, and asmdeactivationoutcome=Yes for all grid disks.
b. List the grid disk to confirm that they are offline.
[root@db1cell2 ~]# cellcli -e list griddisk
DATAC1_CD_00_db1cell2 inactive
DATAC1_CD_01_db1cell2 inactive
DATAC1_CD_02_db1cell2 inactive
DATAC1_CD_03_db1cell2 inactive
DATAC1_CD_04_db1cell2 inactive
DATAC1_CD_05_db1cell2 inactive
RECOC1_CD_00_db1cell2 inactive
RECOC1_CD_01_db1cell2 inactive
RECOC1_CD_02_db1cell2 inactive
RECOC1_CD_03_db1cell2 inactive
RECOC1_CD_04_db1cell2 inactive
RECOC1_CD_05_db1cell2 inactive
6. Shut down the cell services and ocrvottargetd service.
[root@db1cell2 ~]# cellcli -e alter cell shutdown services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Note: The ocrvottargetd service is not included in some releases. [service ocrvottargetd stop]
7. Use the ipconf utility to change the DNS settings.
[root@db1cell2 ~]# /usr/local/bin/ipconf
[Info]: ipconf command line: /opt/oracle.cellos/ipconf.pl -nocodes
Logging started to /var/log/cellos/ipconf.log
Interface ib0 is Linked. hca: mlx4_0
Interface ib1 is Linked. hca: mlx4_0
Interface eth0 is Linked. driver/mac: igb/00:10:e0:eb:28:6e
The current nameserver(s): 88.88.88.10 88.88.88.11
Do you want to change it (y/n) [n]: y
Nameserver: 99.99.99.10
Add more nameservers (y/n) [n]: y
Nameserver: 99.99.99.11
Add more nameservers (y/n) [n]: n
The current timezone: Asia/Baku
Do you want to change it (y/n) [n]: n
The current NTP server(s): 88.88.88.250
Do you want to change it (y/n) [n]: y
Fully qualified hostname or ip address for NTP server. Press enter if none: 99.99.99.250
Continue adding more ntp servers (y/n) [n]: y
Fully qualified hostname or ip address for NTP server. Press enter if none: 99.99.99.251
Continue adding more ntp servers (y/n) [n]: n
Network interfaces
Name State Speed Status IP address Netmask Gateway Net type Hostname
ib0 Linked UP 77.77.77.7 255.255.252.0 Private db1cell2-priv1.mlspp.gov.az
ib1 Linked UP 77.77.77.8 255.255.252.0 Private db1cell2-priv2.mlspp.gov.az
eth0 Linked default UP 99.99.99.4 255.255.255.0 99.99.99.253 Management db1cell2.mlspp.gov.az
Select interface name to configure or press Enter to continue:
Select canonical hostname from the list below
1: db1cell2-priv1.mlspp.gov.az
2: db1cell2-priv2.mlspp.gov.az
3: db1cell2.mlspp.gov.az
Canonical fully qualified domain name [3]:
Select default gateway interface from the list below
1: eth0
Default gateway interface [1]:
Canonical hostname: db1cell2.mlspp.gov.az
Nameservers: 99.99.99.10 99.99.99.11
Timezone: Asia/Baku
NTP servers: 99.99.99.250 99.99.99.251
Default gateway device: eth0
Network interfaces
Name State Speed Status IP address Netmask Gateway Net type Hostname
ib0 Linked UP 77.77.77.7 255.255.252.0 Private db1cell2-priv1.mlspp.gov.az
ib1 Linked UP 77.77.77.8 255.255.252.0 Private db1cell2-priv2.mlspp.gov.az
eth0 Linked default UP 99.99.99.4 255.255.255.0 99.99.99.253 Management db1cell2.mlspp.gov.az
Is this correct (y/n) [y]: y
Do you want to configure basic ILOM settings (y/n) [y]: y
Loading basic configuration settings from ILOM ...
ILOM Fully qualified hostname [db1cell2-ilom.mlspp.gov.az]:
Inet protocol (IPv4,IPv6) [IPv4]:
ILOM IP address [99.99.99.9]:
ILOM Netmask [255.255.255.0]:
ILOM Gateway or none [99.99.99.253]:
ILOM Nameserver (multiple IPs separated by a comma) or none [88.88.88.10]: 99.99.99.10
ILOM Use NTP Servers (enabled/disabled) [enabled]:
ILOM First NTP server. Fully qualified hostname or ip address or none [88.88.88.250]: 99.99.99.250
ILOM Second NTP server. Fully qualified hostname or ip address or none [none]: 99.99.99.251
ILOM Vlan id or zero for non-tagged VLAN (0-4079) [0]:
Basic ILOM configuration settings:
Hostname : db1cell2-ilom.mlspp.gov.az
IP Address : 99.99.99.9
Netmask : 255.255.255.0
Gateway : 99.99.99.253
DNS servers : 99.99.99.10
Use NTP servers : enabled
First NTP server : 99.99.99.250
Second NTP server : 99.99.99.251
Timezone (read-only) : Asia/Baku
VLAN id : 0
Is this correct (y/n) [y]: n
ILOM Fully qualified hostname [db1cell2-ilom.mlspp.gov.az]:
Inet protocol (IPv4,IPv6) [IPv4]:
ILOM IP address [99.99.99.9]:
ILOM Netmask [255.255.255.0]:
ILOM Gateway or none [99.99.99.253]:
ILOM Nameserver (multiple IPs separated by a comma) or none [88.88.88.10]: 99.99.99.10,99.99.99.11
ILOM Use NTP Servers (enabled/disabled) [enabled]:
ILOM First NTP server. Fully qualified hostname or ip address or none [88.88.88.250]: 99.99.99.250
ILOM Second NTP server. Fully qualified hostname or ip address or none [none]: 99.99.99.251
ILOM Vlan id or zero for non-tagged VLAN (0-4079) [0]:
Basic ILOM configuration settings:
Hostname : db1cell2-ilom.mlspp.gov.az
IP Address : 99.99.99.9
Netmask : 255.255.255.0
Gateway : 99.99.99.253
DNS servers : 99.99.99.10,99.99.99.11
Use NTP servers : enabled
First NTP server : 99.99.99.250
Second NTP server : 99.99.99.251
Timezone (read-only) : Asia/Baku
VLAN id : 0
Is this correct (y/n) [y]: y
[Info]: Run /opt/oracle.cellos/validations/init.d/saveconfig
[Info]: Custom changes have been detected in /etc/resolv.conf
[Info]: Original file /etc/resolv.conf will be saved in /etc/resolv.conf.backupbyExadata
[Info]: Stopping cellwall service ...
[Info]: cellwall service stopped
[Info]: Restart chronyd service
[Info]: Starting cellwall service ...
[Info]: cellwall service started
[Info]: Save /etc/sysctl.conf in /etc/sysctl.conf.backupbyExadata
[Info]: Adjust settings for IB interfaces in /etc/sysctl.conf
[Info]: Retarting cellwall service ...
active
[Info]: cellwall service restarted
Re-login using new IP address 99.99.99.4 if you were disconnected after following commands
ip addr show eth0
sleep 4
[Warning]: You modified NTP server.
Ensure you also update the Infiniband Switch NTP server
if the same NTP server was also used by the Infiniband switch.
[Warning]: You modified DNS name server.
Ensure you also update the Infiniband Switch DNS server
if the same DNS server was also used by the Infiniband switch.
8. Restart the cell services and ocrvottargetd service.
[root@db1cell2 ~]# cellcli -e alter cell startup services all
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
The server does not need to reboot.
9. Activate the grid disks when the cell comes online.
[root@db1cell2 ~]# cellcli -e alter griddisk all active
GridDisk DATAC1_CD_00_db1cell2 successfully altered
GridDisk DATAC1_CD_01_db1cell2 successfully altered
GridDisk DATAC1_CD_02_db1cell2 successfully altered
GridDisk DATAC1_CD_03_db1cell2 successfully altered
GridDisk DATAC1_CD_04_db1cell2 successfully altered
GridDisk DATAC1_CD_05_db1cell2 successfully altered
GridDisk RECOC1_CD_00_db1cell2 successfully altered
GridDisk RECOC1_CD_01_db1cell2 successfully altered
GridDisk RECOC1_CD_02_db1cell2 successfully altered
GridDisk RECOC1_CD_03_db1cell2 successfully altered
GridDisk RECOC1_CD_04_db1cell2 successfully altered
GridDisk RECOC1_CD_05_db1cell2 successfully altered
10. Verify the disks are active.
[root@db1cell2 ~]# cellcli -e list griddisk
DATAC1_CD_00_db1cell2 active
DATAC1_CD_01_db1cell2 active
DATAC1_CD_02_db1cell2 active
DATAC1_CD_03_db1cell2 active
DATAC1_CD_04_db1cell2 active
DATAC1_CD_05_db1cell2 active
RECOC1_CD_00_db1cell2 active
RECOC1_CD_01_db1cell2 active
RECOC1_CD_02_db1cell2 active
RECOC1_CD_03_db1cell2 active
RECOC1_CD_04_db1cell2 active
RECOC1_CD_05_db1cell2 active
The output should show active.
11. Verify the grid disk status.
a. Check that all grid disks are online.
cellcli -e list griddisk attributes name, asmmodestatus
b. Wait for Oracle ASM synchronization to complete for all grid disks. Each disk will go to a SYNCING state first then ONLINE.
[root@db1cell2 ~]# cellcli -e list griddisk attributes name, asmmodestatus
DATAC1_CD_00_db1cell2 ONLINE
DATAC1_CD_01_db1cell2 SYNCING
DATAC1_CD_02_db1cell2 SYNCING
DATAC1_CD_03_db1cell2 SYNCING
DATAC1_CD_04_db1cell2 SYNCING
DATAC1_CD_05_db1cell2 SYNCING
RECOC1_CD_00_db1cell2 ONLINE
RECOC1_CD_01_db1cell2 ONLINE
RECOC1_CD_02_db1cell2 SYNCING
RECOC1_CD_03_db1cell2 SYNCING
RECOC1_CD_04_db1cell2 SYNCING
RECOC1_CD_05_db1cell2 SYNCING
Oracle ASM synchronization is complete when all grid disks show asmmodestatus=ONLINE.
12. Repeat this procedure for each Oracle Exadata Storage Server.
references: https://docs.oracle.com/cd/E80920_01/DBMMN/maintaining-exadata-components.htm#DBMMN22932
Changing IP addresses on Exadata Database Machine (Doc ID 1317159.1)
No comments:
Post a Comment