Tuesday, July 27, 2021

DNS and NTP changes on Exadata

 DNS and NTP involves the changes in the four components on Exadata.

 - DB nodes

 - Cell nodes

 - IB Switches

 - Ethernet Switches

---IB Switches DNS--


If your switch is using firmware 2.0.4 or later:


ssh ilom-admin@99.99.99.12/13

-> set /SP/clients/dns nameserver=99.99.99.10,99.99.99.11

---IB Switches NTP--

ssh ilom-admin@99.99.99.12/13

-> show -d properties /SP/clock

  /SP/clock

Properties:

datetime = Mon Jul 26 21:44:16 2021

timezone = +04 (Asia/Baku)

uptime = 464 days, 06:34:49

usentpserver = enabled

-> show -d properties /SP/clients/ntp/server/1

  /SP/clients/ntp/server/1

Properties:

address = 88.88.88.250

-> show -d properties /SP/clients/ntp/server/2

  /SP/clients/ntp/server/2

Properties:

address = 0.0.0.0

-> set  /SP/clients/ntp/server/1 address=99.99.99.250

Set 'address' to '99.99.99.250'

-> set  /SP/clients/ntp/server/2 address=99.99.99.251

Set 'address' to '99.99.99.251'

-> show -d properties /SP/clock

  /SP/clock

Properties:

datetime = Mon Jul 26 21:50:48 2021

timezone = +04 (Asia/Baku)

uptime = 464 days, 06:41:20

usentpserver = enabled


->  show -d properties /SP/clients/ntp/server/1

  /SP/clients/ntp/server/1

Properties:

address = 99.99.99.250

->  show -d properties /SP/clients/ntp/server/2

  /SP/clients/ntp/server/2

Properties:

address = 99.99.99.251


        --Cisco (Ethernet) switch DNS---

ssh admin@99.99.99.11  

dm1sw-adm0# show running-config

!Command: show running-config

!Running configuration last done at: Mon Jul 26 21:00:03 2021

!Time: Mon Jul 26 21:01:42 2021

version 7.0(3)I7(6) Bios:version 05.34

......

no password strength-check

username admin password 5 $5$g6qgjbgv$hhEgdfgdfgdfgdfGhDIIc678GrOa82L6Lj.  role network-admin

username ciscosnmp password 5 $5$ZWkcwADg$JgUzO9DCsdvsdvf65yMIJCcdD  role network-operator

ip domain-lookup

ip domain-name mlspp.gov.az

ip name-server 88.88.88.10 88.88.88.11

system default switchport

copp profile lenient

.....

dm1sw-adm0# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

dm1sw-adm0(config)# no ip name-server 88.88.88.10

dm1sw-adm0(config)# no ip name-server 88.88.88.11

dm1sw-adm0(config)# end

dm1sw-adm0# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

dm1sw-adm0(config)# ip name-server 99.99.99.10

dm1sw-adm0(config)# ip name-server 99.99.99.11

dm1sw-adm0(config)# end

Finally Verify the changes

dm1sw-adm0# show running-config

Save the configuration.

dm1sw-adm0# copy running-config startup-config

[########################################] 100%

Copy complete, now saving to disk (please wait)...

Copy complete.

Exit the session

dm1sw-adm0# exit


-- Cisco (ethernet) switch NTP ---  

ssh admin@99.99.99.11 

dm1sw-adm0# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

dm1sw-adm0(config)# no ntp server 88.88.88.250 

dm1sw-adm0(config)# end

dm1sw-adm0# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

dm1sw-adm0(config)# ntp server 99.99.99.250 prefer

dm1sw-adm0(config)# ntp server 99.99.99.251

dm1sw-adm0(config)# end

Finally Verify the changes

dm1sw-adm0# show running-config

Save the configuration.

dm1sw-adm0# copy running-config startup-config

[########################################] 100%

Copy complete, now saving to disk (please wait)...

Copy complete.

Exit the session

dm1sw-adm0# exit

--- DB nodes DNS ---

1. Log in to the database server as the root user.

Edit the /etc/resolv.conf file.


2. Set the DNS server and domain name using an editor such as vi. There should be a name server line for each DNS server.

search        example.com

nameserver 99.99.99.10

nameserver 99.99.99.11


3.Set the DNS server in the server ILOM.

  ssh root@99.99.99.6/7

  -> show /SP/clients/dns

 /SP/clients/dns

    Targets:

    Properties:

        auto_dns = enabled

        nameserver = 88.88.88.10

        retries = 1

        searchpath = mlspp.gov.az

        timeout = 5

    Commands:

        cd

        set

        show

-> set /SP/clients/dns nameserver=99.99.99.10,99.99.99.11

Set 'nameserver' to '99.99.99.10,99.99.99.11' [99.99.99.10, 99.99.99.11]

--- DB nodes NTP ---

1. Stop the NTP/Chrony services on each database server.

[root@dm1db1 ~]# systemctl stop chronyd.service


2. Update the ntp.conf/chrony.conf file with the IP address of the new NTP server.

Start the NTP/Chrony services on the database server.

[root@dm1db1 ~]# vi /etc/chrony.conf

[root@dm1db1 ~]# systemctl start chronyd.service


3.Set the DNS server in the server ILOM for each node.

  ssh root@99.99.99.6/7

-> set /SP/clients/ntp/server/1 address=99.99.99.250

Set 'address' to '99.99.99.250'

-> set /SP/clients/ntp/server/2 address=99.99.99.251

Set 'address' to '99.99.99.251'

-> show /SP/clients/ntp/server/1

/SP/clients/ntp/server/1

Targets:

Properties:

address = 99.99.99.250

->  show /SP/clock

/SP/clock

Targets:

Properties:

datetime = Mon Jul 26 23:03:43 2021

timezone = +04 (Asia/Baku)

uptime = 15 days, 06:27:05

usentpserver = enabled


  

--- Cell nodes DNS and NTP---

1. Log in to the Oracle Exadata Storage Server as the root user. (each cell separately and respectively)

2. Specify a time interval to repair the disk and bring it back online.

   The default DISK_REPAIR_TIME attribute value of 3.6 hours should be long enough for most environments.

    a. Check the repair time for all mounted disk groups.

       Log in to the Oracle ASM instance and run the following query:

SQL> SELECT dg.name,a.value FROM v$asm_diskgroup dg, v$asm_attribute a WHERE dg.group_number=a.group_number AND a.name='disk_repair_time';

    b. Adjust the DISK_REPAIR_TIME parameter, if needed.

   In the following command, h.n is the amount of time in hours, such as 4.6.

SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'DISK_REPAIR_TIME'='h.nH';

3. Check that putting the grid disks offline will not cause a problem for Oracle ASM.

   [root@db1cell2 ~]# cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome

DATAC1_CD_00_db1cell2 ONLINE Yes

DATAC1_CD_01_db1cell2 ONLINE Yes

DATAC1_CD_02_db1cell2 ONLINE Yes

DATAC1_CD_03_db1cell2 ONLINE Yes

DATAC1_CD_04_db1cell2 ONLINE Yes

DATAC1_CD_05_db1cell2 ONLINE Yes

RECOC1_CD_00_db1cell2 ONLINE Yes

RECOC1_CD_01_db1cell2 ONLINE Yes

RECOC1_CD_02_db1cell2 ONLINE Yes

RECOC1_CD_03_db1cell2 ONLINE Yes

RECOC1_CD_04_db1cell2 ONLINE Yes

RECOC1_CD_05_db1cell2 ONLINE Yes


   The value Yes should be returned for the grid disks. If one or more disks does not return a Yes value, then restore data redundancy for the disk group, and repeat the command until all disk groups return a Yes value.  

4. Inactivate all grid disks on the cell.

   [root@db1cell2 ~]# cellcli -e alter griddisk all inactive

GridDisk DATAC1_CD_00_db1cell2 successfully altered

GridDisk DATAC1_CD_01_db1cell2 successfully altered

GridDisk DATAC1_CD_02_db1cell2 successfully altered

GridDisk DATAC1_CD_03_db1cell2 successfully altered

GridDisk DATAC1_CD_04_db1cell2 successfully altered

GridDisk DATAC1_CD_05_db1cell2 successfully altered

GridDisk RECOC1_CD_00_db1cell2 successfully altered

GridDisk RECOC1_CD_01_db1cell2 successfully altered

GridDisk RECOC1_CD_02_db1cell2 successfully altered

GridDisk RECOC1_CD_03_db1cell2 successfully altered

GridDisk RECOC1_CD_04_db1cell2 successfully altered

GridDisk RECOC1_CD_05_db1cell2 successfully altered

   This command may take more than 10 minutes to complete. Inactivating the grid disks automatically sets the disk to offline in the Oracle ASM instance.


5. Confirm the grid disks are offline.

   a. Check the status of the grid disks.

       [root@db1cell2 ~]# cellcli -e list griddisk attributes name, asmmodestatus,asmdeactivationoutcome

DATAC1_CD_00_db1cell2 OFFLINE Yes

DATAC1_CD_01_db1cell2 OFFLINE Yes

DATAC1_CD_02_db1cell2 OFFLINE Yes

DATAC1_CD_03_db1cell2 OFFLINE Yes

DATAC1_CD_04_db1cell2 OFFLINE Yes

DATAC1_CD_05_db1cell2 OFFLINE Yes

RECOC1_CD_00_db1cell2 OFFLINE Yes

RECOC1_CD_01_db1cell2 OFFLINE Yes

RECOC1_CD_02_db1cell2 OFFLINE Yes

RECOC1_CD_03_db1cell2 OFFLINE Yes

RECOC1_CD_04_db1cell2 OFFLINE Yes

RECOC1_CD_05_db1cell2 OFFLINE Yes

      The output should show asmmodestatus=OFFLINE or asmmodestatus=UNUSED, and asmdeactivationoutcome=Yes for all grid disks.

   b. List the grid disk to confirm that they are offline.

      [root@db1cell2 ~]# cellcli -e list griddisk

DATAC1_CD_00_db1cell2 inactive

DATAC1_CD_01_db1cell2 inactive

DATAC1_CD_02_db1cell2 inactive

DATAC1_CD_03_db1cell2 inactive

DATAC1_CD_04_db1cell2 inactive

DATAC1_CD_05_db1cell2 inactive

RECOC1_CD_00_db1cell2 inactive

RECOC1_CD_01_db1cell2 inactive

RECOC1_CD_02_db1cell2 inactive

RECOC1_CD_03_db1cell2 inactive

RECOC1_CD_04_db1cell2 inactive

RECOC1_CD_05_db1cell2 inactive


6. Shut down the cell services and ocrvottargetd service.

    [root@db1cell2 ~]# cellcli -e alter cell shutdown services all

Stopping the RS, CELLSRV, and MS services...

The SHUTDOWN of services was successful.

Note: The ocrvottargetd service is not included in some releases. [service ocrvottargetd stop]

7. Use the ipconf utility to change the DNS settings.

[root@db1cell2 ~]# /usr/local/bin/ipconf

[Info]: ipconf command line: /opt/oracle.cellos/ipconf.pl -nocodes

Logging started to /var/log/cellos/ipconf.log

Interface ib0   is                      Linked.    hca: mlx4_0

Interface ib1   is                      Linked.    hca: mlx4_0

Interface eth0  is                      Linked.    driver/mac: igb/00:10:e0:eb:28:6e


The current nameserver(s): 88.88.88.10 88.88.88.11

Do you want to change it (y/n) [n]: y

Nameserver: 99.99.99.10

Add more nameservers (y/n) [n]: y

Nameserver: 99.99.99.11

Add more nameservers (y/n) [n]: n

The current timezone: Asia/Baku

Do you want to change it (y/n) [n]: n

The current NTP server(s): 88.88.88.250

Do you want to change it (y/n) [n]: y

Fully qualified hostname or ip address for NTP server. Press enter if none: 99.99.99.250

Continue adding more ntp servers (y/n) [n]: y

Fully qualified hostname or ip address for NTP server. Press enter if none: 99.99.99.251

Continue adding more ntp servers (y/n) [n]: n


Network interfaces

Name  State  Speed    Status  IP address   Netmask       Gateway       Net type   Hostname

ib0   Linked          UP      77.77.77.7 255.255.252.0               Private    db1cell2-priv1.mlspp.gov.az

ib1   Linked          UP      77.77.77.8 255.255.252.0               Private    db1cell2-priv2.mlspp.gov.az

eth0  Linked default  UP      99.99.99.4  255.255.255.0 99.99.99.253 Management db1cell2.mlspp.gov.az

Select interface name to configure or press Enter to continue: 


Select canonical hostname from the list below

1: db1cell2-priv1.mlspp.gov.az

2: db1cell2-priv2.mlspp.gov.az

3: db1cell2.mlspp.gov.az

Canonical fully qualified domain name [3]: 


Select default gateway interface from the list below

1: eth0

Default gateway interface [1]: 


Canonical hostname: db1cell2.mlspp.gov.az

Nameservers: 99.99.99.10 99.99.99.11

Timezone: Asia/Baku

NTP servers: 99.99.99.250 99.99.99.251

Default gateway device: eth0

Network interfaces

Name  State  Speed    Status  IP address   Netmask       Gateway       Net type   Hostname

ib0   Linked          UP      77.77.77.7 255.255.252.0               Private    db1cell2-priv1.mlspp.gov.az

ib1   Linked          UP      77.77.77.8 255.255.252.0               Private    db1cell2-priv2.mlspp.gov.az

eth0  Linked default  UP      99.99.99.4  255.255.255.0 99.99.99.253 Management db1cell2.mlspp.gov.az

Is this correct (y/n) [y]: y


Do you want to configure basic ILOM settings (y/n) [y]: y

Loading basic configuration settings from ILOM ...

ILOM Fully qualified hostname [db1cell2-ilom.mlspp.gov.az]: 

Inet protocol (IPv4,IPv6) [IPv4]: 

ILOM IP address [99.99.99.9]: 

ILOM Netmask [255.255.255.0]: 

ILOM Gateway or none [99.99.99.253]: 

ILOM Nameserver (multiple IPs separated by a comma) or none [88.88.88.10]: 99.99.99.10

ILOM Use NTP Servers (enabled/disabled) [enabled]: 

ILOM First NTP server. Fully qualified hostname or ip address or none [88.88.88.250]: 99.99.99.250

ILOM Second NTP server. Fully qualified hostname or ip address or none [none]: 99.99.99.251

ILOM Vlan id or zero for non-tagged VLAN (0-4079) [0]: 


Basic ILOM configuration settings:

Hostname             : db1cell2-ilom.mlspp.gov.az

IP Address           : 99.99.99.9

Netmask              : 255.255.255.0

Gateway              : 99.99.99.253

DNS servers          : 99.99.99.10

Use NTP servers      : enabled

First NTP server     : 99.99.99.250

Second NTP server    : 99.99.99.251

Timezone (read-only) : Asia/Baku

VLAN id              : 0

Is this correct (y/n) [y]: n

ILOM Fully qualified hostname [db1cell2-ilom.mlspp.gov.az]: 

Inet protocol (IPv4,IPv6) [IPv4]: 

ILOM IP address [99.99.99.9]: 

ILOM Netmask [255.255.255.0]: 

ILOM Gateway or none [99.99.99.253]: 

ILOM Nameserver (multiple IPs separated by a comma) or none [88.88.88.10]: 99.99.99.10,99.99.99.11

ILOM Use NTP Servers (enabled/disabled) [enabled]: 

ILOM First NTP server. Fully qualified hostname or ip address or none [88.88.88.250]: 99.99.99.250

ILOM Second NTP server. Fully qualified hostname or ip address or none [none]: 99.99.99.251

ILOM Vlan id or zero for non-tagged VLAN (0-4079) [0]: 


Basic ILOM configuration settings:

Hostname             : db1cell2-ilom.mlspp.gov.az

IP Address           : 99.99.99.9

Netmask              : 255.255.255.0

Gateway              : 99.99.99.253

DNS servers          : 99.99.99.10,99.99.99.11

Use NTP servers      : enabled

First NTP server     : 99.99.99.250

Second NTP server    : 99.99.99.251

Timezone (read-only) : Asia/Baku

VLAN id              : 0

Is this correct (y/n) [y]: y


[Info]: Run /opt/oracle.cellos/validations/init.d/saveconfig

[Info]: Custom changes have been detected in /etc/resolv.conf

[Info]: Original file /etc/resolv.conf will be saved in /etc/resolv.conf.backupbyExadata

[Info]: Stopping cellwall service ...

[Info]: cellwall service stopped

[Info]: Restart chronyd service

[Info]: Starting cellwall service ...

[Info]: cellwall service started

[Info]: Save /etc/sysctl.conf in /etc/sysctl.conf.backupbyExadata

[Info]: Adjust settings for IB interfaces in /etc/sysctl.conf

[Info]: Retarting cellwall service ...

active

[Info]: cellwall service restarted

Re-login using new IP address 99.99.99.4 if you were disconnected after following commands

ip addr show eth0

sleep 4


[Warning]: You modified NTP server.

Ensure you also update the Infiniband Switch NTP server

if the same NTP server was also used by the Infiniband switch.



[Warning]: You modified DNS name server.

Ensure you also update the Infiniband Switch DNS server

if the same DNS server was also used by the Infiniband switch.


8. Restart the cell services and ocrvottargetd service.

   [root@db1cell2 ~]# cellcli -e alter cell startup services all

Starting the RS, CELLSRV, and MS services...

Getting the state of RS services...  running

Starting CELLSRV services...

The STARTUP of CELLSRV services was successful.

Starting MS services...

The STARTUP of MS services was successful.


   The server does not need to reboot.


9. Activate the grid disks when the cell comes online.

[root@db1cell2 ~]# cellcli -e alter griddisk all active

GridDisk DATAC1_CD_00_db1cell2 successfully altered

GridDisk DATAC1_CD_01_db1cell2 successfully altered

GridDisk DATAC1_CD_02_db1cell2 successfully altered

GridDisk DATAC1_CD_03_db1cell2 successfully altered

GridDisk DATAC1_CD_04_db1cell2 successfully altered

GridDisk DATAC1_CD_05_db1cell2 successfully altered

GridDisk RECOC1_CD_00_db1cell2 successfully altered

GridDisk RECOC1_CD_01_db1cell2 successfully altered

GridDisk RECOC1_CD_02_db1cell2 successfully altered

GridDisk RECOC1_CD_03_db1cell2 successfully altered

GridDisk RECOC1_CD_04_db1cell2 successfully altered

GridDisk RECOC1_CD_05_db1cell2 successfully altered

10. Verify the disks are active.

[root@db1cell2 ~]# cellcli -e list griddisk

DATAC1_CD_00_db1cell2 active

DATAC1_CD_01_db1cell2 active

DATAC1_CD_02_db1cell2 active

DATAC1_CD_03_db1cell2 active

DATAC1_CD_04_db1cell2 active

DATAC1_CD_05_db1cell2 active

RECOC1_CD_00_db1cell2 active

RECOC1_CD_01_db1cell2 active

RECOC1_CD_02_db1cell2 active

RECOC1_CD_03_db1cell2 active

RECOC1_CD_04_db1cell2 active

RECOC1_CD_05_db1cell2 active

The output should show active.

11. Verify the grid disk status.

    a. Check that all grid disks are online.

   cellcli -e list griddisk attributes name, asmmodestatus

b. Wait for Oracle ASM synchronization to complete for all grid disks. Each disk will go to a SYNCING state first then ONLINE.

   [root@db1cell2 ~]# cellcli -e list griddisk attributes name, asmmodestatus

DATAC1_CD_00_db1cell2 ONLINE

DATAC1_CD_01_db1cell2 SYNCING

DATAC1_CD_02_db1cell2 SYNCING

DATAC1_CD_03_db1cell2 SYNCING

DATAC1_CD_04_db1cell2 SYNCING

DATAC1_CD_05_db1cell2 SYNCING

RECOC1_CD_00_db1cell2 ONLINE

RECOC1_CD_01_db1cell2 ONLINE

RECOC1_CD_02_db1cell2 SYNCING

RECOC1_CD_03_db1cell2 SYNCING

RECOC1_CD_04_db1cell2 SYNCING

RECOC1_CD_05_db1cell2 SYNCING


   Oracle ASM synchronization is complete when all grid disks show asmmodestatus=ONLINE.

   

12. Repeat this procedure for each Oracle Exadata Storage Server.



references: https://docs.oracle.com/cd/E80920_01/DBMMN/maintaining-exadata-components.htm#DBMMN22932

            Changing IP addresses on Exadata Database Machine (Doc ID 1317159.1) 

No comments:

Post a Comment