Oracle RAC Scan and resolv.conf issue

Oracle RAC evicted. SCAN no longer working. At first, everything look sporadic. Then, it seem to have something to do with network bounced.


Things to check/verify and double check in case someone changed something.

/etc/hosts
/etc/hostname or hostname
/etc/resolv.conf
/etc/named.conf
/var/named/mhrac1.zone (the zone file)
 /etc/sysconfig/network-scripts   -- check all the eth0, eth1, eth2 files here to have a good picture of how the networking looks like in this setup

Long story short -
When running an nslookup check, I was getting the following. I have disabled the firewall for the test, so port 53 should not be blocked. I can ping freely in and out of the node yet I kept getting the following. All configured IPs ping-ed fine.

nslookup dnsrac-scan.wwdom.com

Server:          10.11.188.211
Address:     10.11.188.211#53

** server can't find mhrac1-scan.wwdom.com: NXDOMAIN

Upon checking my resolv.conf, the following did not look right at all. I should have a "search" and the bottom 2 IPs addresses I set up in my mhrac1.zone file, instead, I am having the following

Generated by NetworkManager
search wwdom.com
nameserver 10.11.188.211
nameserver 10.11.188.212
nameserver 10.16.169.90
# NOTE: the libc resolver may not support more than 3 nameservers.
# The nameservers listed below may not be recognized.
nameserver 10.10.1.110

nameserver 75.75.75.75

After fixing the extra 3 IPs from the top, I was able to perform an expected nslookup. I was also able to replicate the issue by restarting the network - "service network restart" and new IPs get inserted every time I did that.

This could be due to when network restarted, it appended other IP addresses on top and the intended ones did not get read from - as the message already hinted.

My guess to permanently preventing from this issue, I need to change the PEERDNS in the  ifcfg-eth0 so, when network restarted, the VM will not reach outside and grab the real domain DNS IPs.  (By the way, the  ifcfg-eth0 is a make up name. Centos7 will provide a random name by default. So, keep track of you MAC address and assigned IPs. Default might look like this  ifcfg-234133445. the same goes for eth1 and eth2.