Is there a way to force heartbeat to add new ip addresses to the system without a full restart?

Question

We utilize heartbeat for High Availability. I'd like to add an additional ip address to the heartbeat cluster, but I don't want to do a full restart of the cluster in the process. Is there a signal I can send to heartbeat that would prompt it to re-parse the "haresources" file and action upon it? heartbeat -r does not appear to do the trick.

Answer 1

The problem was that I didn't wait long enough after executing "heartbeat -r" (the command that is executed in the init.d script when you run "service heartbeat reload".) After a few minutes, the IP showed up on the interface as expected.

Answer 2

You don't need to reload Heartbeat at all. Simply add the new IPaddr resource to your haresources file, something like this

IPaddr::xx.xx.xx.xx

and then start it

/etc/ha.d/resource.d/IPaddr xx.xx.xx.xx start

Of course, you should make sure to issue the IPaddr start on the active node. You should now be able to send and receive traffic on the just added IP address.

Answer 3

Hearbeat only has to be restarted on the secondary machine, hence avoiding any downtime related to resource management.

In this case, the primary node detects that the slave machine is 'dead' and forces a 'failover' which reloads the resources file and start the missing resources.

The logs are quite explicit when doing this:

May  9 12:10:40 gw2 heartbeat: [3684]: info: Received shutdown notice from 'gw1'.
May  9 12:10:40 gw2 heartbeat: [3684]: info: Resources being acquired from gw1.
May  9 12:10:40 gw2 heartbeat: [26469]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
May  9 12:10:40 gw2 harc[26469]: info: Running /etc/ha.d//rc.d/status status
May  9 12:10:40 gw2 mach_down[26521]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
May  9 12:10:40 gw2 mach_down[26521]: info: mach_down takeover complete for node gw1.
May  9 12:10:40 gw2 heartbeat: [3684]: info: mach_down takeover complete.
May  9 12:10:40 gw2 heartbeat: [3684]: debug: StartNextRemoteRscReq(): child count 1
May  9 12:10:40 gw2 IPaddr2[26520]: INFO:  Running OK
May  9 12:10:40 gw2 IPaddr2[26640]: INFO:  Running OK
May  9 12:10:40 gw2 IPaddr2[26725]: INFO:  Running OK
May  9 12:10:40 gw2 IPaddr2[26805]: INFO:  Running OK
May  9 12:10:40 gw2 IPaddr2[26890]: INFO:  Resource is stopped
May  9 12:10:40 gw2 heartbeat: [26470]: info: Local Resource acquisition completed.
May  9 12:10:40 gw2 heartbeat: [3684]: debug: StartNextRemoteRscReq(): child count 1
May  9 12:10:40 gw2 heartbeat: [26953]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
May  9 12:10:40 gw2 harc[26953]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
May  9 12:10:40 gw2 ip-request-resp[26953]: received ip-request-resp IPaddr2::1.2.3.4 OK yes
May  9 12:10:40 gw2 ResourceManager[26976]: info: Acquiring resource group: gw2 IPaddr2::1.2.3.4
May  9 12:10:40 gw2 IPaddr2[27006]: INFO:  Resource is stopped
May  9 12:10:40 gw2 ResourceManager[26976]: info: Running /etc/ha.d/resource.d/IPaddr2 1.2.3.4 start
May  9 12:10:40 gw2 IPaddr2[27115]: INFO: ip -f inet addr add 1.2.3.4/24 brd 1.2.3.255 dev brwan
May  9 12:10:40 gw2 IPaddr2[27115]: INFO: ip link set brwan up
May  9 12:10:40 gw2 IPaddr2[27115]: INFO: /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-1.2.3.4 brwan 1.2.3.4 auto not_used not_used
May  9 12:10:40 gw2 IPaddr2[27091]: INFO:  Success

May  9 12:10:47 gw2 heartbeat: [3684]: WARN: node gw1: is dead
May  9 12:10:47 gw2 heartbeat: [3684]: info: Dead node gw1 gave up resources.
May  9 12:10:47 gw2 heartbeat: [3684]: info: Link gw1:eth0 dead.

May  9 12:10:59 gw2 heartbeat: [3684]: info: Heartbeat restart on node gw1
May  9 12:10:59 gw2 heartbeat: [3684]: info: Link gw1:eth0 up.
May  9 12:10:59 gw2 heartbeat: [3684]: info: Status update for node gw1: status init
May  9 12:10:59 gw2 heartbeat: [3684]: info: Status update for node gw1: status up
May  9 12:10:59 gw2 heartbeat: [3684]: debug: StartNextRemoteRscReq(): child count 1
May  9 12:10:59 gw2 heartbeat: [28604]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
May  9 12:10:59 gw2 heartbeat: [3684]: debug: get_delnodelist: delnodelist= 
May  9 12:10:59 gw2 harc[28604]: info: Running /etc/ha.d//rc.d/status status
May  9 12:10:59 gw2 heartbeat: [3684]: info: Status update for node gw1: status active
May  9 12:10:59 gw2 heartbeat: [3684]: debug: StartNextRemoteRscReq(): child count 1
May  9 12:10:59 gw2 heartbeat: [28619]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
May  9 12:10:59 gw2 harc[28619]: info: Running /etc/ha.d//rc.d/status status
May  9 12:10:59 gw2 heartbeat: [28634]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
May  9 12:10:59 gw2 harc[28634]: info: Running /etc/ha.d//rc.d/status status
May  9 12:11:00 gw2 heartbeat: [3684]: info: remote resource transition completed.