Merce services failover setup.

Submitted by srinivas on Sat, 2010-04-17 18:32

Merce Service Failover setup

Merce failover is used to backup of services and data from one Merce(Primary) server to another Merce(Backup) server and in-case of any problem with a Merce(Primary) server then the Merce(Backup) server will take charge and run those services which was running on Merce(Primary) server.

To setup Merce failover we required 2 server with identical setup and all services should run on virtual IP. I will take Supreme Merce servers as an example. In each Supreme location we setup 2 Merce servers, 1-File sever and 2-Mail server for that particular location.

In Supreme :-

Primary server act as file, dhcpd, DNS slave and compute server for this smb, dhcpd, cups, XDMCP, iptables and named services will run. Installed Merce in GUI mode and installed applications like OpenOffice, Firefox, Thunderbird, Acrobat, SAP client, DOSEMU and configured printers and XDMCP on this server.

Secondary server act as mail, proxy, webmail and DNS slave for this amavisd, sendmail, sendmail-rx, cyrus-imapd, mfilter, squid, merce-webguard, httpd, iptables and named services will run. Installed Merce in GUI mode, and installed applications like OpenOffice, Firefox, Thunderbird, Acrobat, SAP client, DOSEMU and configured XDMCP on this server too.

Both the servers having same partition table(this is not mandatory). Existing 2 main data partitions are like this.

/cyrus --> In this partition /var/spool/imap & /var/lib/imap data will store here.

/var/usr --> In this partition /var/home & /va/win/data data will store here.

Backup Method :-

Cross backup is running between the 2 servers. From mail server data will be backup on file server /cyrus partition and from file server data will be backup on mail server /var/usr partition.

Failover Setup:-

Note :- Here I will refer Primary server as SERVER-1 and Secondary server as SERVER-2.

Setting failover on SERVER-1(File server).

Part - I

Copy failover scripts failover-final.tar on SERVER-1:/var/warehouse.

	# cd /var/warehouse
	# mkdir failover
	# cd failover
	# tar -xvf /var/warehouse/failover-final.tar

        Three tgz files will get untar.
	# ls
	# file.tgz mail.tgz hbeat.tgz

We required file.tgz and hbeat.tgz for file server.

        Extract file.tgz
	# tar -zxvf file.tgz (This tgz file contain 2 folder ie "etc" and "opt")
	# ls
	# etc opt
	# cd etc/merce (In "etc/merce" 2 folder will be there ie "failover" and "xbackup")
	# ls 
	# failover xbackup
	# cd failover (In failover directory you will get remote server folder)
	# ls
	# SERVER-2.example.com (SERVER-2 is remote server, folder name should be FQDN of remote server)

Now go to SERVER-1 /etc/merce folder.

	# cd /etc/merce
	# mkdir failover

        create a folder with the name of remote server ie.Server-2.example.com(full FQDN name)
	# mkdir server-2.example.com

Copy failover service configuration folders from /var/warehouse/failover/etc/merce/failover/server-2.example.com

	# cd server-2.example.com
	# cp /var/warehouse/failover/etc/merce/failover/server-2.example.com/* .
	# ls
	# 00-xbackup 08-iptables 13-named 56-cups 65-dhcpd 91-smb 99-gdm (These are the service configuration folders)

In each of these folder contain backup.conf file, bringup.conf file, data folder and saved folder.

In backup.conf file we have to specify the path of the service.(ex: path /etc/sysconfig/iptables for iptables).

In bringup.conf file contain the service name which will bringup the service after failover.(ex: servicerestart iptables and addtostartup iptables

Data folder contain the service data with full path structure.

Saved folder will not contain any thing. But this will use when failover takes place, all old configuration files will save in this folder in tgz format.

        Go to /etc/merce/
	# cd /etc/merce

        Creat a folder called xbackup
	# mkdir xbackup

Copy xbackup service configuration files from /var/warehouse/failover/etc/merce/xbackup to /etc/merce/xbackup

	# cd xbackup
	# cp /var/warehouse/failover/etc/merce/xbackup/* .
	# ls
	# global xbackup-cups.conf xbackup-dhcpd.conf xbackup-gdm.conf xbackup-iptables.conf
          xbackup-named.conf xbackup-smb.conf xbackup-xbackup.conf

These files contain the source and destination path of configuration files data. Note:- The data and configuration path should be correct other wise it create new problems.

In global you have go specify FQDN of remote server and local server names.

        Go to /opt/merce/
	# cd /opt/merce/

Copy /var/warehouse/failover/opt/merce/failover folder in /opt/merce/

	# cp -a /var/warehouse/failover/opt/merce/failover  .

This failover folder contain backup and bringup scripts, utils and doc folders.

Backup script will backup the data.

Bringup script will bringup services on backup server.

Utils folder contain gdm-restart, mail-gendb and reconstruct-mboxes script which will run according the options set in backup.conf & bringup.conf.

Doc folder contains documentation.

        Go to /opt/merce/lib/perl/
	# cd /opt/merce/lib/perl/

Copy /var/warehouse/failover/opt/merce/lib/perl/Failover.pm file in /opt/merce/lib/perl

	# cp /var/warehouse/failover/opt/merce/lib/perl/Failover.pm .

Now login to the remote server as root.

	# ssh root@SERVER-2.example.com

Goto /var/usr/ where we can copy the SERVER-1 services data(Here I am using /var/usr to copy the data).

	# cd /var/usr
	# mkdir -p merce/failover
	# cd merce/failover

Create a folder with the name of SERVER-1.example.com(FQDN of remote server).

	# mkdir server-1.example.com

Logout from remote server and login to SERVER-1.

        # ssh root@SERVER-1.example.com

Set the cron job for backup the services configuration files & data.

	# crontab -l
	# crontab -e
	# * * * * * /opt/merce/failover/backup SERVER-2.example.com >/dev/null 2>&1

Now one part of failover setup is over on SERVER-1 i.e copying the data and services configuration files.

Part – II

        # ssh root@SERVER-1.example.com

Go to /var/warehouse/failover

	# cd /var/warehouse/failover
	# ls
	# file.tgz hbeat.tgz
        Extract hbeat.tgz

	# tar -zxvf hbeat.tgz (These "hbeat.tgz" file contain "etc", "opt" and "var")
	# ls 
	# etc opt var

Go to /etc/merce

	# cd /etc/merce

Copy /var/warehouse/failover/etc/merce/hbeat.cf file to /etc/merce

	# cp /var/warehouse/failover/etc/merce/hbeat.cf .

Edit this file and make following changes.

	# vim hbeat.cf
	
REMOTE=10.3.0.6  --> Add the SERVER-2.example.com physical IP. 
REMOTE_VIRT=10.3.0.9 --> Add the SERVER-2.example.com virtual IP.
REMOTE_VIRT_MASK="255.255.0.0"  --> Add the SERVER-2.example.com subnet.
REMOTE_VIRT_IFACE="eth0:2"
 HBEAT_MAX_RETRY_COUNT=3

# Must be physical name of the remote system-- used by merce to sent Qfile
REMOTE_MERCE_HOST=FQDN name of SERVER-2 
example :- REMOTE_MERCE_HOST=talung.supremegroup.co.in

ARPING=/sbin/arping
ARP_SRC_HOST="$REMOTE_VIRT"
ARP_DEST_HOST=10.3.0.1 --> Gateway of SERVER-2.example.com

Save and close this file.

Now go to /etc/init.d

 
	# cd /etc/init.d

Take the backup of network file.

	# mkdir RCS
	# ls -l network
	# -rwxr-xr-x 1 root root 8199 Apr 12 15:11 network
	# ci -l network
	comment : Backup before changing the network file with failover network file.

Copy /var/warehouse/failover/etc/init.d/network file to /etc/init.d/ overwrite network file.

	# cp /var/warehouse/failover/etc/init.d/network .

Edit this file and make following changes.

	# vim network
        search setvirtip the following line will found.

       /opt/merce/sbin/setvirtip "10.3.0.8" "255.255.0.0" "eth0:1" "10.3.0.1"

       Here replace “10.3.0.8  255.255.0.0” to SERVER-1 virtual IP and subnet. 
       Replace “eth0:1”  with SERVER-1 virtual IP interface.
       Replace “10.3.0.1” with SERVER-1 gateway.

       Save and close.

Note:Move virtual IP config file from SERVER-1 i.e "/etc/sysconfig/network-scripts/ifcfg-eth0:1" to /var/warehouse and delete this file once every thing works fine.

	# cd /etc/sysconfig/network-scripts/
	# mv ifcfg-eth0:1 /var/warehouse

Go to /opt/merce/sbin

	# cd /opt/merce/sbin

Copy /var/warehouse/failover/opt/merce/sbin/hbeat.sh & setvirtip script to /opt/merce/sbin

	# cp /var/warehouse/failover/opt/merce/sbin/hbeat.sh  .
	# cp /var/warehouse/failover/opt/merce/sbin/setvirtip  .

Edit setvirtip and make the following changes.

	# vim setvirtip
	
        Search for ARP_SRC_HOST="$MY_VIRT_IP"
        Replace “$MY_VIRT_IP” with “$1”

        Save and close.

Create a folder called hbeat in /var/merce/var/data.

	# cd /var/merce/var/data/
	# mkdir hbeat

Now restart the network

	# /etc/init.d/network restart

This is assign virtual IP for SERVER-1. And hbeat will start working.

Follow the same steps to setup failover on SERVER-2.example.com(mail server). Here you have to configure one more script which is imap-sync.

Failover on mail server. (SERVER-2.example.com)

Login to SERVER-2.example.com (mail server).

Part – I & Part – II will be same as file server configuration. In PART – I services will be different and rest will be same as file server. In Part – II configuration will be same but you need to change the remote server FQDN and IP address and also local server virtual IP.

Add xbackup in crontab as below.

	# crontab -l
	# crontab -e
	# * * * * * /opt/merce/failover/backup SERVER-1.example.com >/dev/null 2>&1

Now the differenet part is imap-sync, configure it on mail server. It will synchronize mails data from mail server to file server.

Copy imap-sync.tgz file on SERVER-2:/var/warehouse.

	# cd /var/warehouse
	# mkdir imap-sync
	# cd imap-sync
	# tar -zxvf  /var/warehouse/imap-sync.tgz (This file contain "etc" and "opt" folders)
	# ls
	# etc opt

Goto /etc/merce

	# cd /etc/merce

Copy imapsync folder from /var/warehouse/imap-sync/etc/merce/imapsyn to /etc/merce

	# cp -a /var/warehouse/imap-sync/etc/merce/imapsync .
	# chown root.root imapsync
	# chmod 100 imapsync
	# cd imapsync  (In imapsync folder 2 files will be there with the FQDN names of servers)
	# ls
	# SERVER-1.example.com SERVER-2.example.com
	# chown root.root SERVER-1.example.com SERVER-2.example.com
	# chmod 400 SERVER-1.example.com SERVER-2.example.com

Edit these 2 files and add cyrus admin password in each file. In SERVER-1.example.com file add cyrus-admin password of SERVER-1.example.com, and in SERVER-2.example.com file add cyrus-admin password of SERVER-2.example.com.

Take cyrus admin password from /etc/merce/Siteconfig.pm file.

Go to /etc take the backup and edit cyrus.conf file.

	# cd /etc
	# ci -l cyrus.conf
	Backup before configuring IMAPS service.
	# vim cyrus.conf
        Search for imaps in cyrus.conf you will get the following line.

        #  imaps         cmd="imapd -s" listen="imaps" prefork=0

        Remove the '#' in front of imaps.

In iptables open port 993 for IMAPS and restart iptable service to access IMAPSremotely.

        # cd /etc/sysconfig
        # vim iptables
        -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 993 -j ACCEPT
        
        save and close it.
        # /etc/init.d/iptables restart

Now create new cyrus-imapd.pem certificate. Go to /etc/pki/tls/certs.

	# cd /etc/pki/tls/certs
	# mkdir old

Move existing cyrus-imapd.pem to old folder.

	# mv cyrus-imapd.pem old

Create new cyrus-imapd certificate.

	# make cyrus-imapd.pem

Fill the required information while creating the certificate. After this change the permission of cyrus-imapd.pem

	# chown root.mail cyrus-imapd.pem
	# chmod 640 cyrus-inapd.pem

Now edit /etc/imapd.conf file the and change the parameters as below.

	# cd /etc/
	# ci -l imapd.conf
	# vim imapd.conf

Search for tls_cert_file: you will get 3 lines, replace the path of these lines as below.

	tls_cert_file: /etc/pki/tls/certs/cyrus-imapd.pem
	tls_key_file: /etc/pki/tls/certs/cyrus-imapd.pem
	tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt

Again search for proxyserver word and replace it as below.

	proxyserver: cyrus

Goto /opt/merce/sbin

	# cd /opt/merce/sbin

Copy /var/warehouse/imap-sync/opt/merce/sbin/imap-sync file to /opt/merce/sbin

	# cp /var/warehouse/imap-sync/opt/merce/sbin/imap-sync .

Set the cronjob for imap-sync

	# crontab -l
	# crontab -e
        * * * * * /opt/merce/sbin/imap-sync SERVER-2.example.com SERVER-2.example.com >/dev/null 2>&1

Now login as root on Remote server. i.e SERVER-1.example.com(file server).

	# ssh root@SERVER-1.example.com

Copy imap-sync.tgz file from SERVER-2.example.com:/var/warehouse/imap-sync.tgz to /var/warehouse.

	# scp root@SERVER-2.example.com:/var/warehouse/imap-sync.tgz  /var/warehouse
	# cd /var/warehouse

Follow the same steps to configure imap-sync as configured on SERVER-2.example.com

Restoring the failover services.

Restoring failover services on file server:-

Stop the following mail server service.

# /etc/init.d/sendmail stop 
# /etc/init.d/sendmail-rx stop
# /etc/init.d/amavisd stop
# /etc/init.d/mfilter stop
# /etc/init.d/httpd stop
# /etc/init.d/squid stop
# /etc/init.d/merce-webguard stop

Now remove this services from runleve 3 & 5.

# chkconfig –level 35 sendmail off
# chkconfig –level 35 sendmail-rx off
# chkconfig –level 35 amavisd off
# chkconfig –level 35 mfilter off
# chkconfig –level 35 httpd off
# chkconfig –level 35 squid off
# chkconfig –level 35 merce-webguard off

Remove “taken-over.flag” file from “/var/merce/var/data/hbeat”.

Now restart the network service on server.

# /etc/init.d/network restart

This will remove the remote server virtual IP, and hbeat will start again. Also at same time start/restart the remote server so it will work as before.

Restoring failover services on mail server:-

Stop the following mail server service.

# /etc/init.d/smd stop 
# /etc/init.d/cups stop
# /etc/init.d/dhcpd stop

Now remove this services from runleve 3 & 5.

# chkconfig –level 35 smb off
# chkconfig –level 35 cups off
# chkconfig –level 35 dhcpd off

Remove “taken-over.flag” file from “/var/merce/var/data/hbeat”.

Now restart the network service on server.

# /etc/init.d/network restart

This will remove the remote server virtual IP, and hbeat will start again. Also at same time start/restart the remote server so it will work as before.

Note:- After restoring the services restart the server is better option.

Attachment	Size
failover.tgz	59.07 KB

Merce

Projects

Life @ Merce

S&P: Standards and Procedures

Merce

BD

Tech Support

Read, learn, grow

SQL-Ledger

Drupal Help

Miscellaneous

Printer