We previously mentioned how we had moved from a tape backup to using the free IBM backup application (GO BACKUP) to using the NFS capability to move IMGCLG entries to our remote NAS.
Initially everything seemed to be going OK but as the systems got bigger we found that the save process was taking hours to complete after the save of the objects and IFS had completed. We added a new 10Gb LAN to the mix in the hope it would bring the speed back up as the saves were starting to eat into the next day before the CPY was completed.
The code provided in the post above uses the IBM backup to carry out a save to the IFS via IMGCLGE. We feel Disk is so much more affordable these days and many people will have disk utilization figures well below what were available with the older expensive disks, so moving to an IMGCLG based backup should not be beyond the reach of most small IBM i users. I recently signed onto a customer system and they were at 2% disk utilization with all internal disks on a new Power9!
A recent post on LinkedIn by Diego Kesselman also mentions how you can set up the NFS for cloud like backups.
We expressed our concern about the slowness of the CPY function but never did anything about showing the results we had seen.
We have now decided to show our findings and how a move to FTP based replication of the IFS objects improved the overall backup time and actually utilises the LAN capability we added.
We use an internal LAN to host the servers so we have not opted for FTPS or SFTP, we have noticed that adding secure connectivity can have a significant impact to the transfer rates over TCP/IP. The CPY function was only using non secure connections so adding on secure connections would not have given us a like for like comparison (maybe we will add that later). The Power 8 runs all internal disk and is partitioned into 6 partitions with various memory and CPU allocations. The main development LPAR hosts the other LPAR’s using the IBM i hosted LPAR technology (no VIOS), this is the LPAR we do most back ups from.
The Power 8 is connected to a 10GB QNAP switch via copper CAT6e cable through a 10Gb card, this in turn is connect via CAT6e to a further 10Gb Microtik switch. the Microtik has 2 10Gb fiber connections to the QNAP NAS. The network is not the problem here because it runs the FTP process in the same environment as it runs the NFS connection.
The monthly backup saves all libraries, folders and IFS, its not a full system backup. It normally saves around 9Gb of data and takes around 10 minutes to complete the backup to the IMGCLG. After the save completes we use a CPY command to move the object from the IFS backup location to a mounted directory over NFS, this is where the problem lies.
As the copy proceeds we see a lot of messages in the joblog “An error has occurred in the Network File System (NFS).” when you look at the message you will see that the remote server has stopped responding! We have not dug into why the messages are sent or what causes them, but they occur regardless of which LPAR (Different OS levels) we try the command from.
It keeps retrying the connection and sometimes will complete others it ends with a message stating a function is not supported?
If you review the code you will see we had no monitoring on the CPY command so if it failed we just ignored it. This is obviously not a good option but at the time we felt we still had the original copy in the IMGCLG so we would look at the problem later, we never did!
The problem is the CPY function, when it did manage to complete it would take hours to move the object to the mounted drive. The above test took 40 minutes only to have nothing on the target system. We took the same backup object and copied it to the NAS using FTP, the transfer took 84 seconds!
9476309153 bytes transferred in 84.002 seconds. Transfer rate 112810.578 KB/sec.
The Code used
The following ‘C’ code is what we have used to carry out the save and subsequent copy, its not production code (we would need to add a lot more checking and recovery..) But its definitely better than we had for the NFS.
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <time.h>
#include <recio.h>
// function bld_ftp_cmd()
// Purpose : Create the command to transfer the save object
// @parms
// Local Path
// Target Path
// File to transfer
// returns 1
int bld_ftp_cmd(char *Lcl_Path,
char *Tgt_Path,
char *File,) {
_RFILE *fp; // file pointer
_RIOFB_T *fdbk; // feddback area
int len = 0; // counter
char buffer[92]; // file buffer
char temp_buf[256]; // temp buffer
char *start; // temp ptr
char Q_File[25] = "YOURLIB/QCLSRC(CPYSAVDTA)"; // file name and member
// We will create a file that will control the FTP process
// because each save goes to a different location and copies to a different location
if((fp = _Ropen(Q_File,"wr")) == NULL) {
printf("Failed to open File %s\n",Q_File);
exit(-1);
}
start = &buffer[12];
// write the content to the file
// the user name and password
memset(buffer,' ',92);
// set the user name and password to use for the remote FTP server
memcpy(start,"user password",13);
fdbk = _Rwrite(fp,buffer,92);
// binary transfer
memset(buffer,' ',92);
memcpy(start,"bin",3);
fdbk = _Rwrite(fp,buffer,92);
// name format 1 IFS
memset(buffer,' ',92);
// the target will complain about the namefmt but its required for the IBM i
memcpy(start,"namefmt 1",9);
fdbk = _Rwrite(fp,buffer,92);
// cd local directory
memset(buffer,' ',92);
len = sprintf(temp_buf,"lcd %s",Lcl_Path);
memcpy(start,temp_buf,len);
fdbk = _Rwrite(fp,buffer,92);
// cd remote directory
memset(buffer,' ',92);
len = sprintf(temp_buf,"cd %s",Tgt_Path);
memcpy(start,temp_buf,len);
fdbk = _Rwrite(fp,buffer,92);
// create PUT command
memset(buffer,' ',92);
len = sprintf(temp_buf,"PUT %s",File);
memcpy(start,temp_buf,len);
fdbk = _Rwrite(fp,buffer,92);
// quit
memset(buffer,' ',92);
memcpy(start,"quit",4);
fdbk = _Rwrite(fp,buffer,92);
_Rclose(fp);
return 1;
}
// The main entry point.
int main(int argc, char **argv) {
int dom[12] = {31,28,31,30,31,30,31,31,30,31,30,31}; // days in month
char wday[7][3] = {"Sun","Mon","Tue","Wed","Thu","Fri","Sat"}; // dow array
int dom_left = 0; // days left in month
int len = 0; // counter
char Path[255]; // path to cpy save to
char Cmd[255]; // command string
char Lcl_Path[16] = "/Backup"; // where the backup is stored
char File[11]; // file name
char Tgt_Path[96]; // Target path
time_t lt; // time struct
struct tm *ts; // time struct GMTIME
int LY; // Leap year flag
// get the time
if(time(<) == -1) {
// could do more here but never failed so far.
printf("Error with Time calculation Contact Support \n");
exit(-1);
}
// create the local time structure so we can get the day of the week etc
ts = localtime(<);
// if leap year LY = 0
LY = ts->tm_year%4;
// if leap year increment feb days in month
if(LY == 0)
dom[1] = 29;
// check for end of month
dom_left = dom[ts->tm_mon] - ts->tm_mday;
// see if this is a scheduled request, it could be manually run from the command line
if(memcmp(argv[1],"*SCHED",6) == 0) {
// see how close we are to the end of the month and if its a Friday
if((dom_left < 7) && (ts->tm_wday == 5)) {
// should check for complete but errors not always an issue
// as some object cannot be saved
// replace the catalogue entry. If you use the same one it never gets reduced in size
// so you could be wasting a lot of space
system("LODIMGCLG IMGCLG(BACKUP) OPTION(*UNLOAD) DEV(VRTTAP01)");
system("RMVIMGCLGE IMGCLG(BACKUP) IMGCLGIDX(*VOL) VOL(MTHA01) KEEP(*NO)");
system("ADDIMGCLGE IMGCLG(BACKUP) FROMFILE(*NEW) TOFILE(MTHA01) VOLNAM(MTHA01) IMGSIZ(50000)");
system("LODIMGCLG IMGCLG(BACKUP) DEV(VRTTAP01)");
// call the IBM backup process, it uses the control files created using GO BACKUP
system("RUNBCKUP BCKUPOPT(*MONTHLY) DEV(VRTTAP01)");
// the system command does not return until the save finishes
// build the local and target paths
len = sprintf(Tgt_Path,"/Backups/%s/Monthly",argv[2]);
len = sprintf(File,"MTHA01");
// move the save object to the NAS
// call the function above to create the control file, its different for each save
bld_ftp_cmd(Lcl_Path,Tgt_Path,File);
// call the ftp batch process.
system("CALL SASLIB/FTPCTLPGM");
}
else if(ts->tm_wday == 5) {
// replace the catalogue entry
system("LODIMGCLG IMGCLG(BACKUP) OPTION(*UNLOAD) DEV(VRTTAP01)");
system("RMVIMGCLGE IMGCLG(BACKUP) IMGCLGIDX(*VOL) VOL(WEKA01) KEEP(*NO)");
system("ADDIMGCLGE IMGCLG(BACKUP) FROMFILE(*NEW) TOFILE(WEKA01) VOLNAM(WEKA01) IMGSIZ(50000)");
system("LODIMGCLG IMGCLG(BACKUP) DEV(VRTTAP01)");
system("RUNBCKUP BCKUPOPT(*WEEKLY) DEV(VRTTAP01)");
len = sprintf(Tgt_Path,"/Backups/%s/Weekly",argv[2]);
len = sprintf(File,"WEKA01");
// move the save object to the NAS
bld_ftp_cmd(Lcl_Path,Tgt_Path,File);
// call the ftp batch process.(future may need to create a secure FTP batch process?)
system("CALL SASLIB/FTPCTLPGM");
}
else {
// replace the catalogue entry
system("LODIMGCLG IMGCLG(BACKUP) OPTION(*UNLOAD) DEV(VRTTAP01)");
system("RMVIMGCLGE IMGCLG(BACKUP) IMGCLGIDX(*VOL) VOL(DAYA01) KEEP(*NO)");
system("ADDIMGCLGE IMGCLG(BACKUP) FROMFILE(*NEW) TOFILE(DAYA01) VOLNAM(DAYA01) IMGSIZ(10000)");
system("LODIMGCLG IMGCLG(BACKUP) DEV(VRTTAP01)");
system("RUNBCKUP BCKUPOPT(*DAILY) DEV(VRTTAP01)");
len = sprintf(Tgt_Path,"/Backups/%s/Daily/%.3s",argv[2],wday[ts->tm_wday]);
len = sprintf(File,"DAYA01");
// move the save object to the NAS
bld_ftp_cmd(Lcl_Path,Tgt_Path,File);
// call the ftp batch process.
system("CALL SASLIB/FTPCTLPGM");
}
}
// not come from a schedule but from a submitted job. Same process though
else if(memcmp(argv[1],"*DAILY",6) == 0) {
// replace the catalogue entry
system("LODIMGCLG IMGCLG(BACKUP) OPTION(*UNLOAD) DEV(VRTTAP01)");
system("RMVIMGCLGE IMGCLG(BACKUP) IMGCLGIDX(*VOL) VOL(DAYA01) KEEP(*NO)");
system("ADDIMGCLGE IMGCLG(BACKUP) FROMFILE(*NEW) TOFILE(DAYA01) VOLNAM(DAYA01) IMGSIZ(10000)");
system("LODIMGCLG IMGCLG(BACKUP) DEV(VRTTAP01)");
system("RUNBCKUP BCKUPOPT(*DAILY) DEV(VRTTAP01)");
len = sprintf(Tgt_Path,"/Backups/%s/Daily/%.3s",argv[2],wday[ts->tm_wday]);
len = sprintf(File,"DAYA01");
// move the save object to the NAS
bld_ftp_cmd(Lcl_Path,Tgt_Path,File);
// call the ftp batch process.
system("CALL SASLIB/FTPCTLPGM");
}
else if(memcmp(argv[1],"*WEEKLY",7) == 0) {
// replace the catalogue entry
system("LODIMGCLG IMGCLG(BACKUP) OPTION(*UNLOAD) DEV(VRTTAP01)");
system("RMVIMGCLGE IMGCLG(BACKUP) IMGCLGIDX(*VOL) VOL(WEKA01) KEEP(*NO)");
system("ADDIMGCLGE IMGCLG(BACKUP) FROMFILE(*NEW) TOFILE(WEKA01) VOLNAM(WEKA01) IMGSIZ(50000)");
system("LODIMGCLG IMGCLG(BACKUP) DEV(VRTTAP01)");
system("RUNBCKUP BCKUPOPT(*WEEKLY) DEV(VRTTAP01)");
len = sprintf(Tgt_Path,"/Backups/%s/Weekly",argv[2]);
len = sprintf(File,"WEKA01");
// move the save object to the NAS
bld_ftp_cmd(Lcl_Path,Tgt_Path,File);
// call the ftp batch process.
system("CALL SASLIB/FTPCTLPGM");
}
else if(memcmp(argv[1],"*MONTHLY",8) == 0) {
// replace the catalogue entry
system("LODIMGCLG IMGCLG(BACKUP) OPTION(*UNLOAD) DEV(VRTTAP01)");
system("RMVIMGCLGE IMGCLG(BACKUP) IMGCLGIDX(*VOL) VOL(MTHA01) KEEP(*NO)");
system("ADDIMGCLGE IMGCLG(BACKUP) FROMFILE(*NEW) TOFILE(MTHA01) VOLNAM(MTHA01) IMGSIZ(50000)");
system("LODIMGCLG IMGCLG(BACKUP) DEV(VRTTAP01)");
system("RUNBCKUP BCKUPOPT(*MONTHLY) DEV(VRTTAP01)");
len = sprintf(Tgt_Path,"/Backups/%s/Monthly",argv[2]);
len = sprintf(File,"MTHA01");
// move the save object to the NAS
bld_ftp_cmd(Lcl_Path,Tgt_Path,File);
// call the ftp batch process.
system("CALL SASLIB/FTPCTLPGM");
}
// clean up the FTP file and remove the password..
system("CLRPFM FILE(SASLIB/QCLSRC) MBR(CPYSAVDTA)");
return 0;
} Result
So we have now changed the backups to always use the FTP process and dropped the NFS as a viable option. We have not identified the reason for the slowness of the NFS from the IBM i so cannot offer and conclusion as to why it is so slow and falls over?
The above is a very basic save with no tape sets or clever day naming scheme, but it does store each DAYA01 file in a separate directory for each day of the week. The objects saved are determined using the files generated when you use the option to Set up the backup, the IFS is not well covered in the backup options (you either save it or not with no filtering capability, maybe that could be a Change Request to IBM) so we could definitely work on that.
I thought I would share the findings, we have been meaning to figure this out for some time and finally found a few minutes to investigate and add the FTP process to the original save process. Its not a clever piece of code by any means but should give you the basics to start working from. Our next challenge will be building a remote source code repository over a secure connection and fully encrypted content :-), if we get it working sufficiently well we will probably share the code.
Chris…