Jul 04

Who would have thought! I am starting to use RPG!

I have always said that I did not need to learn or use ‘RPG’ on the IBM i as I always found that ‘C’ could do all that I needed. Recently I was asked by a friend to help them with some RPG code to handle Java and the clean up of the objects it created (Java would not automatically clean up objects because they were effectively created by the RPG program and this program ran constantly, so temp storage just kept growing until it blew up). Not knowing RPG or understanding how the layout worked (I jumped straight into ‘/Free’!) I found this very difficult as ‘/Free’ is not really free format(there are still some column constraints) while ‘C’ really is free format. Still after some research and a lot of head scratching I finally got some sample code working, we then built a Service program that could handle the Java clean up and built code into the existing RPG programs to call it. The solution works the clients systems are not blowing up with memory issues caused by Java objects not being cleaned up.

I though OK that’s the last time I will have to do that and was happy that I could get back to good old ‘C’ programming. Unfortunately, I came across another issue which required me to pick up the RPG manuals and code up a test application.

We have a client who was experiencing problems with an application that uses commitment control and constraints that required us to build a test which would emulate the problem on our systems. As usual the first thing I did was to write a ‘C’ based solution, I did find a Commitment control test which was written by Paul Tuohy here. This was all written using RPG so I thought I would just follow the program logic and write a ‘C’ version which seemed the easiest option. While I could get the simple file update logic built and the program would work without Commitment Control, I found that as soon as Commitment Control was started the program would freeze on receipt of data from STDIN, (I will have to ask IBM why when I have time) so I decided my bets options was to take the code that Paul had provided and build my own interpretation of the program with some additional features I needed.

I wanted the program accept multiple entries plus allow deletes by key before the commit of the data so I had to make a few changes to the logic and add a new delete option. While the program is very clunky it does achieve what I needed it to do and I found out a lot about commitment control and constraints as a result. I am also unsure if the program is as efficient as it could be, but it works and for now that is all that’s needed.

Here is the code I ended up using.


H Option(*SrcStmt : *NoDebugIO) DftActGrp(*No) ActGrp('COMMITDEMO')
FHeader1 UF A E K Disk Commit(SomeTimes)
FDetails1 UF A E K Disk Commit(SomeTimes)

D Commit1 PR ExtPgm('COMMITRPG')
D SomeTimes n

D Commit1 PI
D SomeTimes n

D ToDo S 1a
D ToDel S 1a
D ToAdd S 1a
/Free
ToAdd = *Blanks;
Dow (ToAdd <> 'n');
Dsply 'Enter a Key Value (2 long): ' ' ' Key;
If (Key <> *Blanks);
Text = 'Header for ' + Key;
Write Header;
For Sequence = 1 to 3;
Text = 'Detail for ' + Key + ' ' + %Char(Sequence);
Write Details;
Chain Key Header1;
NumRows += 1;
Update Header;
EndFor;
NumRows = 0;
EndIf;
Key = *Blanks;
Dsply 'Enter a more Keys y/n : ' ' ' ToAdd;
EndDo;
ToDel = *Blanks;
Dsply 'Do you want to delete entries : ' ' ' ToDel;
Dow (ToDel <> 'n');
Key = *Blanks;
Dsply 'Enter the Key to Delete : ' ' ' Key;
if (Key <> *Blanks);
chain Key header1;
if %found;
delete header1;
EndIf;
EndIf;
ToDel = *Blanks;
Dsply 'Do you want to delete more entries : ' ' ' ToDel;
EndDo;
ToDo = *Blanks;
Dow (ToDo <> 'c' and ToDo <> 'r' and ToDo <> 'i');
Dsply 'c - Commit, r - Rollback, i - Ignore ' ' ' ToDo;
EndDo;

If SomeTimes and (ToDo = 'c');
Commit;
ElseIf SomeTimes and (ToDo = 'r');
RolBk;
EndIf;

*InLR = *On;
/End-Free

Note: the Blog does not allow RPG code indentation so the view you see is not what it was copied in as!

The database was exactly the same that Paul had defined including the cascading delete for the details file (I liked that bit) so when we delete the Header Record the matching records in the Details file are also deleted. That saved us having to chain (see I can speak RPG) the details file and remove the entries. Now we can see the problem the client was experiencing and know how to resolve it.

As usual Google was our best friend, thanks to Paul Tuohy and ITJungle for providing the sample code we based the test application on. I am now a little less resistant to RPG and may delve a little more into its capabilities and how I can use it effectively, who knows I may even become good at it?? The point I am trying to make here is that while I still do not want to use RPG, I did what I keep telling others to do, I used the best tool for the job. Using any language just because it is all you know is not always the best option, sometimes you have to jump outside of your comfort zone and try something new.

Chris…

Jun 24

New RDX Drive not supported by the BACKUP menu commands on our V7R1 Power 720

When we read that IBM was recommending users who are currently using the DAT160GB tape system move to the RDX drive system for backup purposes, we decided we would give it a try. First of all we checked with a number of people that the RDX drives were a suitable backup device and to make sure it was going to be supported on our small 720 system (IBM seemed to be positioning at our size of company and hardware) before we placed the order. We placed the order over a week ago and the drive finally arrived today.

After finding out that our planned move to VIOS based partitioning was flawed and not possible with internal disk, we had high hopes that the support for RDX technology would be a big step forward from our current backup technology (tape is painfully slow and error prone) especially as we have multiple partitions. We made the decision to purchase the enclosure and 2x320GB drives at the same time we purchased the hardware/software to support the move to VIOS partitions. We understood the ethernet card and PowerVM licenses were now extra to requirements, but we still hoped that the investment in the new drive technology would be worthwhile.

All of the hardware and software turned up today so we unpacked the drives and enclosure and attached it to the IBM i via the front USB port. It was powered on and the drive inserted which showed all green lights on the front of the enclosure.

First problem we came across was the drive would not show up in the hardware configs, when we previously migrated back to i-hosting-i we did not allocate the USB adapter in the hardware profiles, so a quick configuration update was made and the drive finally showed up in the available resources on the partition. Next we decided to test moving the drive between partitions (systems) using the DLPAR options, it all seemed to work fine as long as we ensured the device was varied off prior to the DLPAR move request. It did take some time for the adapter to move and even longer for the device to show up in the partitions.

Once we were happy with the ability to move between partitions we formatted the drive in anticipation of using it for our daily/weekly backups. The format was very quick and showed the correct 320Gb of available space on the drive. We then tried to add it to the backup schedule in place of our tape device was where we came across the biggest problem, the IBM i BACKUP options provided with the OS only support tape drives!! So we are now faced with having to develop our own backup processes to allow us to use it the drive for our backups.

We are still unsure how the backup will be stored on the drive, IBM has indicated that it should function in the same manner as a tape meaning that it will allow multiple saves to be carried out to the same device and write to the end of the last save. We now have to build the save processes and set them up to replace the OS based solution that we have today. Once we get the save processes developed we will report back just how good the drives are and how easy they will be to use for our simple backup requirements. Have to stick with Tape for now unless IBM adds support for the RMS devices in the OS BACKUP solution in the near future. Yet again our IBM connections really didn’t know the capabilities of the RDX drives in an IBM i environment, maybe we can come up with some answers…

Chris…

Jun 23

Annoying CPF9E7F message fixed

After the attempted migration from i-hosting-i to a VIOS based partition configuration and subsequent rebuild of the i-hosting-i partitions, we found that the QSYSOPR message queue was being sent CPF9E7F messages constantly. We checked the HMC configurations and everything looked OK because we had configured 4 partitions with a total of 2 Processors out of the 4 we have available. We had upgraded the system to have 4 available processors ready for the VIOS configurations where we intended to use 2 for IBMi, 1 for AIX and 1 for Linux.

We asked our sales rep what the problem was especially as we have a license for the additional AIX core which we wanted to implement as well, his response was to speak with support as it looked like we were exceeding our licenses. Eventually we raised a PMR and spoke with IBM, they informed us that while we were not technically exceeding our entitlement the way the IBMi OS calculated the available CPU cores meant it saw a problem. The answer was pretty simple to implement, we had to set up Shared Processor Pools and allocate a maximum number of available cores to that pool. Then we then had to make each partition use that pool so that we could not exceed our entitlement. This was done using the Shared Processor Pool Management option in the HMC where we created the new pool and set the partitions to use that pool. That fixed the immediate problem, but the partition profiles also needed updating and the to be re-booted for the changes to take permanent effect.

When we created the IBM i shared pool we also took the opportunity to create a AIX pool and a Linux pool so that when we add those partitions to the system we can correctly allocate the additional processors to them.

We no longer see the CPF9E7F messages and everything runs just the same as it always did. We continue to learn just how capable the IBM i Power system can be, the downside to that is just how complex it can be as well. We hope to set up the AIX partition and Linux partitions in the near future, we will post our experiences as we go along.

Chris…

Jun 12

Issue with ‘restore 21′ resolved, everything running

The problems with the restore 21 of the partition data have been resolved and all of the partitions are now up and running.

The problem which gave us the most grief was the update to the content of the partition which was running V7R2. For some reason the restore operation kept hanging at different spots in the restore 21 process. One of the problems seemed to be with damaged objects on the system which caused the restore to hang and required a forced power off of the partition (SYSREQ 2 did nothing). We cleaned up the damaged objects and started the restore again only to hang again while restoring the IFS only this time we could end the restore operation with SYSREQ 2 and get back to a command line. There was nothing in the joblog to show why the restore was hanging so we eventually manually run the command to restore the IFS. We then started the partition and everything looked OK, but when we tried to start the HTTP server (we like the mobile support so we needed it running) it kept ending abnormally, turns out we forgot to run the RSTAUT command. Restore 21 does this after the RST for the IFS completes. After we ran the RSTAUT the jobs all started up correctly and we had the partition up and running again.

The other problem we had was with a V6R1 partition, it refused to start complaining about a lack of resource (B2008105 LP=00004). As this was a deployment of a running configuration so we thought nothing had changed and wondered why it would no longer start up. In the back of our minds we had a vague recollection that setting up partitions for V6R1 on Power7+ systems required the RestrictedIO partition flag to be set so we looked through the partition profile to find where it was set without success. We discovered that it is not part of the profile, you have to set the flag in the properties for the partition. Once we had done this the partition came up without any further problems and we now had all of our original configuration up and running.

We made a couple of additional changes to the configs because one of the reasons we really liked the VIOS option was being able to start everything up at once. With our set up we were powering up the host partition and then powering up each of the clients manually. We wanted to be able to power on the system and all of the partitions would fire up automatically. Also when we wanted to power down we just wanted to power down the host partition and it would take care of all the hosted partitions, the answers is the Power Controlling settings. We set up each of the NWSD objects in the hosting server to be Power Control *YES, we then updated the profiles for the hosted partitons to be Power Controlled by the hosting partition. After initializing the profiles with the NWSD object varied off and shutting down the profiles we then varied on the NWSD objects and the partitions automatically started up. Now when we start the main partition the other partitions all start once the NWSD is activated (they are all set to vary on at IPL). We also set the hosting partition to power on when the server was powered on and the server to power off when all of the partitions were ended. We have not tested the power down sequence to make sure the guest partitions are ended normally when we PWRDWNSYS *IMMED on the hosting partition but it should shut down each partition gracefully before shutting itself down.

Now its back to HA4i development and testing for the new release, manuals to write and a new PHP interface to design and code. Even though we like the Web Access for i interface it is not as comprehensive as the PHP interface in terms of being able to configure and manage the product.

If you are planning a move to partitioning your Power system we hope the documenting of our experiences is helpful.

Chris…

Jun 11

Rebuild of the i-hosting-i underway.

We have finally started the rebuild of the data for the i-hosting-i partitions and came across a few problems.

First problem was to do with the system plan. Before we started down the VIOS route we created a system plan from the existing partition and system information and checked it to make sure we had no errors logged. Nothing was shown as a problem so our plan was to use it to deploy again if we could not get the VIOS set up functioning. As it turns out we could not use the system plan, the deployment failed every time because of adapter issues which did not show up when we viewed the plan on the HMC.

This required us to edit the system plan which required us to use the system planning tool. We downloaded the SPT to a PC and installed it, a slight issue with Windows 8 meant we had to run the program in Windows 7 mode to get it to install, but once it was up and running we managed to import the original system plan. Even though the system plan was created from a running system with active partitions the planning tool threw up a lot of errors. We had problems with the addition of the internal SATA tape drive blocking the USB adapter and so on which took a pretty long time to understand, in the end we just configured few things we must have to export the plan and exported it ready for import to the HMC. Eventually the plan did deploy on the HMC so it looked like we were ready to go.

We did an IPL D using the SAVSYS tape and all seemed to go well until we got to the DASD configuration in DST. We had the LIC installed the first drive as the load source but we needed to add all of the other drives and Raid protect them. As we progressed through the DST options we kept getting errors about connections being missing, a search using Google turned up nothing so decided to take the F10 option (ignore the message and continue). It turned out to be a problem because we only had one of the Raid cards set up, not have both (I thought we only had one but 2 show up in the hardware list) so when we took the option to add the drives to ASP1 and then started Raid protection it took hours (IBM support did try to help by DLPAR’ing the additional Raid card but we were too late to gain any benefit) so 6 hours later we had the drives set up and protected.

Because this is the hosting partition the other partition data was restored at the same time which took about 5 hours to complete. We checked the NWSD objects for the hosted partitions were restored correctly and configured, we saw that they were were in a VARIED OFF state so we VARIED them ON and watched as they became ACTIVE, so far so good.

At this point we thought OK we are now ready to start the other partitions. We took the option to activate the first partition profile on the HMC but it quickly came to a grinding halt! the SRC code displayed was B2004158 LP=0002, not much information turned up with a Google search so I tried to get a console up to see what was actually going on. It appears that when you first start the partition you need to specifically set the advanced start up parameters the first time (the normal setting is do not override the Mode and source settings), we just set it to B,N and the partition started up.

We still have one partition which fails to start, this is a V6R1 partition and while we did see some reference in the VIOS configurations to dedicated IO for V6R1 on Power 7+ we know this was running before so we think it was damaged on the restore of the NWSD? We have a full system save on tape for it so as soon as everything else is fixed we will try a IPL D with the SAVSYS and rebuild the data.

After over a week of fighting with IBM to get the right hardware and software to run a VIOS based partitioned system we have accepted that i-hosting-i will be the solution for now. We have already started to look at SAN in the hopes of one day having enough bandwidth to trek down this road again, this time we know that internal disks are not for VIOS partitioning! Pity the IBM sales team didn’t know that before we ordered the additional hardware for Ethernet and the additional core activations for PowerVM. I am sure that with enough trail and error you could get a VIOS running with internal disk running, but if the performance is degraded as IBM suggests (they don’t say by how much) I think it may be a futile exercise?

Hope you find the information useful, maybe it will help you avoid some of the pitfalls we came across and save you time and money :-).

Chris..

Jun 10

Its a bust!

Finally we get the answer we have been looking for..

Generally we don’t recommend VIOS and virtualised partitions using internal disks.
Usually organisations are using VIOS with external storage.
There are many reasons – performance, benefits, etc.

Yep, mostly for performance reasons, its i-hosting-i on internal disk, vios for external disk…..The big problem with that is that very few people are crossing those boundaries.

So all of the work so far to get the VIOS set up has been in vain.. Well not entirely because we have learned a lot of very good lessons about the AIX/VIO interfaces and how to set up and install. But for now we are just going to back peddle and use i-hosting-i until we can get a SAN to test out what a VIOS implementation can provide. I am also interested in how we could set up the internal disks to run IBM i hosting while having a single drive for VIOS that could manage the external drives (if that is in fact possible).

If we do actually get to the stage of implementing we will again publish our experiences. May take us a while to get back to this as we need to ensure the HA4i product release is put back on track.

Keep watching.

Chris…

Jun 10

Setting up the new VIOS based Partitions in question

We have been trying to migrate our existing IBM i hosting IBM i partitions to a VIOS hosting IBM i, AIX,Linux configuration. As we have mentioned in previous posts there a re a lot of traps that have snagged us so far and we still have no system that we can even configure.

The biggest recommendation that we took on board was to create a Dual VIOS setup, this means we have a backup VIOS that will take over should the first VIOS partition fail. This is important because the VIOS is holding up all of the other clients and if it fails they all come tumbling down. As soon as we started to investigate this we found that we should always configure the VIOS on a separate drive to the client partitions, my question is how do we configure 2 VIOS installs (each with its own disk) that addresses the main storage to be passed out to the client partitions. We have a Raid controller which we intend to use as the storage protection for the Clients Data but we still struggle with how that can be assigned to 2 instances of the VIOS?? The documentation always seems to be looking at either LVM or MPIO to SCSI attached storage, we have all internal disk (8 SAS drives attached to the raid controller) so the technology we use to configure the drives attached to the raid controller as logical volumes which are in turn mirrored via LVM is stretching the grey matter somewhat? If in fact that is what we have to do? I did initially create a mirrored DASD pair for a single VIOS in the belief that if we had a DASD failure the mirroring would help with recovery, however The manuals clearly state that this is not a suitable option (I did create the pair and install VIOS which seemed to function correctly so not sure why they do not recommend?).

The other recommendation is to attach dual network controllers and assign them to each of the VIOS with one in standby mode which will be automatically switched over should a failure occur on the main adapter. As we only have a single adapter we have now ordered a new one from IBM(we have started the process and it has taken over 1 week so far and the order is still to be placed..) Once that adapter arrives we can then install it and move forward.

Having started down this road and having a system which is non functioning I have to question my choices. IBM has stated that the VIOS will be the preferred choice for the controlling partition for Power8 and onwards, but the information to allow small IBM i customers to implement (without being a VIOS/AIX expert) in in my view very limited or even non existent. If I simply go back to the original configuration of IBM i hosting IBM i, I may have to at some time in the future bite the bullet and take the VIOS route anyhow? Having said that, hopefully more clients would have been down this route and the information from IBM could be more meaningful? I have read many IBM redbooks/Redpapers on PowerVM and even watched a number of presentations on how to set up PowerVM, however most of these (I would say all but that may be a little over zealous) are aimed at implementing AIX and Linux partitions even though the IBM i gets a mention at times. If IBM is serious about getting IBM i people to really take the VIOS partitioning technology on board they will need to build some IBM i specific migration samples that IBM i techies can relate to. If I do in fact keep down this path I intend to show what the configuration steps are and how they relate to an IBM i system so they can be understood by the IBM i community.

We have a backup server that we can use for our business so holding out a few more days to get the hardware installed is not a major issue, we hope that by the time we have the hardware we have some answers on how the storage should be configured to allow the VIOS redundancy and make sure we have the correct technology implemented to protect the client partitions from DASD failure.

If you have any suggestions on how we should configure the storage we are all ears :-)

Chris…

Jun 06

Why do I do it…

Well this week has been a total write off, having spent 4 days trying to get the systems ready to migrate from IBM i based partitioning to VIOS based partitioning (IBM had incorrectly configured the core activations and it took 4 days to get me the information to correct it!) I finally got to a state where I could start the VIOS install.

I had the trusty Red Books on hand and decided to follow one of the set ups described, I removed the existing system definitions and partitions definitions from the HMC (I have backed up the partitions individually and created a system plan) so I felt secure that if required I could simply deploy the system plan again.

I created a single VIOS partition definition as per the manual and started the installation process. The HMC level we have has an option to install the VIOS as part of the partition activation which is not noted in the manuals, this came to a grinding halt with a message about incorrectly formatted commands??? So we went back and followed the instructions on installing the VIOS using the SMS install and a terminal. Next mistake was trying it from a remote HMC connection (you have to be on the main HMC display to allow the terminals to be launched) so we them moved to the main HMC display. Everything started to look good, the SMS installations screens came up and we dutifully selected the options to install from the DVD. Again the install just hung, so we rebooted and tried again, this time we noticed that the installation could not find the Disk to install the VIOS on.

I am not sure why, perhaps its because the system was installed with a single Raid6 setup with IBM i as the controlling partition? But no matter which options we looked at we cannot find the method to re-initialize the disk to allow the VIOS to install.

Logged a support call and waiting for the support gods to give us a call and hopefully get past this stage. Once we get the information I will be sure to write it up :-)

Chris…

Jun 05

What does V8R0 of HA4i look like?

While we wait for IBM to get back to us about our PowerVM activations (3 days and counting, I often wonder does IBM want to service clients?) I thought I would start to show some of the changes we have made in the next release of HA4i. The announcement date for the next release is a little way off as we still have to get the manual and new PHP interfaces finished, but we all feel excited about some of the new capabilities so we thought we would start to share.

As the PHP interface is not completed and we have found the IBM Access for Web product is performing very well, we thought it would be an ideal opportunity to show it off at the same time we display some of our new features. So far the displays have been pretty pleasing with no problems in showing the content effectively. Again we will point out the fact that the web interface is being run on one system (shield7) and the system running HA4i is another (shield8), the ability to launch a 5250 session from the web interface to another system without the web software running on that system is pretty neat in our view.

The first screen we will share is the main monitoring screen, this is a screen shot of the 5250 green screen output using the standard Client Access emulator.

5250 Roleswap Status Green screen

5250 Roleswap Status Green screen

Here is the IBM Access for Web output of the same screen, we have placed arrows and markers to show some of the features which we will describe below.

Roleswap Status Access for Web

Roleswap Status Access for Web

Arrow 1.
A)These are the options that are available against each of the environment definitions, these can be used to drill down into more specific data about each of the processes involved in the replication of the objects and data.

B)You will notice that we can end and start each environment separately, there is also an option on the operations menu which will start and stop every environment at once.

C) You can Roleswap each individual environment, the previous version only allowed a total system Roleswap.

Arrow 2.
A) Some environments should not allow Roleswaps to be carried out, we have defined 2 such environments to replicate the JQG4i data. Because the data is only ever updated on the generating system and each system has its own data sets you would never want to switch the direction of replication. The Y/N flags show that the BATCHTST environment can be switched while the JQG4i environments cannot.

Arrow 3.
A) These are the environment names, each environment runs its own configurations and processes.

Arrow 4.
A) This is the mode of the environment on this system *PROD states that this is a source system where the object changes are captured while the *BACKUP is where the changes will be applied. when viewing the remote system these roles will be reversed.

Arrow 5.
A) If there are any errors or problems found within any of the replication processes you should not carry out a roleswap, HA4i retrieves the status from both the local and remote system to determine if an environment is capable of being roleswapped based on the state of the replication processes. As you can see if an environment should not be roleswapped the entry is marked as *NA.

Arrow 6/7/8.
A) This is the state of the various replication processes, *GOOD states that there are no errors and everything that should be running is. *NOCFG states that no configurations exist that require the replication process to be running. Data status is the journal apply process and which could encompass more than one apply process if there is more than one journal configured to the environment.

Arrow 9.
A) You can view the configs from any system but changes to the configs can only be carried out on the *BACKUP system. the configuration pages can be accessed using this button (F17 on the 5250 Green screen).
B) The Remote Sys button (F11 on the 5250 green screen) just displays the remote system information.

There are a lot more new features in the next release which will make HA4i more competitive in complex environments, over the next few weeks/months we will show you what they are and why they are important. The big take away from above is the ability to define a much more granular approach to your replication needs. Becuase we can define multiple systems and multiple environments HA4i is going to be a lot more useful when you need to migrate to new hardware and expand data replication beyond 2 systems.

We hope that you like the features and if you are looking at implementing a new HA solution or looking to replace an existing one that you consider HA4i.

Chris…