Let’s ‘C’ how!

I hope that you are following the series and found the previous posts  and the tools we have produced useful. In this post I am going to expand on the tooling to loop through a message file and check all of the source code for the messages. This will allow us to verify they are been utilized in the code somewhere, if not we could delete them. This is going to take a lot of code and explanation overall so be prepared to do a lot of code break down to get through, as always the code on the GitHub site will be updated to reflect all of the changes we have been making.

First we need to think about the best way to build the process, as always we want to try to make it as simple and as efficient as possible. The same process will be used to extract the messages from the message file we have used  so far with a few slight changes. The source file will be read through line by line looking for the message ID. As you can probably understand with 100,000’s of lines of code that could be quite a task. As such the tool may not be one that gets used a lot  but its one that not only provides a life saver, it also shows a number of cool techniques to achieve the end goal.

We had previously written similar programs already as we are often reviewing our code looking for more efficiencies where we stumbled into a number of issues related to how the IBM i ‘C’ functions work under the covers particularly in relation to file management. The code we are going to produce does resolve those issues and we will explain as we go along where appropriate.

A source file is generally made up of lots of members. When you open the file for reading by record (_Ropen()) you can pass in the member you want to open, or pass in no member (results in the first member only being opened), or pass in *ALL. You could use the relevant API’s and get a list of all of the members which can then be used in a loop to open each member, or you could create an override over the file which links all of the members into a single file open which is the same result as using the *ALL option for the member.  While doing our initial testing we came across a very bad side effect which was not apparent in the documentation, when you use the override or the *ALL open method each member results in a number messages being sent to the job log about the member being opened (see below). In our first attempt we did not take this into consideration so the loop consisted of reading through each of the message ID’s with an inner loop reading through each line of source code looking for the message ID, it would break the inner loop on finding a match and move to the next message ID.
This quickly filled the job log with useless information and caused a severe slow down in the. In the end the program would creask because the job log was filled to capacity (could have changed job to *WRAP but we needed a better answer).  To over come this we created a temporary file which would be filled with relevant lines of code ready to be read through.

This is a sample message, it is sent for every member in the source file so in our case we had over 300 members, this was looped over for over 500 message ID’s. these messages are sent if (*ALL) is used for opening the member or if we created an override.

Message ID . . . . . . : CPI5209 Severity . . . . . . . : 00
Message type . . . . . : Information 
Date sent . . . . . . : 04/24/18 Time sent . . . . . . : 09:11:44

Message . . . . : Moved from member HA4I978 to member HA4I415.
Cause . . . . . : File QCSRC in library HA4I72SRC has a data base file
override MBR(*ALL) and an end of file was detected. If a read next, a force
end of data, or a POSDBF command to position *END was specified, the next
member of the file was opened. If a read previous or a POSDBF command to
position *START was specified, the previous member of the file was opened.
Another improvement was to reduce the number of lines that will be looked through for the message ID.  We will show 2 approaches to the problem and you can decide which is the correct one for you. For now we will use the inclusion process and look at the exclusion process next time.

The following program is built to create the temp file and populate it with the source records we think might contain the message ID’s from the message files. Before that here is the command we will use to front end this program.
/*===================================================================*/
/*  COMMAND name :  COPYSRCA                                         */
/*  Author name..:  Chris Hird                                       */
/*  Date created :  April 2018                                       */
/*                                                                   */
/*                                                                   */
/*  Purpose......:  Copy Source by inclusion                         */
/*  CPP..........:  COPYSRCA                                         */
/*  Revision log.:                                                   */
/*  Date     Author    Revision                                      */
/*                                                                   */
/*  @Copyright Chris Hird 2018                                       */
/*===================================================================*/

             CMD        PROMPT('Copy Source include')

             PARM       KWD(SRCF) TYPE(QUAL1) MIN(1) PROMPT('Source File')
             PARM       KWD(TGTF) TYPE(QUAL1) MIN(1) PROMPT('Target File')
             PARM       KWD(INCSTR) TYPE(*CHAR) LEN(50) MIN(1) VARY(*YES *INT2) PROMPT('Include String')

 QUAL1:      QUAL       TYPE(*NAME) LEN(10)
             QUAL       TYPE(*NAME) LEN(10) DFT(*LIBL) SPCVAL((*LIBL)) PROMPT('Library name:')   
Here is the program code, its in member QCSRC/COPYSRCA.
//
// Copyright (c) 2018 Chris Hird
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions
// are met:
// 1. Redistributions of source code must retain the above copyright
//    notice, this list of conditions and the following disclaimer.
// 2. Redistributions in binary form must reproduce the above copyright
//    notice, this list of conditions and the following disclaimer in the
//    documentation and/or other materials provided with the distribution.
//
// Disclaimer :
// This code is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.      

#include <recio.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// function cpy_src()
// Purpose: To copy the source to a single member for better reads
// @parms
//      NONE
// returns 1 success

int cpy_src(char *NewSrc,
            char *OldSrc,
            char *filter,
            int rec_len) {
_RFILE *fp;                                 // file ptr
_RFILE *fp1;                                // file ptr
_RIOFB_T *fdbk;                             // feed back
_RIOFB_T *fdbk1;                            // feed back
char cmd[256];                              // command string
char SrcPf[32];                             // source file
char buf[151] = {'\0'};                     // read buffer
char *code;                                 // code ptr

// create the new srource file
sprintf(cmd,"CHKOBJ OBJ(%s) OBJTYPE(*FILE)",NewSrc);
if(system(cmd) != 0) {
   sprintf(cmd,"CRTSRCPF FILE(%s) MBR(SRCMBR) RCDLEN(%d)",NewSrc,rec_len);
   system(cmd);
   }
// open the old source file
sprintf(SrcPf,"%s(*ALL)",OldSrc);
if((fp = _Ropen(SrcPf, "rr" )) == NULL )  {
   printf ( "Open failed %s\n",SrcPf);
   exit(1);
   }
// target file is a single member   
if((fp1 = _Ropen(NewSrc,"rr+")) == NULL )  {
   printf ( "Open failed %s\n",NewSrc);
   _Rclose(fp);
   exit(1);
   }
code = buf + 12;
do {
   fdbk = _Rreadn(fp,buf,rec_len,__NO_LOCK);
   // include the line if it has the filter string
   if(strstr(code,filter)) {
      fdbk1 = _Rwrite(fp1,buf,rec_len);
      if(fdbk1->num_bytes != rec_len) {
         printf("Failed to add record\n");
         _Rclose(fp);
         _Rclose(fp1);
         reeturn(-1);
         }
      }
   }while(fdbk->num_bytes != EOF);
_Rclose(fp);
_Rclose(fp1);
return 1;
}

// function Crt_Q_Name()
// Purpose: To create a qualified object name. LIB/OBJ
// @parms
//      string object
//      string q name
// returns 1 success

int Crt_Q_Name(char *Object,
               char *Q_Name) {
int i,j = 0;                                // counters

for(i = 10,j = 0; i < 20; i++,j++) {
   Q_Name[j] = (Object[i] == ' ') ? '\0' : Object[i];
   }
Q_Name[j] = '\0';
strcat(Q_Name,"/");
j = strlen(Q_Name);
for(i = 0;i < 10;i++,j++) {
   Q_Name[j] = (Object[i] == ' ') ? '\0' : Object[i];
   }
Q_Name[j] = '\0';
return 1;
}

int main(int argc, char **argv) {
int rec_len = 150;                          // record length
short int filter_len = 0;                   // filter length
char NewSrc[22];                            // new source file
char OldSrc[22];                            // old source file
char filter[51] = {'\0'};                   // filter string
char *tmp;                                  // temp ptr

Crt_Q_Name(argv[1],OldSrc);
Crt_Q_Name(argv[2],NewSrc);
// set up the filter string
filter_len = *(short int *)argv[3];
tmp = argv[3]; 
tmp += sizeof(short int);
memcpy(filter,tmp,filter_len); 
cpy_src(NewSrc,OldSrc,filter,rec_len);
return 1;
}                       
The program is going to read through all of the members in the source file passed in as the SRCF parameter and write them out to the TGTF if they contain the filter string we have passed in. This is useful in the case like ours where we have a number of functions that use the message file content that all end with ‘msg(‘. Based on that we know that any message ID’s will be contained in records that have that string in them. Any other line which does not have that content will be irrelevant unless we split the lines (which we don’t).

Important notes:

The string functions all require they operate on null terminated character arrays, this requires any content passed into them to be NULL terminated, we do this for both the filter string and the record as we read it. The record length in our source files is 150 bytes, this means we have to have a buffer that can be 1 byte longer to contain the NULL termination, if you have smaller records lengths (112 is the default for RDi and 92 is the default for SEU on our systems) you will need to resize these buffers accordingly.

The command has been created to allow *VARY length parameters to be passed which have a counter at the beginning to show the length passed.
First we get the length of the argument passed, we use a cast against the start of the parameter where the counter is placed and copy that to a variable we have created.
filter_len = *(short int *)argv[3];
Next we have to set the start point of the character array we are going to search with, we do this by using a character pointer, setting it to the start of the parameter, then increment it past the short int.
tmp = argv[3];
tmp += sizeof(short int);
Next we copy the characters to the filter variable we have created, using the initialization {‘\0’} tells the compiler to set all of the character spaces in the buffer as NULL terminators when it is mapped. This means we will have a NULL terminated string after we have copied the characters from the parameter.
memcpy(filter,tmp,filter_len);
We have created a simple function that will create a qualified name from a 20 character string so ‘QCSRC CHLIB ‘ will become ‘CHLIB/QCSRC’ which is how the file open function require the file name to be passed. I would suggest this is placed in one of our service programs at some time in the future as it is possibly a function that will get used a lot.

The copy source program is quite simple, we check for the target file and if it does not exist we will create it, then we will append the qualified name we passed in for the source file and append with (*ALL) as we want to open all of the members in the file, the target will only have a single member. Once we open the files we read through each of the records in the source file to the NULL terminated buffer, then we can search the buffer for the string we are interested in. You will notice we have added a character pointer called ‘code’, we use this to skip over the first 12 bytes of the record which contains the date and line number content. Its not so important in this instance but if we were looking for line start content such as ‘//’ for comments we would need to make sure we are at the start of the line where the code could exist.

Commands to create the program and the Command to call the program.
CRTBNDC PGM(OSPGM/COPYSRCA) SRCFILE(OSLIB/QCSRC) SRCMBR(COPYSRCA) DBGVIEW(*SOURCE) REPLACE(*YES))
CRTCMD CMD(OSPGM/COPYSRCA) PGM(*LIBL/COPYSRCA) SRCFILE(OSLIB/QCMDSRC) SRCMBR(COPYSRCA) REPLACE(*YES)
So once you have created the program and the command, run it against one of your source files which meets the search requirements and check out the content of the generated file, this is what we will use in the next phase to check for the message ID’s which are actually being utilized.

Here is a sample of our output when call as follows. We still had over 4,000 lines of code after the filter was applied and we could trim it further by simply changing the filter string to omit the error messages ( filter should be ‘snd_msg(‘ ). but far better than looking though close to half million lines of code.
CALL PGM(CHLIB/RTVMSGL) PARM(‘msg(‘ ‘QCSRC HA4I72SRC ‘ ‘QCSRC QTEMP ‘)
003800170217   snd_msg("CVT0000",msg_dta,strlen(msg_ptr->msg)+2); 
007800170217   snd_msg("CVT0000",msg_dta,strlen(msg_ptr->msg)+2); 
036600170217   snd_error_msg(Error_Code);                         
004000170217      snd_error_msg(Error_Code);                      
004600170217      snd_error_msg(Error_Code);                      
006200170217   snd_error_msg(Error_Code);                         
007600170217   snd_error_msg(Error_Code);                         
032000170217   snd_error_msg(Error_Code);                         
048400170217   snd_error_msg(Error_Code);                         
054000170217   snd_error_msg(Error_Code);                         
045000170217      snd_error_msg(Error_Code);                      
054000170217      snd_error_msg(Error_Code);                      
064100170217      snd_msg("APY0020",msg_dta,strlen(msg_dta));     
090500170217   snd_msg("APY0021",msg_dta,strlen(msg_ptr->msg)+2); 
221300170217      snd_error_msg(Error_Code);                      

In this post we have looked at the following.
  • Reading through source files
  • Passing variable length parameters to the ‘C’ program.
  • Use of NULL terminated strings.
  • Ensure we pad variables to hold the NULL terminator.
  • Initializing all variable length with single character (‘\0’ which is NULL in this instance)
  • Create a qualified name from a 20 character array.
I am trying to limit the amount of information contained in each post so decided to split this functionality into 2 separate posts as time is short and reading too many lines makes it more difficult to take in. Plus it takes me longer to create 🙂

If you have any questions or comments about the code etc please let me know.

Chris…

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.