Let’s ‘C’ using TLS to the web.

Its been a long few weeks since the last real techie post. We had a number of issues with the Secure Sockets program talking to our externally hosted website which took a lot of debugging to finally figure out what was going wrong!  The post this week will cover updates to the HTTPGET program we wrote originally to pull data back from a web site to add functionality that allows it to be carried out over a secure socket.

Before you start you should be aware of a few requirements, the Secure Sockets API’s we are using require the installation and set up of DCM. this is because they need to verify the certificates returned from the web sites you contact. You can overrule this requirement and just go for a secure socket connection without verification, but in today’s environment we suggest you use the full power of TLS to ensure the site you are contacting is actually who you expect it to be. Once you have DCM installed and set up you should be good to go. You will also notice that we check for the installation of DCM is a secure connection is requested. The API’s we chose for Secure Sockets (GSK Global Secure Toolkit) are pretty easy to use and understand so if you need more information check out the documentation.

There are a number of new Service program updates and new service programs to support the secure connectivity so if you have already built the programs previously you may want to adjust the build process to take that into account. Here is a new service program that we are going to use.
//
// Copyright (c) 2018 Chris Hird
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions
// are met:
// 1. Redistributions of source code must retain the above copyright
//    notice, this list of conditions and the following disclaimer.
// 2. Redistributions in binary form must reproduce the above copyright
//    notice, this list of conditions and the following disclaimer in the
//    documentation and/or other materials provided with the distribution.
//
// Disclaimer :
// This code is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

#include <H/GENFUNC>                        // General functions Hdr
#include <H/MSGFUNC>                        // Message functions
#include <H/COMMON>                         // common header


// function get_lpp_status()
// Purpose to retrieve the status of a LPP
// @parms
//      LPP
// returns status

int get_lpp_status(char *lpp,
                   char *ver,
                   char *opt) {
char msg_dta[_MAX_MSG];                     // message buffer
Qsz_Product_Info_Rec_t Prd_Inf;             // Prod Info Struct
Qsz_PRDR0100_t Prod_Dets;                   // returned data
Os_EC_t Error_Code;                         // error struct

Error_Code.EC.Bytes_Provided = sizeof(Error_Code);

memcpy(Prd_Inf.Product_Id,lpp,7);
memcpy(Prd_Inf.Release_Level,ver,6);
memcpy(Prd_Inf.Product_Option,opt,4);
memcpy(Prd_Inf.Load_Id,"*CODE     ",10);
QSZRTVPR(&Prod_Dets,
         sizeof(Prod_Dets),
         "PRDR0100",
         &Prd_Inf,
         &Error_Code);
if(Error_Code.EC.Bytes_Available > 0) {
   if(memcmp(Error_Code.EC.Exception_Id,"CPF0C1F",7) == 0) {
      sprintf(msg_dta,"Product %.7s - %.6s - %.4s not installed",lpp,ver,opt);
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      return 0;
      }
   else {
      snd_error_msg(Error_Code);
      return -1;
      }
   }
if(memcmp(Prod_Dets.Symbolic_Load_State,"*INSTALLED",10) == 0) {
   return 1;
   }
return 0;
}

// function reg_appid()
// purpose: register application with DCM
// @parms
//      Application ID
//      Application description
// returns 1

int reg_appid(char *appID,
              char *appDesc) {
int appIDLen = 0;                           // app ID Length
int appDescLen = 0;                         // app Description Length
char msg_dta[_MAX_MSG];                     // message buffer
Ctl_Rec_t AppCtls;                          // control record
Os_EC_t Error_Code;                         // error struct

Error_Code.EC.Bytes_Provided = sizeof(Error_Code);
appIDLen = strlen(appID);
appDescLen = strlen(appDesc);
// number of keys non default
AppCtls.numRecs = 4;
// application type
AppCtls.appType.size = sizeof(_Packed struct App_Type_x);
AppCtls.appType.key = 8;
AppCtls.appType.dtaLen = 1;
AppCtls.appType.dta = '2';
// application description
AppCtls.appDesc.size = sizeof(_Packed struct App_Desc_x);
AppCtls.appDesc.key = 2;
AppCtls.appDesc.dtaLen = 50;
memset(AppCtls.appDesc.dta,' ',50);
memcpy(AppCtls.appDesc.dta,appDesc,appDescLen);
// certificate trust
AppCtls.caTrust.size = sizeof(_Packed struct CA_Trust_x);
AppCtls.caTrust.key = 4;
AppCtls.caTrust.dtaLen = 1;
AppCtls.caTrust.dta = '0';
// replace existing cert
AppCtls.certRpl.size = sizeof(_Packed struct Cert_Rpl_x);
AppCtls.certRpl.key = 5;
AppCtls.certRpl.dtaLen = 1;
AppCtls.certRpl.dta = '1';
// register
QsyRegisterAppForCertUse(appID,
                         &appIDLen,
                         (Qsy_App_Controls_T *)&AppCtls,
                         &Error_Code);
if(Error_Code.EC.Bytes_Available > 0) {
   snd_error_msg(Error_Code);
   return -1;
   }
return 1;
}
There are 2 new functions, the first one simply checks for the installation status of an IBM LPP, this can be used to verify the installation of any IBM LPP but for this instance we are going to use it to verify that DCM (5770SS1 Option 34) is installed. The next function is used to register an Application in DCM, this allows you to determine the supported Protocols (SSL/TLS), encryption ciphers etc. We will state that the programs which use this Application ID are going to set most of the information in the program. You can set this application ID up in DCM and set the appropriate values, but as we wanted this program to be self managed without having to have to manually go through the DCM set up we set all of the values to be program defined. This program is just setting the basics so be prepared to improve the code to meet your production needs.

Next we updated the HTTPGET code to allow a secure connection to be made, you will remember we already had the ability to determine from the URL passed in whether this was a secure site or not (parse_url function). If you attempt to contact a site using ‘HTTP’ and the site has redirection set from HTTP to HTTPS the program will simply return the redirection message and go no further, it would be a simple update to look for the return code associated with that and step into a HTTPS request.

Here is the new HTTPGET code.
//
// Copyright (c) 2018 Chris Hird
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions
// are met:
// 1. Redistributions of source code must retain the above copyright
//    notice, this list of conditions and the following disclaimer.
// 2. Redistributions in binary form must reproduce the above copyright
//    notice, this list of conditions and the following disclaimer in the
//    documentation and/or other materials provided with the distribution.
//
// Disclaimer :
// This code is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

#include <H/IPFUNC>                         // IP functions
#include <H/COMMON>                         // common header
#include <H/MSGFUNC>                        // message functions
#include <H/SRVFUNC>                        // Server functions
#include <H/GENFUNC>                        // General functions
#include <iconv.h>                          // conversion header
#include <qtqiconv.h>                       // iconv header
#include <errno.h>                          // error number
#include <gskssl.h>                         // secure sockets toolkit

int gsk_clean(gsk_handle my_env_handle,
              gsk_handle my_session_handle,
              url_t *parsed_url) {
if(my_env_handle != NULL)
   gsk_environment_close(&my_env_handle);
if(my_session_handle != NULL)
   gsk_secure_soc_close(&my_session_handle);
free_mem(parsed_url);
return 1;
}

int main(int argc, char **argv) {
int sockfd = 0;                             // socket
int server_port = 80;                       // server port defaults to http
int rc = 0;                                 // return counter
int secure = 0;                             // secure session initiated
int amtRead = 0;                            // secure read
int amtSent = 0;                            // secure send
int out = 0;                                // init output val
int snd_len = 0;                            // send length
char msg_dta[_MAX_MSG];                     // message data
char req[2048];                             // maximum allowed request 2048 bytes
char recv_buf[_32K];                        // receive buffer
char convBuf[_32K];                         // conversion buffer
char _LF[2] = {0x0d,0x25};                  // LF string
char appID[50];                             // application ID
char appDesc[50];                           // application description
QtqCode_T jobCode = {0,0,0,0,0,0};          // (Job) CCSID to struct
QtqCode_T asciiCode = {819,0,0,0,0,0};      // (ASCII) CCSID from struct
iconv_t a_e_ccsid;                          // convert table struct
iconv_t e_a_ccsid;                          // convert table struct
url_t parsed_url = {NULL};                  // parsed url structure
gsk_handle my_env_handle=NULL;              // environment handle
gsk_handle my_session_handle=NULL;          // session handle

// we need the conversion tables for talking to the web server
a_e_ccsid = QtqIconvOpen(&jobCode,&asciiCode);
if(a_e_ccsid.return_value == -1) {
   sprintf(msg_dta,"QtqIconvOpen Failed %s",strerror(errno));
   snd_msg("GEN0001",msg_dta,strlen(msg_dta));
   exit(-1);
   }
// EBCDIC to ASCII
e_a_ccsid = QtqIconvOpen(&asciiCode,&jobCode);
if(e_a_ccsid.return_value == -1) {
   iconv_close(a_e_ccsid);
   sprintf(msg_dta,"QtqIconvOpen Failed %s",strerror(errno));
   snd_msg("GEN0001",msg_dta,strlen(msg_dta));
   exit(-1);
   }
// check the url passed in
if(parse_url(argv[1],&parsed_url) != 1) {
   sprintf(msg_dta,"URL %s is incorrectly formatted",argv[1]);
   snd_msg("GEN0001",msg_dta,strlen(msg_dta));
   exit(-1);
   }
//printf("scheme %s\nhost %s\nport %s\npath %s\nquery %s\nsegment %s\n",
//       parsed_url.scheme, parsed_url.host,parsed_url.port,parsed_url.path,parsed_url.query,parsed_url.fragment);
// check if defined port is used
if(parsed_url.port != NULL) {
   server_port = atoi(parsed_url.port);
   }
else {
   // default is 80 so only change if https
   if(memcmp(parsed_url.scheme,"https",5) == 0) {
      server_port = 443;
      }
   }
// connect to the server
if(memcmp(parsed_url.scheme,"http",5) == 0) {
   if(rmt_connect(parsed_url.host,server_port,&sockfd) != 1) {
      sprintf(msg_dta,"Failed to connect");
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   }
else {
   // need DCM installed to use API's
   if(get_lpp_status("5770SS1","*CUR  ","0034") != 1) {
      sprintf(msg_dta,"Requires DCM to be installed for Secure Sockets connections");
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   // register the application ID
   sprintf(appID,"SAS_TEST_CLIENT");
   sprintf(appDesc,"This is a test application for TLS");
   if(reg_appid(appID,appDesc) != 1) {
      sprintf(msg_dta,"Failed to register the application ID");
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   // get secure environment handle
   rc = gsk_environment_open(&my_env_handle);
   if(rc != GSK_OK) {
      // cannot set env
      sprintf(msg_dta,"gsk_environment_open() failed with rc = %d : %s",rc,gsk_strerror(rc));
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   // set the application ID to use
   rc = gsk_attribute_set_buffer(my_env_handle,GSK_OS400_APPLICATION_ID,appID,strlen(appID));
   if(rc != GSK_OK) {
      // cannot set application ID
      sprintf(msg_dta,"gsk_attribute_set_buffer() failed with rc = %d : %s",rc,gsk_strerror(rc));
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   // set the server name (otherwise may get incorrrect certificate returned)
   rc = gsk_attribute_set_buffer(my_env_handle,GSK_SSL_EXTN_SERVERNAME_CRITICAL_REQUEST,parsed_url.host,strlen(parsed_url.host));
   if(rc != GSK_OK) {
      // cannot set application ID
      sprintf(msg_dta,"gsk_attribute_set_buffer() failed with rc = %d : %s",rc,gsk_strerror(rc));
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   // set as client side
   rc = gsk_attribute_set_enum(my_env_handle,GSK_SESSION_TYPE,GSK_CLIENT_SESSION);
   if(rc != GSK_OK) {
      sprintf(msg_dta,"gsk_attribute_set_enum() failed with rc = %d : %s",rc,gsk_strerror(rc));
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   // disable SSLV3 not secure
   rc = gsk_attribute_set_enum(my_env_handle,GSK_PROTOCOL_SSLV3,GSK_PROTOCOL_SSLV3_OFF);
   if (rc != GSK_OK) {
      sprintf(msg_dta,"gsk_attribute_set_enum() failed with rc = %d : %s",rc,gsk_strerror(rc));
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   // initialize environment
   rc = gsk_environment_init(my_env_handle);
   if (rc != GSK_OK) {
      sprintf(msg_dta,"gsk_environment_init() failed with rc = %d : %s",rc,gsk_strerror(rc));
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   // connect as per normal to get socket
   if(rmt_connect(parsed_url.host,server_port,&sockfd) != 1) {
      sprintf(msg_dta,"Failed to connect");
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   // open secure session
   rc = gsk_secure_soc_open(my_env_handle, &my_session_handle);
   if (rc != GSK_OK) {
      sprintf(msg_dta,"gsk_secure_soc_open() failed with rc = %d : %s",rc,gsk_strerror(rc));
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      close(sockfd);
      exit(-1);
      }
   // associate the sock and secure session
   rc = gsk_attribute_set_numeric_value(my_session_handle,GSK_FD,sockfd);
   if (rc != GSK_OK) {
      sprintf(msg_dta,"gsk_attribute_set_numeric_value() failed with rc = %d : %s",rc,gsk_strerror(rc));
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      close(sockfd);
      exit(-1);
      }
   // start the handshake process
   rc = gsk_secure_soc_init(my_session_handle);
   if (rc != GSK_OK) {
      sprintf(msg_dta,"gsk_secure_soc_init() failed with rc = %d : %s",rc,gsk_strerror(rc));
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      close(sockfd);
      exit(-1);
      }
   secure = 1;
   }
// build the request
sprintf(req,"GET /");
if(parsed_url.path != NULL) {
   strcat(req,parsed_url.path);
   if(parsed_url.query != NULL) {
      strcat(req,"?");
      strcat(req,parsed_url.query);
      }
   }
sprintf(msg_dta," HTTP/1.1%.2sHost: %s%.2sConnection: close%.2s%.2s",_LF,parsed_url.host,_LF,_LF,_LF);
strcat(req,msg_dta);
//printf("Request : %s\n",req);
snd_len = strlen(req);
// convert to ascii
convert_buffer(req,convBuf,snd_len,_32K,e_a_ccsid);
// send off the request
if(secure == 1) {
   rc = gsk_secure_soc_write(my_session_handle,convBuf,snd_len,&amtSent);
   //printf("rc = %d amtSent = %d snd_len = %d\n",rc,amtSent,snd_len);
   if(amtSent != snd_len) {
      if(rc != GSK_OK) {
         sprintf(msg_dta,"gsk_secure_soc_write() failed with rc = %d : %s",rc,gsk_strerror(rc));
         snd_msg("GEN0001",msg_dta,strlen(msg_dta));
         }
      else {
         sprintf(msg_dta,"Failed to send request %s",req);
         snd_msg("GEN0001",msg_dta,strlen(msg_dta));
         }
      close(sockfd);
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   }
else {
   rc = send(sockfd,convBuf,strlen(req),0);
   if(rc != snd_len) {
      sprintf(msg_dta,"Failed to send request %s",req);
      snd_msg("GEN0001",msg_dta,strlen(msg_dta));
      close(sockfd);
      // need to clean up parsed url memory
      gsk_clean(my_env_handle,my_session_handle,&parsed_url);
      exit(-1);
      }
   }
// receive the response
if(secure == 1) {
   do {
      rc = gsk_secure_soc_read(my_session_handle,recv_buf,_32K, &amtRead);
      if(amtRead > 0) {
         //printf("rc = %d amtRead = %d\n",rc,amtRead);
         memset(convBuf,'\0',_32K);
         convert_buffer(recv_buf,convBuf,amtRead,_32K,a_e_ccsid);
         printf("returned data %s\n",convBuf);
         }
      } while(amtRead > 0);
   // close out the secure environment
   gsk_clean(my_env_handle,my_session_handle,&parsed_url);
   }
else {
   do {
      rc = recv(sockfd,recv_buf,_32K,0);
      if(rc <= 0) {
         break;
         }
      // convert back to ebcdic
      memset(convBuf,'\0',_32K);
      convert_buffer(recv_buf,convBuf,rc,_32K,a_e_ccsid);
      printf("returned data %s\n",convBuf);
      } while(rc > 0);
   }
// close the socket
close(sockfd);
// still need to clean up parsed url even if not secure
gsk_clean(my_env_handle,my_session_handle,&parsed_url);
return 1;
}    
The program flow is pretty much the same as it was before except we now jump into setting up a secure connection if the URL is pointing to HTTPS. You will notice that we still contact the remote server using a simple socket regardless of the protocol to be used even though the connection is called in different places. If it is to be a secure connection we then step into a new process which sets up the secure environment.
As we are going to use the application ID to determine the level of security we have to verify that we have DCM installed before we register the application ID. Once it is registered we can then set up the secure environment and link the application ID. We set the application type as CLIENT and we have stated that this client will not be sending any security verifications (you can have a set up where the server requires the client to verify who its is before it will communicate). We know SSL has a number of vunerabilities so we have decided to disable that protocol entirely in the program, please read the information about the links between TLS 1.0 and 1.1 and SSL V3 to understand why TLS 1.2 is the suggested protocol to use. Go for the higher standard at all times. We have left a lot of the secure attributes to be the defaults defined in the *SYSTEM certificate store, if you need to be more specific about the ciphers to be used etc you can set them up at this stage.
Once we have connected to the remote server we need to start the handshake process, under the covers the systems negotiate a link based on what we have said is acceptable to us and then connect the socket to a secure session. If all goes well we should now have a secure session over the socket.
If this is a secure session we cannot use the same socket API’s to read and write data over them we have to use the secure read/write API’s (the system will manage the encryption and decryption for you) but we still have to ensure we convert from EBCDIC to ASCII and vice versa.
If we start a secure connection we need to esnure we clean up the resources generated by the system to manage it, so we have created a function which will clean up the resources fo us if required. This is called at the end of the program even if it is a non secure connection as we still need to clean up the parse_url() memory.

That’s all for this update, we will be updating the GitHub repository with the changes required so the code will be available for download. If you have any questions please give us a call or contact us via the website contact form.

Chris…