Finally managed to get Bob IDE working on a Ubuntu PC so thought I would put it to work with a simple task. For those who just want the headlines, didn’t go so well even though I hoped it would.
Simple C Program
Wanted to test the ability to generate a simple C program that would use the available API’s to retrieve a list of objects from a library and display them to StdOut from the command line. Here is the initial question I posed:
create a c program that uses the API QUSLOBJ to list object in a library to stdout.
It first asked if it could scan through my directory structure looking for something that would use the QUSLOBJ API that it could review, as I have no other source on this system I refused the request. It does not connect to the IBM i at any time, maybe if it had done that the exercise might have provided better results? it then asked for more information about the type of information I wanted plus the level of error handling. This was my response:
IBM i system – List all objects in a library passed as parameter – Show name, type, and attribute – Basic error handling
So far so good I thought, it was asking the right questions and giving me some prompts for the data required. So this is the plan it came up with to generate the program:
- Research and document QUSLOBJ API structure and requirements
- Define the program structure and required IBM i headers
- Implement user space creation for API output
- Set up QUSLOBJ API parameter structure
- Implement the QUSLOBJ API call with proper error handling
- Implement user space data retrieval logic
- Parse the object list entries from the user space
- Format and display object information to stdout
- Add command-line argument parsing for library name
- Implement cleanup and user space deletion
- Add comprehensive error handling and messages
- Test the program with various library names
All looks very good, the fact that it was going to research the QUSLOBJ API and use the required IBM i headers was great! It understood that it would need to set up the *USRSPC for the data retrieval and parse the object list from the User space. This was exactly what I wanted to see. So I approved and it generated the first pass of code. the code was incorrect and would not have worked for the following reasons:
- It built its own structures for addressing the output (should have used the defined headers). Even after many rejections and passing in additional information
- The headers it built were totally incorrect as they had content which was not available in the format requested.
- Use the wrong format for the request, it was using data that did not exist in that format. Even after I corrected and told it which format to use it still failed to address the data correctly.
- It recognized the need to use qusgen.h header but then addressed elements that did not exist in the defined structure (Exception Data).
- Failed to call the API correctly, it used the ASP optional parameter, used the wrong data format for the parameter and did not include the previous optional parameters before the ASP selection parameter.
- Used 16K for the Userspace size which considering that about 500 bytes are used up for the headers leaves only enough room for less than 200 objects to be listed. It also took no account for incomplete/ partial data where there were more items available than could fit into the provided space. It specifically noted that it would be able to handle large libraries.
- Used some irrelevant code for dealing with the input, but simple to cut out and fix with some additional knowledge.
- Even though it said it would address the data directly in the UserSpace, it built code that would retrieve the data in chunks to memory that was malloc’d to the size of each list entry (also wrong size). Did rewrite after mentioning this but used incorrect header offsets.
- Built a makefile, not really useful for IBM i as there is no make command. CLP program to do the build would have been better.
Note: We did have to keep changing things after each iteration, we have not added all of the information we kept feeding back into the chat as it was hard to cut and paste into a doc (Bob kept reacting and recreating the code). We took about 4 hours to get the code we ended up with (lots of research and code review), but that code would not have compiled.
Thoughts!
I had to spend a lot of time going through the code, checking the manuals and reading the actual header files to see where it was wrong. My hope is that it will learn from its mistakes and do better next time round.
The code would not have compiled even after many iterations of passing back corrective actions (using my own knowledge to push it in the right direction).
Bob has full confidence in itself, the conversation flow also gave me confidence it knew what it was doing. Yet we found it provided poor code during the reviews. You have to remember that unlike human programmers, it does not care about being wrong….
We have written a lot of code that uses API’s such as these, maybe a cut and paste process using those files would have been a lot quicker? Maybe we should have been a lot better at giving it the rules that we take as standard when writing for the IBM i, these have come from years of making mistakes and figuring out why and then correcting it. Is this where Bob is heading? maybe.
I am still very hopeful for the future of the Bob project, it can certainly cut out a lot of the mundane tasks associated with general programming tasks, the data flow diagrams and other ancillary output is also a plus as it helps with future maintainability.
Conclusion…
I still think we have value as programmers beyond building code.