How to Migrate CL Programs

For any large-scale migration off the IBM i platform, the RPG business logic is often the main focus. Yet, the entire system is orchestrated by Control Language (CL), and modernizing these scripts is a crucial enabler for the whole project. You can’t move the applications without the scripts that govern them. This guide tackles the specific challenges of CL migration—from its tight coupling with RPG to its reliance on OS functions and GOTO logic—ensuring your broader modernization strategy succeeds.

Can You Migrate Just CL Programs?

The IBM i system is an integrated system. It comes with its own database management system (DB2), its own programming languages (e.g., RPG) and even its own scripting language: Control Language (CL to its friends).

So, CL is managing the whole system. The issue is that CL programs are typically used in an existing system to orchestrate the execution of RPG programs. They contain the logic to decide which programs to run and to pass arguments to these programs. Therefore, they are inextricably linked to RPG programs, and they are meant to work, or be migrated, together.

So, you are typically thinking of migrating the overall software system, including RPG and CL programs, rather than just CL programs. However, it does make sense to look at the specific technical issues related to migrating CL programs.

That is why in this article we are not addressing why you should migrate, we have written plenty of articles on this and the overall picture of a migration:

Here we concentrate on specific issues related to migrate CL programs.

CL is More Powerful Than Your Typical Scripting Language

Although CL shares similarities with scripting languages like Bash, it is fundamentally more powerful, with a scope and level of system integration that is more comparable to PowerShell. To understand the specificity of CL, let’s see what CL programs can do.

They can:

handle error routines (e.g., typically jumps to labels with GOTO statements)
interact with the job queue
monitor results from program runs
run queries (i.e., a tool of the IBM i platform to generate reports)
execute batch processing
check and set up data used by programs
…and more

The short version is that CL programs functionally are similar to bash scripts. They can execute procedural programming as well, but in practice their integration with the IBM i system makes them effectively more powerful. It enables them to access operating system features, like the running SQL statements or accessing a Data Area. Therefore, a successful migration requires replicating these OS-level features in the target language. So, you need to work on a hefty runtime library.

In addition to that expressive power, there are the design issues typical of an older language, namely GOTO statements. The thing is that GOTO statements are usually absent in any language you want to migrate to, so you need to address this design difference between your source and target languages. This can be a big problem because it is not always possible to refactor arbitrary GOTO statements in other statements supported by the target language.

When Two Problems Make One Solution

We have seen that the main issues related to migrate CL programs are:

Coupling with programs written in RPG
Accessing or replicating features available exclusively on the IBM i system
Transpiling GOTO statements

The second issue can be somewhat mitigated by the first one. You can reduce the work needed to replicate IBM i features by using one target language for both RPG and CL programs. You will need to replicate IBM i features both to transpile RPG and CL programs, so if you transpile them to one language, you will need to do the work just once.

This is a reason we have seen more clients picking Python as the target language of their choice. Java could also be an excellent choice to transpile RPG programs, but Python is a more natural choice to replicate CL (scripting) programs. Therefore, it makes sense to transpile both RPG and CL to Python.

An Iterative or One-shot Migration?

Once you pick a target language, you might wonder whether it’s better to do an iterative migration, one program at a time while keeping the IBM i system active, or a one-shot migration, migrating all the programs at once. If you are looking at the most efficient way, the answer is easy: the one-shot migration requires a concentrated but overall smaller effort.

There are good reasons to choose an iterative migration, though. For once, it is safer and you risk less downtime. You can go at your own pace, double-checking and testing everything you want. There is an obvious problem, though: you need to deal with the second issue: replicating features of the IBM i system. Presumably doing that requires a big effort and the solution might not be compatible with the original approach. For example, the Data Area:

A data area is an object used to hold data for access by any job running on the system.

A data area can be used whenever you need to store information of limited size, independent of the existence of procedures or files.

You can create a runtime environment to replicate this functionality in, let’s say Python, but how can you make other non-migrated CL or RPG programs access it? There is no easy way to do it. You will need to change the remaining CL or RPG programs to access it. That rarely make sense, so you will need, at minimum, to migrate each cluster of CL and RPG programs interacting with the same Data Area in one go. This is better than migrating everything at once, but less than ideal, if you are hoping for an iterative migration.

Luckily, thanks to XMLSERVICE, software provided by IBM, an iterative migration is doable. It would still require more effort than a one-shot migration, but it is a viable option.

The Typical Scenario

Consider a common scenario: you plan to eventually migrate completely off the IBM i platform but need the system to remain active for some time. This phased approach is practical when your existing hardware has years of remaining usable life, allowing you to maximize its value before the final transition.

During the transition: While your applications are split between the IBM i and the new platform, you retain native IBM i features (e.g., data areas). Migrated programs access these legacy components using XMLSERVICE.

After the transition: Once all programs are off the IBM i, you replace the legacy components with modern alternatives and remove the XMLSERVICE dependencies. When the new system is fully independent, the IBM i can be shut down.

This is the typical use case of XMLSERVICE. However, you can also use it indefinitely to access IBM i features from other languages, like Python or C#. You just migrate CL programs into Python and keep using the IBM i platform.

What is XMLSERVICE?

Let’s start with explaining what is XMLSERVICE, by looking at the documentation:

XMLSERVICE library is simply a collection of Open Source RPG modules that essentially allow you to access anything on your IBM i machine, assuming proper profile authority. Simply stated, XMLSERVICE accepts XML document containing actions/parameters (<pgm>,<cmd>,<sh>,<sql>,etc.), performs the requested operations on IBM i, then sends an XML document of results back to the client.

It basically gives you access to IBM i features by sending XML documents back and forth. There are also language libraries available to access it without needing to learn the XML interface:

.NET
Node.js
PHP
Python
Ruby
Swift

Let’s see an example using itoolkit, the Python interface for XMLSERVICE: you want to run the WRKACTJOB CL command from Python, that is how you can do it.

from itoolkit import *

from itoolkit.transport import DatabaseTransport

import pyodbc

conn = pyodbc.connect("DSN=*LOCAL")

itransport = DatabaseTransport(conn)

itool = itoolkit()

itool.add(iCmd('wrkactjob', 'WRKACTJOB'))

itool.call(itransport)

wrkactjob = itool.dict_out('wrkactjob')

print(wrkactjob)

The availability of itoolkit simplifies the transpilation of CL program to Python and ensures the ability to reproduce the same behavior, given that we are relying on the same IBM i system functions used by CL programs. You are using the same IBM i runtime that your remaining RPG and CL program still use, so you do not risk running into incompatibility issues. This makes possible to pick an iterative migration.

Limitations of XMLSERVICE

XMLSERVICE is not a magic solution, though, it has some limitations. The first one is probably obvious, but it is worth mentioning it: it must be installed, configured and run on an IBM i system. This implies that you need to keep a running IBM system, and you need to manage XMLSERVICE.

This is not a big ask if you are choosing a slow migration, but it is something to keep in mind for the long term. You can access XMLSERVICE from outside the IBM i system, but you need to have an IBM i system running.

The major limitations are clear when you understand that XMLSERVICE essentially is a way to perform a remote procedure call: this approach is not suited to interactivity since the context of the call is not kept.

Handling Errors

PGM
MONMSG   MSGID(CPF0001  CPF1999)  EXEC(GOTO  ERROR)

This is a typical example of the impact of the limitations of XMLSERVICE. The CL code checks whether any programs have sent the specific messages CPF0001 and CPF1999,. If so, it jumps to the label ERROR. This is a common problematic pattern to translate in another language.

You can’t use XMLSERVICE and you have a GOTO to handle. This is not ideal, but this is contextually manageable: you want to change how you handle errors, using a more modern pattern. This can be done, because basically all the CL programs we have examined use GOTO like this:

PGM

MONMSG   MSGID(CPF0001  CPF1999)  EXEC(GOTO  ERROR)

[..]

/* Error routine */

ERROR:

     [..]

/* Finally-equivalent, cleanup code */

EXIT:

    ENDPGM

This combination of MONMSG and GOTO has a neat equivalent in a try..catch block. This shows both how to overcome the limitations of XMLSERVICE and dealing with GOTOs.

In short, statements like MONMSG (that monitors a message returned by a program) or DSPJOB (that shows information of a current Job), are not supported, but there are ways to deal with these limitations.

Running Shells and System Management

QSH CMD('/QOpenSys/pkgs/bin/bash example.sh')

The second set of problems we have often encountered are commands run with a shell. For example, QSH, which starts an interactive qsh session. As you can imagine by the interactive adjective, this can be a problem for XMLSERVICE. Well, is it actually a problem? It depends on the actual command you are running and the parts of the system you are touching.

If the program doesn’t require interaction, it’s okay. There are can be many details and specificities to address, though. For instance, imagine you are using SFTP to upload a file somewhere, and you have set up Public Key Authentication. This isn’t interactive, but the authentication has been configured on the IBM i system. So, you either need to create a system to run all programs under the specific user for which the authentication was set up (not ideal for security), or use the native shell on your new platform and set up the authentication there.

In practical terms, especially if your long-term plan involves leaving the IBM i platform, it makes sense to change all calls to the shell with a native solution on your new platform, rather than relying on XMLSERVICE. More than a code issue, it is a system administration issue: it is better to gradually do the sysadmin work to manage the new system rather than doing all the work at the end.

Let’s look at the example at the beginning of this section, and assume you were planning to migrate on Linux. You could just use something like this code:

import subprocess

result = subprocess.run(['./example.sh'])

Of course, that would require to have the bash script example.sh on your Linux machine, accessible from the Python program that should launch it.

Transpiling CL Code Into a Idiomatic Code

While a library like itoolkit allows solving the important problem of implementing features in Python, it leaves us with the issue of making the code not Pythonic, i.e. hard to understand and maintain for Python developers. We are using Python for our example, but this is valid for all target languages. Adherence to shared standards is important for readability and productivity particularly in all programming languages, even if it might be felt more by the Python community.

This isn’t simply an issue of style or reduction of productivity. A series of calls to itoolkit functions, while effective in reproducing the behavior of a CL program, requires an understanding of CL behavior and structure.

In simple terms, seeing Python code that just calls CL commands is as hard to understand and write as writing a CL program directly. This is bad because few developers know a language like CL; the obscurity of the language is one of the reasons to move off the IBM i platform.

It is also bad because it is unproductive. It would be like interacting with a database using strings containing SQL statements, rather than using an ORM library. This would require knowing SQL and you would lose all the benefits of tools designed for writing Python code, like autocompletion or static analysis, since you would effectively be writing just strings.

Wrapping XMLSERVICE Calls

Let’s see an example of the whole transformation.

Original code

DLTF FILE(MYLIB/MYFILE)

Direct itoolikit code

from itoolkit import *

from itoolkit.transport import DatabaseTransport

import pyodbc

conn = pyodbc.connect("DSN=*LOCAL")

itransport = DatabaseTransport(conn)

itool = itoolkit()

library = 'MYLIB'

file_name = 'MYFILE'

dltf_command_string = f'DLTF FILE({library}/{file_name})'

itool.add(iCmd('dltf', dltf_command_string))

itool.call(itransport)

result = itool.dict_out('dltf')

Wrapped itoolkit code

result =  DeleteFile('MYLIB', 'MYFILE')

The solution is to wrap itoolkit calls with more natural Python function calls. Again, itoolkit is specifically the Python library, there are similar ones available for many other languages.

By doing so, the transpiled code will contain calls to functions in your target language, like DeleteFile() and will come with a runtime implementing these functions. The implementation of DeleteFile (DLTF in CL) will rely on itoolkit, but developers of Python scripts will not need to know such detail.

This approach will allow Python developers to understand and maintain existing CL-transpiled programs and also develop new Python programs in a Pythonic way. The code will always look like Python code and will also allow using native Python code and functions and thus have a smooth transition to a Python codebase.

Using itoolkit allows you to preserve the functionality of things like data areas in both the transpiled CL and RPG programs. So, you can start with your transpiled CL-program in Python calling the original RPG program using XMLSERVICE. When you then decide to transpile the original RPG program in Python, the original CL program’s call to the RPG program is translated, the resulting code is a new call that executes the Python program (the transpiled RPG).

This eventually allows you to evolve the code even further towards the idiomatic Python way.

You will not call an external program, but a Python function perfectly integrated in your code.

How Wrapping itoolkit Calls Can Help Transition

You could also replace, for instance, the usage of the data area using another medium to store and pass information from the orchestrating program to the program itself. That is how you would organize the passage:

During migration, the wrapper methods will use itoolkit to call IBM i system functions using data areas
After migration, the wrapper methods will use another technology to pass data between programs. For example, you could use Redis

The advantage of this approach is that there is no rush and no need to make all of these changes immediately. It would be possible to keep using the transpiled Python programs for as long as necessary, but there would be the possibility to evolve the overall Python code towards a more natural form.

For example, we can translate the CL command DLF into the Python function DeleteFile(). The initial implementation of DeleteFile() could rely on itoolkit and IBM system functions to actually delete the file. However, any Python developer can change the code implementing DeleteFile() to a version that doesn’t use itoolkit and can run on any Linux platform.

Handling GOTO Statements

The third large issue when transpiling CL programs to another language is how to handle GOTO statements. Most likely your new language will not support it. This is because our programming community has come to the conclusion that GOTO is too powerful to be safe: when you can jump basically anywhere in a program, it becomes very difficult to build safe, reliable and memory-efficient code.

The good news is that in large part what was expressed using GOTO can be expressed safely without it: GOTO was used to implement loops, handle exceptions, default choice for a series of if statements (this became a default case in a switch), etc. So, the good patterns of GOTO usage are preserved: you can replace such good uses of GOTO with equivalent statements in a new language. We have seen an example of using GOTO to handle exceptions before. This is an example of using a GOTO to implement a loop.

LOOP:  CHGVAR &A (&A + 1)
IF  (&A *LT 30)  THEN(GOTO  LOOP)

This snippet of code will increase the value of the variable &A by one until it reaches 30. It’s easy to see how you can convert it to a while or a for loop.

a = 0
while a < 30:
   a += 1

So, you can typically convert good (structured) uses of GOTO with equivalent modern statements. But what about bad (unstructured) uses of GOTO, those for which there is no equivalent? Well, the best way would be to get rid of them, for the obvious reason that they result in bad code. However, it could be difficult or impractical to do it while you are migrating a large codebase.

You will have to study each use and find a solution on a case-by-cases basis. For instance, code such as this.

OUTER_LOOP:
   ... (some logic)
   DOFOR VAR(&A) FROM(1) TO(10)
       DOFOR VAR(&B) FROM(1) TO(10)
           ... (some logic)
           IF COND(&C *EQ &D) THEN(DO)
               CHGVAR VAR(&E) VALUE('Condition met')
               GOTO END_LOOP
           ENDDO
       ENDDO
   ENDDO
END_LOOP:
   ... (post-loop logic)

This use of GOTO would be difficult to reproduce as-is in Python, since you’d need to break out of both loops at once. However, you can solve this case by transforming this snippet into a function and thus gaining the ability of using return instead of break.

def find_condition():

    for a in range(1, 11):

        for b in range(1, 11):

        # ..some logic

            if c == d:

                print("Condition met")

                return # Exit the function immediately

# Call the function

find_condition()

# ... (post-loop logic)

So, you might be able to study your specific case and get away with some modifications. In the cases we have seen for our clients, the usage of GOTO is often standard and the exceptions easy to handle in CL programs, given that they are typically simpler programs. RPG programs can open entire cans of worms, though, but that is an issue for another article…

Conclusions

In this article we have seen the major issues to deal with when transpiling CL programs into a modern language:

Coupling with programs written in RPG
Replicating features available exclusively on the IBM i system
Transpiling GOTO statements

And we have found some practical strategies to deal with them:

Finding one target language for both languages
Using XMLSERVICE to reduce the cost of replicating features of IBM i, and also enable a smoother transition to the new language
Identify and use equivalent modern statements

If you are interested in learning more about the big topic of migrations, we have a whole host of articles on application modernization for you to read.

How to Migrate CL Summary

To migrate CL programs from IBM i presents key challenges: tight coupling with RPG, reliance on system-specific features, and archaic GOTO statements. This article outlines a practical strategy using a unified target language like Python, leveraging XMLSERVICE for a gradual transition, and refactoring legacy code into modern, maintainable programming constructs.