Migrating an ERP system from IBM RPG to Python

Overview

This article will provide insight into the Strumenta approach in migrating an ERP system from IBM RPG to Python based on our experience with a real project in the manufacturing industry.

Migrating a legacy system from an outdated language to a modern programming language can be a daunting task, but it can also be a necessary step for companies to stay up to date with the latest technologies and to improve their overall efficiency. 

RPG is a programming language that has been around since the 1960s and has been widely used in the manufacturing and distribution industries. 

However, it is becoming harder to find experienced developers, and the language suffers from the lack of some of the modern features and functionalities that are available in newer programming languages like Python, which has gained popularity in recent years due to its ease of use, readability, and versatility.

What is an ERP System?

An ERP (Enterprise Resource Planning) system is a software application that integrates and manages a range of business processes such as finance, human resources, procurement, inventory management, customer relationship management, and more. It provides a single platform that streamlines these processes and allows information to be available between different departments in an organization.

The architecture of an ERP system usually consists of a central database that stores all relevant information about the organization’s operations. This includes customer data, inventory levels, suppliers information, financial data, and so forth. Users can access this information in real-time and use it to make better decisions that drive productivity and efficiency.

Typically the system has several modules (or applications), each designed to handle specific business processes. There may be a finance module that handles accounting and financial reporting, a supply chain module that tracks inventory levels and manages supplier relationships and so forth.

The system allows to collect data from users or other systems such as an e-commerce website,    which is stored in the central database. This data is then processed and analyzed by the system, which generates reports and provides information that help users make decisions. The system can also automates many routine tasks, such as data entry, invoicing, and order processing, reducing the workload and increasing productivity.

Many organizations have developed custom programs over many years that are used to analyze data and automate tasks, representing the business logic of the company. These processes are highly efficient in handling a wide range of business use cases specific to the company’s business and structure.

These programs represent a valuable asset for the organization, and it is critical to ensure that translation in another language will preserve the same exact functionality.

The Migration

Our migration process starts with an analysis phase organized in workshops with the customer’s developers and business analysts. The migration process involves several steps, the objective is to identify how the available functionalities need to be migrated, the scope, the complexity and the critical part of the system.

The analysis of the current system typically includes the following steps:

1. Identify the key functionalities 

The first step is to identify the key functionalities of the current ERP system. This involves understanding the business processes that the system supports, and the features and capabilities that are required to support those processes. This information is collected during the workshops with developers and end-users and by reviewing the code, system and business requirements.

2. Create an inventory 

Once the key functionalities have been identified, the next step is to create an inventory of the current system. This includes all the programs, databases, and interfaces that are currently in use. The inventory includes details such as the purpose of each program, the data it processes, and the interfaces it uses.  

3. Evaluate the current system 

With the inventory in hand, the next step is to evaluate the current system. This involves reviewing the current code base, identifying any inefficiencies, code pattern, areas for improvement, and determining how well the current system will be translated in the target language. At Strumenta we make use of custom tools based on our parsers to process the source code. For example, to analyze the complexity and the dependencies of an RPGLE program, we make use of an internal tool called RPG2SEQ which generates a sequence diagram representing the code execution, the files it accesses,  and the external dependencies. The diagram below is an example of such a diagram. 

From the diagram it is possible to extract useful information such as the complexity of the program, the files involved (FILE1,FILE2,FILE3,FILE4), the operation of the files (READ,UPDATE), the invocation of subroutines ($Subroutine1,$Subroutine2,etc) and an external program (PROGRAM2.rpgle). In this particular project we have also to migrate the existing DDS (Data Definition) into SQL Tables. We developed a DDS2SQL transpiler to convert the Logical and Physical files into  SQL statements  to create the tables and the views required.

Our RPGLE parser is able to parse the DDS and create the AST representation of the data, the transpiler implements a AST transformation to translate the DDS AST into a SQL AST and a code generator walks the SQL AST to produce the output code. 

A physical file in RPGLE corresponds to the SQL CREATE TABLE statement that has the same fields and attributes as the physical file. Here’s an example SQL statement that creates a new SQL table based on a physical file MYFILE.

The Strumenta’s DDS2SQL transpiler automates this job resolving all the references to other files and creating a single DDL file in several SQL dialects, which is a very effective solution when there are hundreds of files that must be processed.    

In addition, we make use of our RPG parser to perform static code analysis, to ensure that the codebase is well-structured, efficient, and secure, and that the migration process goes smoothly. These tools can help to identify potential issues early in the process, reducing the risk of bugs or other problems down the line, and ensuring that the new system is of high quality and meets the needs of users.

Code statistics refer to the use of statistical methods and analysis to evaluate the codebase of a software project. It involves collecting data on various aspects of the code, such as the number of lines of code, the number of functions, the number of files, and other code metrics, and then analyzing this data to gain insights into the quality and maintainability of the code.

Code statistics provide a wealth of information about the codebase, including:

1. Code complexity: Code statistics can be used to measure code complexity which is a measure of how difficult the code is to understand, translate, and modify. 
2. Code quality: Code statistics can be used to measure the quality of the code, such as the number of bugs or defects in the codebase. This information can be used to identify areas of the code that need improvement and to prioritize bug fixing efforts.
3. Code maintainability: Code statistics can be used to measure the codebase’s maintainability, such as the number of comments in the code, the ratio of code to comments, and the average length of functions or methods. These metrics can provide insights into how easy it is to maintain and modify the code.
4. Code patterns: Strumenta’s code statistics tools allow to identify patterns present in the code and perform refactorings to eliminate code duplication.

By analyzing code statistics with the customer during workshops, language engineers, developers and project managers can gain insights into the complexity of the codebase, identify 

the areas of the code that need improvement, and prioritize transpilation efforts. This information can be used to ensure that the codebase can be translated into the target language. 

The table is below an example of statistical analysis.

This example aggregates the RPG statements in categories to evaluate what statements are the most frequent and how much are they used in the code base. In this case translating the three statement categories will cover about 83% of the total code base.  

4. Identify potential issues 

Based on the evaluation of the current system, potential issues can be identified. These can include outdated technologies, inefficient code, unsupported hardware or software, and security vulnerabilities. In this particular RPGLE to Python project we identified cases in which the code referred to platform specific  system information of the IBM RPGLE runtime environment. That information is not available in the Python runtime, at least not in the same way, in conjunction with the customer we designed the solution of how the code should be translated in these particular cases.      

5. Develop a migration plan 

Based on the inventory, evaluation, and identification of potential issues, a migration plan can be developed. The plan outlines the steps required to migrate the system from IBM RPG to Python, including timelines, budget, and resources required. It should also identify any risks or issues that may arise during the migration process, and outline strategies for mitigating those risks. All information is collected into the report that Strumenta delivers to the customer at the end of the analysis phase, which is the migration plan.  

6. Code Translation 

Migrating RPGLE code to Python can be a challenging task for developers, especially when dealing with complex business logic and database integration. Transpilers are software tools that can convert RPGLE code into Python code by analyzing the syntax and semantics of the original code and translating it into Python. These tools can save a lot of time and effort for developers who would otherwise need to manually rewrite the code, and they can also help to reduce the risk of errors and inconsistencies in the migration process.

One of the benefits of using automatic migration tools is that they can preserve the functionality and business logic of the original RPGLE code, while also taking advantage of the modern features and libraries available in Python. For example, Python has a rich set of data analysis and visualization libraries that can be used to process and present data in ways that were not possible in RPGLE.

Code that relies heavily on platform-specific features or external dependencies may require significant modifications or manual rewrites. Additionally, the quality and readability of the resulting Python code may vary depending on the complexity of the original RPGLE code and the quality of the transpiler.

7. Testing

Testing is an essential part of any software development process, it  helps to ensure that the new system functions as intended, is free from bugs and errors. There are several types of testing that can be conducted during the migration process, including unit testing, end-to end testing, system testing.

Unit testing involves testing individual units or components of the system, such as functions or modules, to ensure that they function as intended. This type of testing is typically automated and can be conducted using testing frameworks like PyTest or unittest.

End-to-end testing involves testing the interaction between different components of the system, such as different modules or databases, to ensure that they work together as intended. This type of testing is typically conducted manually and may involve creating test cases that simulate real-world scenarios.

System testing involves testing the system as a whole to ensure that it meets the requirements and specifications of the project. This type of testing can involve both automated and manual testing and may include stress testing, security testing, and performance testing.

During the testing process, it is important to document any bugs or errors that are discovered and to track them through to resolution. This helps to ensure that the system is fully tested and provides the same functionality of the original RPGLE code.

The Strumenta’s RPG to Python transpiler can automatically generate the skeleton for the end- to-end testing. The diagram below illustrates the sequence of the execution steps.

The test case loads the database with the test data and verifies the precondition before running. A common precondition for all tests is to verify if the database has been loaded successfully and data is available for processing. The next step is to execute the program under test, which updates the database.  At the end of the execution stage, the control goes back to the Test case that verifies the postconditions.  The postconditions can include performance requirements, in that case the execution time of the program under test is recorded and compared to the performance requirement.

Challenges in Real World Projects

Although the process takes in consideration many of the common challenges, every project is unique and unexpected cases must be properly managed.  

For example one of the steps the analysis performs on the RPG code is the identification of unstructured code such as GOTO statements. In this particular project the RPG developers have been very disciplined and we found very few cases. The Strumenta’s RPGLE to Python transpiler can automatically identify code patterns involving GOTO statements.

The example below is a common case handled automatically by the transpiler. 

The RPG code is a continue-goto pattern which means that there is a single label at the beginning to jump in certain conditions.

In this case, the goto statements are replaced by the transpiler with a Python while loop and a continue statements.

Although the Strumenta Transpiler can automatically eliminate GOTO statements the customer team decided to eliminate the statements from the original RPG code.

As mentioned earlier another challenge was how to implement the platform specific system information used by the programs. In real word projects it depends on the target frameworks and deployment environment, and configuration chosen by the customer. In this case, the code referred to the program status data structure (PSDS) that  makes system information available to an RPG program. On the target platform those information could be stored in environment variables and instruct the transpiler to generate the Python statements required to read the variable value at runtime; another option would be to store those values in a configuration file. There is not one right solution for this, it really depends on the target environment and customer preferences.

Another challenge identified during the code analysis was the use of the READP RPG   statements (read previous) which allows reading a file backwards. As mentioned before files have been translated to SQL tables and the Python Database API Specification v2.0 does not include a method to read the previous record. As part of the RPG-runtime for Python classes to provide such functionality has been implemented using the pagination and implementing the transpiler to invoke the proper method when positioning and reading the database table. 

Conclusion

 This article highlights the challenges companies face when migrating a legacy system from an outdated programming language to a modern one. It emphasizes the benefits of such a migration, including staying up-to-date with the latest technologies and improving overall efficiency. The article specifically focuses on Strumenta’s approach and tools to migrate an ERP system from IBM RPG to Python, based on a real project in the manufacturing industry.