"Our applications aren’t keeping up with our needs. Operating system and file size limits are appearing on the horizon. Can you help us ?" As the owner of a consulting firm that specialized in keeping those applications running, I had been hearing these questions from nearly all of our clients for some time.

The traditional COBOL and CICS applications our clients had been using for the past five years were coming up against constraints that seemed insurmountable. It appeared that regardless of the platform in use - mainframe, UNIX or PC - the COBOL systems that had been built, tuned and made to work just right were about to be written off !

Our clients were running traditional ‘CICS in the day’ and ‘batch at night’ applications on mainframes as well as PC’s and UNIX using Micro Focus tools. Installed five years ago, the systems developed represent core mission critical business applications. Regardless of the platform in use, however, there was a common thread of four major issues that seemed to affect everyone we spoke to, especially those companies that had already migrated their applications off the corporate mainframe.

System capacities with respect to the existing COBOL file system were limited to one or two Gigabytes per VSAM file depending upon access type. Due to continued growth this limitation was fast becoming everyone’s number one concern among our non-mainframe based clients. In fact, this was what first prompted Andrew Harris, Director of Application Development for a major financial institution to contact us two years ago. Having worked with Mr. Harris for some time we thought were well acquainted with his operating environment (see side bar). In fact we had a lot to learn about his business environment.

A recent merger had accelerated the need to complete a unified approach to the desktop, with Windows NT Server and Workstation as the approved operating systems. This represented a problem in terms of the CICS applications because the Micro Focus tools currently in use were OS/2 versions and no current upgrades to a Windows platform were available.

The net result according to Harris ? "A need to move without a place to go to."

Similar circumstances were facing Paul Morris at Delta Dental of Kentucky. His company also used Micro Focus tools in a similar fashion to handle their claims processing and payment system. Several CICS application regions were hosted on an IBM RS/6000. In addition to experiencing the same problems with his VSAM files as Harris, Morris had also been plagued by intermittent problems that could not be traced to either his AIX operating system or his COBOL system. Further, without upgrading his OS and or COBOL runtimes, his vendors were unable to offer any support.

While their primary needs were being driven by what amounted to nothing more than a COBOL file system that was too small, it soon became apparent that the time had arrived for a full scale evaluation of their operating environments. Both Harris and Morris were aware of the current solutions being marketed but, before spending time with vendors, they developed a list of requirements that went far beyond simply fixing their immediate problems.

Supporting their COBOL systems on NT was the central requirement for both companies and for the same reason - supporting multiple desktop and server operating systems had become unfeasible from both a financial and logistics standpoint. A unified OS approach would be the only acceptable answer. This was especially true for Morris whose AIX based system was the odd man out in an otherwise NT and Compaq based shop. Justifying the additional costs for the RS/6000 hardware and software as well as in-house expertise simply couldn’t be done.

The single OS approach also made integrating their core COBOL and CICS applications with existing applications an ideal that, for the first time, could be a reality. For Morris, who ran his COBOL systems on UNIX and all other applications on NT, the ability to replace batch FTP processes between his servers with direct program to program communications was included as a must have.

Due to technical limitations with their older COBOL systems, neither shop was able to compile their COBOL applications to the EXE or DLL level, so performance had always been an issue. Increasing business volume and the need to serve a national user base spread across multiple time zones were only shortening batch windows. Performance, greatly improved performance especially on the batch side of things, shaped up as a real concern for Morris. Month end run times had been steadily increasing and were approaching the 12-hour mark, Morris’s maximum window for off-line processing.

Scalability in terms of hardware and software growth for the foreseeable future was obviously a major concern for both Harris and Morris. Within the past 12 months both of their respective companies had nearly doubled their systems daily transaction volume through large scale business acquisitions. Memories of close calls and the knowledge that "last years magic could not be done again," as Harris put it, showed that the original issue - disk file size - wasn’t the only growth issue to be concerned with. The ability of the applications to handle increased work with a linear or near linear scalability was of critical importance. The solution, whatever it would be, would have to have the ability to handle twice the existing number of transactions overnight if need be.

Both Harris and Morris were running COBOL and CICS applications that served as the core application for their business. The applications, while different, shared the same architecture of a CICS system that performed live updates as well as created batch transaction files that were processed nightly. Both applications were heavily VSAM oriented and used periodic data extracts to feed data warehouse processes which were already NT based. Applications, although they were already PC based, were still using the EBCDIC character set. Again, even though the PC (or in Morris’ case an RS/6000) was the existing platform, all source code was still mainframe compliant COBOL with CICS API calls and BMS as a user interface.

Morris’s application, a purchased package, served as the company’s central claims processing system. Approximately 3,000 insurance claims for dental services are received daily, entered into the system via CICS and processed nightly. Nightly processes included determining payment, deductible and other amounts, producing checks and explanations of benefits as well as numerous other external processes. These external processes consist of extracting information from the COBOL applications and using these files to populate voice recognition and data warehouse systems. At the same time there were also EDI applications that were used to update the COBOL system with claim and membership eligibility information. (See the side-bar for operating system and technical system descriptions).

With their requirements firmly in hand, vendors were contacted and recommendations were solicited. Vendor proposals were evaluated; however, with the exception of migrating back to a mainframe environment every potential solution was deemed too invasive to the applications. Source code changes would be required to use COBOL CICS source code on Windows NT to use any of IBM’s NT based CICS products, and there appeared to be a number of potential migration issues. Morris says "I was uncertain with some of the products we evaluated during the past year. Our current systems were in EBCDIC and we wanted to be ASCII once we were on NT. The conversion strategies that were available for our VSAM files, and our JCL in some cases, were issues that I just did not feel comfortable with".

While no hard numbers are available, Harris’s early investigations also showed that re-hosting his OS/2 and Novell based COBOL systems with either 3rd party solutions or through in-house re-coding would involve application programming costs in the 350,000 to half a million-dollar range.

Morris’s search for a possible re-hosting solution also ended when his cost estimates started to exceed the same numbers Harris was finding.

It was around this time in the development of our clients’ projects that we became involved. We had been serving these clients in the capacity of an off-site application support so it seemed logical to call us for perhaps some non-programming support.

In discussing potential solutions with Mr. Harris, including ideas ranging up to a complete from scratch system development effort, one thing became very clear. The one thing everyone had been focused on fixing was the application software. As Mr. Harris put it at the time, "the one thing that isn’t broken is the application software. Why fix it?"

It was an interesting challenge, one that it seems many people were trying to answer: Find a better way to host, manage and grow with existing legacy applications. And, of course, deliver a working solution within budget.

Starting in 1998, about 18 months before Mr. Harris called, we had begun work on a Windows NT based server framework designed to act as a way to access file and database information across a dial-up or Ethernet network. This product, which we used as the core components of a private network EDI system, had been developed for the purpose of splitting existing applications written in COBOL to function in a multi-tiered environment.

Analysis of existing capabilities in our own software as well as the additional requirements of supporting mission critical applications was going to require some time to develop. However, the foundation was there and so the project was deemed feasible.

The existing framework consisted of a TCP/IP based server module that was capable of conversing with attached client software for the purpose of handling COBOL based file IO across a network. A preprocessor for the client COBOL programs was already in place. All the pieces were tied together already with a standard set of API’s we had developed. So the question of capability rested not with the ability to program a feature, but to recognize what features were needed. What Morris points out about other re-hosting solutions is that "they appeal to the programmer in me, but the systems manager in me needs more. Just running the COBOL program isn’t enough, I have to be able to manage the environment too."

Understanding what needed to be available in terms of usability features boiled down to an analysis of the requirements of both a general transaction processing system as well as how best to implement a traditional batch and CICS legacy application as a true client server system.

After Harris, Morris and their staffs, from programmers to data center console operators, were interviewed, a requirement and a wish list were developed and the underlying framework on both the client and server sides of the network were designed. Due to a familiarity with and the fact that both clients already had Micro Focus tools in house, Net Express was chosen as the IDE that would be used for program development and used by the client for their in house work.

The first beta of the package, dubbed the Eden Server system, was delivered in October of 2000 and put into production December 8, 2000. During that two-month test period the primary focus was on increased performance as well as the correction of several problems with screen handling routines and parallel runs of current production and test systems.

In addition to the migration from Micro Focus MTS to Eden Server, conversion of all data from EBCDIC to ASCII was required and was performed by using a specially developed utility. For Harris’s OS/2 to NT conversion, JCL was already being handled with OS/2 .CMD files. The installation of Eden at Delta Dental of Kentucky was from IBM’s AIX operating system. Unix shell scripts had been created several years earlier through the use of an MVS to shell script conversion utility. Using the original source for this utility, a shell script to NT .BAT file converter was created. According to Morris, the JCL conversion was "98% automatic, with the other two percent converted by hand." Continuing on the subject of the actual migration to the NT platform Morris adds "within 48 hours of installing Eden Server, we had migrated our entire test environment and had begun our user acceptance testing."

With respect to his experience with the Eden Server beta system, Harris comments, "we had very few problems getting our system up and running - and we had no problems that weren’t solved in a very timely manner. In hindsight, we absolutely made the best choice. Migration was painless, the users are happy and I now have a system that stays up all day every day and runs in less than half the time every night."

Morris, in comparisons between his Eden Server test and AIX based production systems, documented even more impressive returns. "One of our extract programs used to run almost 40 minutes on our RS/6000 as intermediate code. Compiled as an EXE on our new Compaq box, this same program is now running in under two minutes."

Through the use of various benchmarks and other testing the majority of the performance increases our clients have noticed have been attributed to the ability of the 32 bit Net Express COBOL system to take existing source and compile it to the EXE / DLL level - something with which older 16 bit products from Micro Focus had problems.

While neither Harris nor Morris had the ability with their previous MTS systems to produce any type of CICS based performance statistics, Harris reports his production server is handling just shy of 40,000 CICS transactions daily with an average terminal response time of just 0.7 seconds during a 12 hour shift. Stability has also played an important role in increasing productivity for Morris. "On the RS/6000 we had at least two or three occasions per week where the system hung. With Eden, I’ve never had to restart the system because one user's program crashed another. The fact that the COBOL programs run at the workstation does more than just take advantage of all those CPU’s out there - it prevents user programs from conflicting with each other and causing the whole system to crash."

The multi-threaded architecture of both the server and client components allowed Morris to nearly double his server throughput when he added a second CPU chip to his Compaq ML 530. "When we run the server in dual-cpu mode, we can handle over 1,000 client messages per second - that equates to 500 or more clients all getting sub-second response times."

Harris takes an interesting view on his experiences. "Because we only fixed what was broken - the underlying systems, not the application - we can still use the same source code so I’m still using 99.9% of my investments in my applications. I just get better returns now." Of the 472 COBOL programs used by Harris, only six required source code changes. These changes were related to the conversion from EBCDIC to ASCII collating sequences. Additionally, because Eden Server works with the Net Express IDE, including the animator, there was no down time for programmer retraining. Harris states his company is also reviewing other applications currently running on an IBM ES/9000 mainframe as potential candidates for Eden Server, adding that "the potential cost savings are huge."

And Morris is spending less time finding and correcting application problems, too. "The fact that every system event, including application program abends, is logged and can be reviewed means we spend a lot less time finding the source of problems. We expect our problem resolution times to go down significantly. That means we’ll have more time for projects that add to the bottom line both financially as well as technologically".

Perhaps the best result for both of clients has been the elimination of their ongoing capacity issues. By using Net Express’s COBOL file system, both of our clients now have the capacity for nearly unlimited expansion. According to Morris, by simply turning on large file support in Eden Server he "has the capacity to store every dental claim ever processed for every person in the world".

Harris agrees, stating he "won’t have to make fundamental changes to the system, for capacity or work load related reasons in the foreseeable future, with the possible exception of additional disk drives on the server".