Mainframe Connector demo series

Key Concepts

• QAM (Queued Access Method): A data format commonly used on mainframes.
• OC (Optimized Columnar): A columnar storage format optimized for analytics, used by Google Cloud.
• BigQuery: Google Cloud’s fully-managed, serverless data warehouse.
• BigQuery ML: BigQuery’s machine learning capabilities, allowing model creation and training directly within the data warehouse.
• Transcoding Configuration: A file used to define how Mainframe Connector interprets specific fields within QAM data.
• Mainframe Connector: A Google Cloud tool facilitating data transfer between mainframes and Google Cloud.
• Vertex AI: Google Cloud’s unified machine learning platform.

Mainframe Data Modernization with Google Cloud Connector

This demonstration showcases the Google Cloud Mainframe Connector’s capabilities for transferring QAM data from a mainframe environment to Google Cloud BigQuery, enabling advanced analytics and AI applications. The core objective is to unlock the value of mainframe data by integrating it with modern cloud services.

QAM to OC Conversion & Data Transfer

The process begins with defining a batch job utilizing the Mainframe Connector’s “QAM decode” command. This command converts QAM files into modern formats – OC, JSON, or CSV – offering flexibility in input and output locations. The input QAM file can reside on the mainframe itself, a local Linux environment, or a cloud storage bucket. Similarly, the converted file can be directed to various destinations.

In this demo, a QSAM file on the mainframe is converted to an OC file within Google Cloud Storage. The Mainframe Connector is designed to handle the complexities of mainframe data, supporting both fixed and variable record formats, and interpreting diverse data categories including alphanumeric, numeric, DBCS, national, and floating-point fields. It also manages clauses like “occurs” and “redefines.”

A crucial element is the “transcoding configuration file.” This file allows granular control over data interpretation. For example, the demo utilizes this file to designate fields ending with “-ashts” as timestamp data types. Performance is addressed through flag options like “parallelism” and “chunk size,” enabling efficient processing of large datasets. Transfer logs provide real-time visibility into the data transfer status.

The Google Cloud Storage destination utilizes a two-folder structure: a “main” folder for the transcoded data and a “spillover” folder for capturing any transcoding errors, aiding in debugging. In this instance, no errors were encountered.

BigQuery Loading & Schema Automation

Once the data is in OC format within Google Cloud Storage, the Mainframe Connector’s “BQ load” command efficiently transfers the data to BigQuery. A key benefit highlighted is the automation of schema mapping. The connector automatically generates an accurate BigQuery schema by reading metadata embedded within the OC file, derived from the original Cobalt copybook and the transcoding configuration. This automation minimizes manual effort and reduces the risk of errors.

Following the load, the data is immediately available for querying. A sample SQL query demonstrates the accessibility of mainframe data as a dynamic, queryable resource within BigQuery.

AI Application: Customer Churn Prediction

The demonstration then illustrates the power of applying Google Cloud AI to the migrated mainframe data. A customer churn prediction model is built using BigQuery ML. The process involves pre-processing historical customer data, identifying churned customers (defined as those inactive for a month), and splitting the data into training and test sets.

BigQuery ML is then used to create a prediction model, enabling the accurate identification of customers at risk of churn. This allows businesses to implement targeted retention strategies. The presenter emphasizes that BigQuery ML is just one entry point, with seamless connectivity to Vertex AI and other Google Cloud AI services for deeper insights.

Data Roundtrip: Cloud to Mainframe

The Mainframe Connector also supports data transfer from Google Cloud back to the mainframe. Using the “QSAM encode” command, data – including subsets derived from BigQuery queries – can be moved to the mainframe, again with flexible output location options.

Performance & Scalability

The Mainframe Connector is “engineered for scale,” offering direct control over performance tuning through options like parallelism and chunk size. This ensures efficient processing of large files. Real-time transfer logs provide visibility into the data transfer status.

Conclusion

The demonstration successfully showcases a complete, modern data pipeline for seamlessly moving QAM data to the cloud, preparing it for analysis, and applying transformative AI capabilities. The Google Cloud Mainframe Connector aims to liberate mainframe data, modernize infrastructure, reduce costs, increase agility, and build a future-ready enterprise. Further information is available at cloud.google.com/solutions/ainframe-modernization.

Notable Quote: “Liberate your mainframe data with Google Cloud mainframe connector.” – Presenter, concluding the demonstration.