Metadata Catchers

Metadata Catchers has been developed to solve automatically the most frequent criticalities that usually stop a Data Lineage generation.

–       Presence of different Tools on the same Application Chain not exchanging natively Metadata such as (but not only) 2 different ETL Tools (e.g. primary Data Warehouse Loading with Data Stage and Secondary Data Warehouse loading with Informatica Powercenter);

–       SQL coded inside a Tool’s transformation objects (Job or Stage);

–       Scripts present along the Application Chain;

–       Calls to external Transformation rules or procedures like Teradata BTEQ or Views or Stored Procedures, etc.

–       Parametric Job and dynamic queries call;

iGovernance Suite’s  Metadata Catchers solve the above criticalities for many different technologies and tools.

* BTEQ, Views & Stored Procedures

AboutDataGovernance has a fast “time-to-market” approach in implementing Metadata Catchers needed by our Customers. On request, we can also develop customized Metadata Catcher and deliver them in 6-8 weeks. This is possible thanks to our own parser quick personalization to Customers’ Specific Needs.

In reality, however, the use of Technical Metadata alone does not enable the solution of certain additional critical issues in which important transformation rules reside, such as:

–       Transformations using “parameters” like multiple scale values (e.g.  0 to 9 or similar);

–       Dynamic Queries (Queries whose values are defined by the application at Run Time only);

The above information is not present in a simple technical declaration because these values can only be recovered in the moment they are expressed and while IT processes are being executed.

AboutDataGovernance has implemented for each different Metadata Catcher a dedicated Agent responsible for rescuing those unencrypted expressed values from system logs – at 0 impact on running processes – and integrating them with the reconciled Technical Metadata, completing the Data Lineage automatically.

The resulting Data Lineage can then be loaded into the Customer’s Metadata Repository or tools like IBM BPM, Informatica Axon, Collibra, SAS, etc. at the depth level requested by business. The deepest and most complete one is still available in iGovernance Suite, ready to be used by IT to solve daily criticalities or tasks.

Metadata Catchers: how they work

As Customers in the Financial Sector are Auditable Subjects, they need to resume their own IT transformation processes, generating certain results During the Time.

Metadata Repositories available on the market do not usually allow that kind of functionality because at declaration level they are able to identify only the “last picture” of a certain Data Lineage. As a consequence, they are enabling only what is called “object versioning”.

On the contrary, users need a real “Time Machine” function, able to recover an entire IT physical process path, run months or years before.

This need is satisfied by iGovernance Suite, which enables the “IT Process Historicization” function on the DB application.  When any IT process is executed and identified by agents, as it ends a unique number is assigned to the entire IT process physical path, and stored in the iGovernance Suite Application DBstory.

If the same process is executed again, a new unique number is assigned and the process is stored again, because, having been executed in a different moment, it could have treated different contents and generated a different value even though the transformation logic remained the same.

In virtue of this functionality, Customers will have the physical IT process representation available at all times, generating a certain result in a certain moment, thereby being able to answer any Audit or request with the whole IT physical processes history.

In addition, the approach to using both Technical and Operational Metadata Integration and IT Processes historicization allows the customer to obtain other, no less important, advantages:

–    to Discover and highlight unknown Application Chains;

–    to Detect the execution of a new, not previously censused, path and if necessary to raise a warning;

–    to Send the identified path to the Customer’s Metadata Repository, if necessary allowing proper 
      Data Lineage construction;

–    to Highlight all transformation objects and data present in the Customer’s system and not used by anyone,
      allowing Application Offloading implementation and money saving;

–    to Help customers to always have the clearest and simplest Metadata View, instead of a constellation of object
      representations, by Allowing the usage of Metadata “Time” in Data Lineage representation;

–    to Obtain the relationship between Data and Applications, transforming them, and to identify temporary tables used
      by the system, useful in GDPR (more exhaustive Data Privacy-Data Flow);

–    to Validate the Data Lineage reversed through Metadata rescued at run-time (or through JCL analysis in case of Cobol)

–    to Validate (if requested) the ER Logical and Physical Data Model, integrating information from Erwin and from Data
      Discovery systems with those found during the Reverse Data Lineage Building;

–    to Historicize any executed Application Chain to satisfy Audit purposes, as iGovernance Suite architecture has implemented two additional applications helping areas different from  CDOs.

The Execution (or Operation) team and the Software Development Team can now take additional advantage from the precious value represented by Technical and Operational Metadata stored in iGovernance Suite Application DB story.

The above give the right answers to the 5 important criteria needed to consider a Data Lineage really “Useful” for Customers: