The post below was contributed by Sundar Majumdar and Prabhakar Talla from Fannie Mae
With hundreds of analysts and business users relying on data to make decisions every day, Fannie Mae understands that, when it comes to defining the quality of the data, end user involvement is critical. However, getting business users to manage the quality of the data is not trivial. Those business users reside in different business units and are using multiple applications supported by a variety of data sources. Because of this, the data they gather may contain inconsistent, inaccurate, and irrelevant information.
While business users are domain experts, many have limited programming skills and are seeking easy-to-use data management tools that can quickly create trusted data they can use for further analysis or reporting. To empower more business users and enable faster time to market/lower cost, the data quality management (DQM) team within Fannie Mae’s Enterprise Data group decided to build a few handy tools to help their business counterparts. Developers on the data quality team created and delivered the following new processes:
Established a unified data quality metrics repository
Data fed into various applications comes from diverse systems and often results in different formats after going through data quality checking on IDQ Analyst tool part of Informatica tool suite; this poses a constant challenge for the business users working on developing enterprise-level DQ reporting. The DQM team came up with the idea of a business-friendly interface to simplify the process that enables the creation of common data format for data quality metrics and delivered the same. Applications executing DQ rules will leverage the standardized format and send the results coming out of IDQ Analyst or any other DQ rule software into a centralized Enterprise DQ Reporting solution, from which the enterprise-level DQ dashboard or reports are generated. By leveraging this new DQ results/metadata interface, an analyst can view DQ rules metadata and results across applications in a consistent format, and focus more on analyzing the data and creating meaningful, high-value reports.
Enabled and enhanced business self-service
Fannie Mae enabled and enhanced business self-service by:
- Training business and operations users to adopt the simple-to-use, yet powerful IDQ Analyst tool;
- Allowing self-scheduling of the IDQ rule execution; and
- Integrating the IDQ rules results with the Enterprise DQ Datamart.
Fannie Mae named the new process “DQ Analyst Self Service.”
First, the DQM team designed a formal onboarding process and template for the business team members to define, build, and test data quality rules in IDQ Analyst, according to consistent metadata naming standards and with additional rule metadata defined for the EDQ Datamart. Once a rule is built in IDQ Analyst, it then can be executed as part of a profile that is scheduled using “Profile Scheduler” – a handy tool designed by the DQM team specifically for business users to schedule the rule execution at their convenience (after-hours, for example) and with no formal project release.
The IDQ profiling results are collected by a “Metrics Collector” – another custom module created by the DQM team, and sent to the EDQ Datamart where the DQ results and metadata can be accessed by users across the entire organization.
The results of these new tools and capabilities have led to significant benefits to the business and enterprise. Thanks to the DQ Analyst Self-Service capability, no more long waiting times for a project to automate execution of data quality rules, or add new rules or change existing ones. The reduced cost and time-to-market/production encourages more business users to adopt Data Quality.
The EDQ Datamart contains all the DQ results/metadata, and can be accessed by the entire organization, thus enabling the transparency/reusability of existing data quality results and helping to improve operational efficiency.
The overall data quality release cycle has been reduced from 3-6 months down to 2-4 weeks by using the DQ Analyst Self-Service capability. Most importantly, with these benefits, the Enterprise Data division is expecting to see increased adoption of Data Quality within the entire enterprise, not just as part of projects.
One of the business analysts from the Single-Family Security Administration/Disclosures and Operations department, Lana Arain, said this:
“With DQ Analyst Self Service, the Disclosures team has seen time savings of approximately 30 minutes per month, and this is due to the automatic refresh of our LLD profiles. This is also a win since we no longer have to refresh the files manually; this new process ensures that the latest data is loaded without manual intervention and reduces the risk of reporting on the incorrect data if the profiles were not refreshed. The DQ Analyst Self Service feature has also standardized the naming conventions of the rules and so for any future enhancements to the profiles or creation of new profiles, we will be able to use the standardized naming conventions for rules.”
Vice President of Enterprise Data Management, Prabhakar Bhogaraju said this:
“Lowering adoption barriers is a key success driver for any enterprise data quality program. Instrumenting data quality rules in a scalable platform with effective business intelligence capabilities is an important yet non-trivial task. The DQ Analyst Self Service capability introduced by Fannie Mae’s enterprise DQM team is a key lever in lowering our cost of implementing data quality rules, as well as substantially decreasing our time-to-market. The IDQ-based Self Service tool we rolled out has been an important tool to expand DQ adoption rapidly.”
To succeed, Data Quality requires business and IT collaboration. As subject matter experts, the business teams are best at articulating the quality of the data requested. As technology and programming specialists, IT can help make all the data quality specifications operational, including offering some handy tools to their business users to automate and accelerate their job performance. Fannie Mae understands this and works hard to improve their processes every day.
The post At Fannie Mae, Data Quality Enhances Self-Service appeared first on The Informatica Blog - Perspectives for the Data Ready Enterprise.