One of the main buzzword that we constantly hear is about insights or unlocking insights from data. This has been one of the main selling aspect when it comes to selling technology to the business. The sophistication of tools is a welcome feature to unearth the insights, at the same time what are the critical components in order to get meaningful insights? One of the fundamental requirement in order to get meaningful insights, is to what have a solid data pipeline end to end. In order to have this there needs to be the following in place:
1. Data Governance/Lineage.
2. Metadata Quality/Entities.
3. Valid Domain Value Chain.
4. Customer Data (Profile/Accounts/Interactions via different Channels)
5. Data Quality including Workflow/Validation/Data Test Beds/Deployment.
6. Track Data Assets Related to a domain.
7. Business Data Owner - A Person or a group of people who can help identifying Business purpose/meaning of all the data points in the Domain.
8. Ability to Handle Technical Debt - How to Systematically handle technical debt. A very common scenario in organizations grown by Merger and Acquisitions.
9. Scale, Share and Speed - Does the Architecture, Infrastructure available can handle the frequency/speed of data requests by business.
The elements mentioned above are very important, a good interplay of the above elements are needed in order to generate valid insights. For insights there are 2 main components
1. Insight Rules - Rules which are executed when certain events happen and certain business conditions are met.
2. Insight Triggers - Capture data points when certain events happen, for example there was a credit card transaction made at lowes or Home Depot, or someone paid a SAT entrance exam fee or a there was mobile deposit made. As part of this process there is also selection criteria around how are the transactions picked, also includes whether the insights are going to be triggered daily, weekly or a monthly basis.
The combination of the above 2 components can help generate insights, now assuming that the 8 elements mentioned above are satisfied or they are in place. It would be also advisable to categorize the Insights based on the Domain so that it would be easier to track and maintain the insights. There is constant mining of data that is being done, in order to generate accurate insights.
AI and ML are used very heavily when generating insights. The effectiveness of AI, ML becomes more apparent if the underlying data infrastructure is really solid.
The purpose of this blog post is to explain highlight the importance of solid data foundations needed to generate valuable insights for business and customers.