Module 0: Introduction to Big Data and Business Intelligence

Lecture Summary

    Data is being produced every second, every minute and every hour at an increasing rate. It reminds me of the universe and how it is expanding at an accelerating rate (NASA). At the same time, the variety of data that is available to us for processing is expanding as well. The volume, variety and velocity of data is getting to the point where it is almost hard to quantify (which also reminds me of the universe). The trail of data that is being left by all of our actions is creating a "digital footprint" or "digital shadow" (IBM) that can be used for a multitude of purposes positive or negative (e.g. Brand Awareness and Reputation Management, Customer Acquisition and Engagement, Security Breaches and Data Leaks) (EPAM). We have more data than we know what to do with and we need a way to synthesize this data for better human and business outcomes. 

   

 Business Intelligence is a set of tools, technologies and techniques to provide actionable insights from the data that we have collected. Traditional business intelligence was limited to traditional data sources (e.g. databases) and methods (e.g. Extract, Transform, Load (ETL) processes) but, Big Data has forced business intelligence processes and procedures to adapt to collect and utilize data from non-traditional sources. The Business Intelligence Lifecycle describes the process that allows us to apply Big Data/BI to various applications (e.g. legal, environmental). But, to apply business intelligence in a proactive way, we need to create a skilled labor workforce that can keep up with the volume, velocity and variety of data that is being created on a daily basis.

Analysis of Materials / Reading

    

    When I read about Big Data/ BI and our ability to collect more information than ever before I think about the following 7 questions -

1. What is the human or business outcome that I am trying to achieve? 

For this step, I think we should turn to the basic steps in the scientific method to "define a question to investigate" and "make predictions or state a hypothesis" (AMNH). To me, this is a critical step because, often analysts trying to use data to answer questions that are not properly defined or irrelevant to the outcome they are trying to achieve. You cannot use data to try to answer a question that is not properly defined this only results in a wasted time and money. 

2.  What is the level of the complexity of the problem?

I do not believe that data-driven decisions are needed to make every decision. I believe that certain decisions can be made based on intuition or past experiences depending on the nature of the problem. For example, I do not need data to inform me that I should not touch a hot iron because, based on past experience I have burn my hand. Of course, there are a significant amount of decisions that need to be data driven but I think we need to understand if the additional analysis is warranted. 

3. What is the type of data that I would need to address the question?

For Big Data, I think this step really needs to be focused and closely aligned with the question we are trying to solve. If we do not bring in the right types of data or too much unnecessary data, it will be difficult to draw conclusions against our hypothesis. The best I can describe this is just because we have "Big Data" sources does not mean we need a lot of data, we need the "right data" and the "right volume" to address the question we are trying to answer which for complex problems is easier said than done. 

4. What is the simplest solution that can be implemented to address this issue?

I think because, we have a lot of Big Data to work with. We often think that the business intelligence solutions that we develop have to be complicated to account for the web of data that is available to us. I believe the opposite is what is actually required. Speaking from an industry perspective, most companies are running the same back-office processes whether it be Source-to-Pay, Record-to-Report, Order-to-Cash etc. There are only a few key processes within an organization that make them unique and are at the core of each business. So, the simpler the solution the better. 

5.  What are the tools and techniques that I need to use to solve this problem?

For the tools and techniques that are used to develop business intelligence these need to be (1) rapidly deployed (2) rapidly iterated (3) rapidly thrown away (if required). We cannot make large investments in business intelligence solutions before iterating to see what business intelligence is required in the short, medium and long term. The tools and techniques that we utilize should address the 3 key points mentioned above. 

6. What are the insights that need to be developed?

Speaking from a business context, the development of useful and meaningful KPIs that are agreed to and measured is much more difficult in practice than in concept. In general, people do not like to be measured. Tying a measure to a business process that a person is responsible for and tied to their employment is often met with resistance and requires significant change management. 

7. What is the best process to take action on these insights?

This to me is the most difficult part of any business intelligence project. I would argue that most leaders of organizations broadly know where their problems are. Many business intelligence solutions provide wonderful insights to say if you do "X" we can gain in "Y". But, time, budget, resources, organization momentum, personalities, biases all play a part in whether an action will be taken. For this, I have been wondering how the advances in AI will allow business intelligence to take action on insights without human intervention, if anyone has any articles around this topic, I would be interested to read more about it (AA).

References:

- NASA: What is Dark Energy? Inside our accelerating, expanding Universe - NASA Science

- IBM: What Is a Digital Footprint? | IBM

- EPA: What Is a Digital Footprint for Your Business? | EPAM SolutionsHub

- AMNH: The Scientific Method: 5 Steps for Investigating Our World | AMNH

- AA: Actionable Analytics: What It Is, Why You Need It, and How to Use It - InnerTrends