The Evolution of the Data Warehouse

In the early days of modern computing, reporting was primarily an IT function. Information was trapped within centralized operational systems which were largely inaccessible to anyone outside of a small, trusted group of IT resources. Consequently, all reporting needs were satisfied by “qualified” IT professionals who created batch reports that were scheduled to run nightly so as to not interfere with daily operational processing tasks.

While these nightly batch reports provided some visibility into operational data, they were limited in scope and primarily output in a tabular format. Further compounding the issue, information silos crept into existence as organizations adopted heterogeneous IT environments consisting of multiple operational systems to support key functions including sales, marketing, logistics, financials and others. Businesses lacked the ability to gain cross functional insights that would allow for data driven, strategic decision making.

This limited access to information gave birth to the data warehouse. With the invention of the data warehouse came the promise of bridging the information silos and creating a unified environment where all information could be accessed for business intelligence. While the first generation data warehouses enabled new insights by bringing disparate data sources together into a single environment, these were typically IT led initiatives that lacked sufficient business user involvement to ensure that the DW solutions delivered the right business value.

Some of the key challenges with the first generation data warehouses include:

• Lacked data governance and conformity in terminology and metric definitions

• Modeled by IT without collaboration of business users resulting in data models that did not accurately reflect the business

• Delivered using a classic waterfall development approach leading to lengthy development cycles and a long time to value

As a result, these IT led data warehouses failed to deliver on the promise of actionable insights and made the term “data warehouse” a bad word in many companies.

Enter DW 2.0 . . .

Experience taught everyone that Business Intelligence has to start with the…BUSINESS!  The failings of the first generation data warehouses can largely be attributed to the lack of business user involvement. A key lesson learned was that the business must lead data warehouse initiatives and it must start at the top with C-level sponsorship. Business users must be active participants in driving requirements, contributing to data modeling and validating the solution early through prototyping and other means.  Another key lesson was the need for a strong Data Governance foundation to guide the data warehouse efforts. The Data Governance function ensures that the business is actively engaged in defining the data, ensuring consistent terminology, standardizing metrics and driving data quality improvements into the source systems - all problems that plagued the first generation data warehouses.

The second generation data warehouses, underpinned by strong data governance and business user engagement, have produced significantly improved outcomes. In particular, DW 2.0 has empowered many businesses to adopt enterprise business intelligence as a core competency, improve the quality of data and transition decision making from uninformed to insight driven. Despite the improvements, there are still challenges to be addressed including:

• The DW environment is still primarily controlled by IT with restrictions on access

• It takes too long to deliver new data sources or augment existing data sources

• IT resistance to bringing in new data from external sources

• IT is overwhelmed and unable to keep pace with the needs of the business

• New data sources take too long to integrate or are never integrated

Since the start of DW 2.0, there has been an explosion in the growth of new data. The rise of new data sources from cloud, social, sensors, third parties and others have challenged the traditional data warehouse architecture. Business users need to be able to access these new data sources and do so quickly.No longer does the single DW platform satisfy the information appetite of data savvy business stakeholders. As a result, the industry has responded by creating new technologies that empower the business to leverage both the governed assets in the corporate DW as well as take advantage of the new data sources that exist outside of the data warehouse.

These new technologies enable what Gartner terms “bi-modal”. Bi-modal is the idea of empowering line of business analysts with tools and data to innovate while also continuing to expand and evolve the traditional governed data in the data warehouse.We no longer speak of a single, monolithic Data Warehouse for insights.The Data Warehouse is undergoing evolution in what TDWI calls the “Modern Date Warehouse” and Gartner calls the “Logical Data Warehouse”.Traditional, governed data warehouses and BI are combined with new breed technologies such as:

Big Data (Hadoop, Spark)

• Virtualization (governed, real time access to disparate data sources through a single interface)

• Data Discovery and data blending to perform analysis of DW data and ungoverned data in a single environment

• Cloud platforms

• Heterogeneous access to data… using the right visualization platform for the right purpose and audience

As the famous quote says, “The only thing that is constant, is change”. We live in a rapidly evolving world with BI, Analytics and Data Management technologies responding quickly to the ever changing needs of business.Businesses that are quick to embrace these changes are able to see farther, go faster and operate more profitably than their competitors. At CCG, we have the opportunity to help our clients navigate the myriad challenges in evolving their BI and DW architectures and realizing the benefits of the bi-modal movement.

For more business intelligence insights, contact a data and analytics professional contact info@ccganalytics.com or call 813.265.3239.

Written by CCG, an organization in Tampa, Florida, that helps companies become more insights-driven, solve complex challenges and accelerate growth through industry-specific data and analytics solutions.