The rise of business enabled analytics has come about in large part due to the failure of traditional IT to quickly respond to the ever-changing needs of the business. Data Warehouses and Operational Data Marts typically contain specific sets of data targeted to answer very specific questions for the business. They also generally have longer development cycles to prepare the data so that the specific questions can be repeatedly answered when desired by the business.
But what about the questions that the business doesn’t know they need to ask? Where do the answers to these questions come from? For most companies they go unanswered, or the business has learned to become data gymnast to contort data as part of a cumbersome weekly or monthly song and dance routine.
However, in the more recent years, there has been a quest to solve this problem for the business. The rise of Data Lakes, which are named as such due to their containment of all available data sources, both internal and external, have now given the business access to raw, untransformed data quickly. The Data Lake would contain all the sources the business is currently contorting, as well as additional sources they have never had access to before. And since the data is initially loaded straight from the source in a raw format, the business can explore and manipulate that data in their desired way until they find those hidden questions and answers.
Therefore agility is baked into Data Lakes. The business no longer needs to wait for lengthily development and QA cycles from IT before they can start answering their day to day business questions. But does that mean that development and QA activities go away? No, certainly not. The business will need to become more technical as they take on these traditional IT roles in the name of speed and agility. This is where the role of Data Engineers and Data Scientists come into play.
Data Engineers are the role type that would establish the data lake and start the initial loading of raw data. This role might sound familiar, and it should because this is not unlike the existing role of EDW Developer, combined with data modeling skills, and some Big Data Ecosystem configuration/administration.
Data Scientists are key players in the world of Big Data and Data Lakes. They are the people who know and understand the data best. They are also the people with the technical skills to produce the kind of “business-friendly” insights so desperately needed in today’s business, yet has continued to remain elusive for so long.
At a Datameer event in Atlanta, I used my time to discuss the inherent agility a Data Lake brings to a company, and how a Scrum workflow is used to organize the business needs and data ingestion workflow.
Below are some highlights of the ideas I shared:
Data Lakes exist to serve the data needs of the entire company quicker than traditional reporting structures
Data Lakes are not intended to replace traditional reporting sources such as EDWs and ODSs
Data Lakes can become a valuable single source for traditional reporting platforms that are intended to answer known questions repeatedly
Each department throughout the business can “subscribe” to any part or piece of the Data Lake to explore and create relevant insights
Insights discovered (explore, find patterns, formulate questions, deliver answers) might be for a single-use scenario, or they might be useful to save back into the data lake as a new set of data available for anyone to reuse
For initial Data Lake build out and incremental changes, use Scrum as a framework to facilitate data ingestion, delivery of raw or minimally transformed data and incorporate data lake user feedback loops (don’t flood the lake… iterate)
Use the Product Owner role and your Data Lake Product Backlog to ensure good communication between the IT data team and the data consumers as this is key to controlling the chaos of Big Data projects.
Just because the Data Lake exists doesn’t mean you’ve succeeded, you still have to create catalogs of the available data sets for easy searching, govern the creation of new data sets to ensure compliance, and secure the data to make sure only those with proper permissions see the right data. Building and maintaining these things will keep your Data Lake fresh and less like a swamp.
For more insight on how to leverage data lakes in a practical business sense, contact a data and analytics expert at (813) 265-3239 or firstname.lastname@example.org.
Written by CCG, an organization in Tampa, Florida, that helps companies become more insights-driven, solve complex challenges and accelerate growth through industry-specific data and analytics solutions.
CCG understood our project needs very well, they are very responsive and we could not ask for anything more. The solution they provided fit perfectly with our expectations and business goals.GOP Data TrustChief Data Officer
I cannot overstate the delight we experienced from the outcome of our project. I would not only recommend CCG to any company, but question why they would engage with anyone but CCG.PgiDirector of Customer Success
Working with CCG is like working with extended team members. Consultants become an integral part of the work bringing expertise for cutting edge design and development.Hillsborough County Public SchoolsChief Information and Technology Officer
CCG's team is positive and eager. They are a great big bunch of wonderful people trying to make a difference.Hillsborough County Public SchoolsDepartment Manager
I knew CCG's technical expertise and dedication to quality results would be invaluable to our project success based on our past partnerships. We could not have implemented in the short timeframe like we did without their assistance. CCG is #1 on my speed dial for successful project implementation.InCommDirector, Financial Information Systems
It was evident from the onset of negotiations through the implementation that CCG took their role in the partnership to heart and we believe it has been instrumental in our success.Interval InternationalDirector of Marketing
CCG works very hard to understand and align with our needs. It truly feels as though we are on the same team!Fortune 500 HomebuilderBI Manager
CCG came to our company in a time of much change. Their team partnered with ours, continually delivering with professionalism and efficiency. We would not be where we are today without the expertise CCG brought to the project.PSCU Financial ServicesSenior Program Manager
CCG has a good industry knowledge, we are very happy that we chose to work with CCG. They have been a great help strategically and are helping us make important decisions.Minneapolis Public SchoolsHuman Capital Coordinator
Other Vendors use the word Partnership, but CCG actually means what they say. I can’t thank them enough for their professionalism and willingness to work with us as a true Partner, not just another vendor.PODSCIO
Our CCG Consultants are total rock stars: very thorough with a solid knowledge of the financial services industry. As a bonus, they are very easy to get along with – a great fit for our team.Raymond James Financial ServicesSenior Manager of Enterprise Data
CCG's team are all amazing. Thank you, CCG, for all that you do to make us great and keep our credit unions moving forward!PSCU Financial ServicesVP Enterprise Analytics & BI
Other Vendors use the word Partnership, but CCG actually means what they say. I can’t thank them enough for their professionalism and willingness to work with us as a true Partner, not just another vendor.PODSChief Information Officer
CCG's Team is very professional and responsive. They are making our job very easy.Rollins, Inc.Senior BI Analyst
CCG did an excellent job! Their team was very flexible. They gave us everything we asked for and then some.Rooms To GoSenior BI Architect
I'm amazed at the talent at CCG, not just the skillset - they're really good people. We've already referred them once and will do so again!Ruth's Chris Hospitality GroupCIO
CCG did a great job! We're extremely impressed with what was built in a short time. CCG has delivered ahead of time and with best practices, it's been a pleasure to work with them.VologyVP of Analytics
2502 N. Rocky Point Drive, #650, Tampa, FL 33607
Phone: 813.968.3238 | Fax: 813.200.1357
8000 Avalon Blvd. Suite #100, Alpharetta, GA
Phone: 404.328.7298 | Fax: 813.200.1357