Data Lake Formation

Extensive Data Lake Experience

The amount of data that is generated and utilized by businesses has been growing at a tremendous scale. The growth in the amount of data has catalyzed the research and development of new purposes and use cases, further driving up the sheer amount of data that is generated. In fact, data grows by 10x every 5 years and hence data platforms need to scale 1000x to be sufficient for 15 years of storage and processing requirements.

Current solutions to on premise data storage and analytics involve Hadoop Clusters, Data Warehouse Appliances and SQL Databases. These are siloed however and have minimal communication amongst each other in addition to having scalability limitations. Data Lakes offered on cloud platforms are a superior solution to meet the demands of data today and in the future, as it grows at a rapid pace.

Let us build your Data Lake

Building a data lake isn’t easy. It entails numerous manual steps, which make the process complex and time-consuming. You have to load data from diverse sources and monitor the data flows. You have to set up partitions, turn on encryption, and manage keys. Redundant data has to be deduplicated. And there’s still much more to do.

Without the right technology, architecture, data quality, and data governance, a data lake can also easily become a data swamp — an isolated pool of difficult-to-use, hard-to-understand, often inaccessible data.

In our experience, following the four-step method outlined below and utilizing cloud data lake services can simplify and streamline the process.

Identify Data Sources Data Source Identification is an important step that will be needed for every new type of information that needs to be collected. It is mostly an analysis task and usually involves a fair amount of inter-departmental communication.

Data Ingestion Determine data flow pipelines and develop migrations patterns from source to cloud.

For batch data, set up processes to schedule periodic file transfers or batch data extracts.
For event data, set up processes to ingest the events - this might be an event endpoint.
Set up processes to bring in reference data (users, departments, calendar events, work project names).
Consider other groups/departments that may be impacted by any new processes established and communicate the changes proactively.

Data Transformation Here is where a lot of the magic happens. Data is transformed for use and performance for analytical applications.

Consider the types of queries that will be needed for the data.
Set up table layouts in the data lake.
You may need to aggregate metrics at logical boundaries .
For performance reasons, store the same data in different formats based on how it will be accessed.
Consider taking advantage of serverless facilities that let you write SQL queries directly against files in cloud storage.
Build out a library of queries that will be useful for dashboards and reports.
Documentation in the form of a data dictionary to help end-users understand the data.

Data Visualization Once the data is staged, it can be accessed in various ways by multiple front end business intelligence (BI) tools.

Consider setting up workbooks with prepopulated queries or table definitions.
Establish a BI environment for ad hoc queries.
Connect with data science professionals to prototype and validate algorithms.
Build out a library of queries that will be useful for dashboards and reports.
Work with key stakeholders to put together some preliminary dashboards and ensure views of the data are understandable and useful.
Maintain regular communication with the user community in order to determine new requirements.