Data Processing Platform with AWS Data Lake
Description
Advanced Data Processing Platform with AWS Data Lake for Multinational Media & Entertainment Conglomerate .
Business Problems
• Need for a data warehousing environment for management of big data.
• Absence of a platform for active data processing.
• Requirement of ETL components for data extraction.
• Data visualization with Tableau as the client has the required licenses.
• Mastering data management from different sources with external and internal integrations, multiple data sources with unstructured data.
• To process raw data for comprehensive visualization with AWS
Solution Details
• Complete data warehousing environment with cutting-edge technologies.
• Development with Informatica, Teradata, and Unix. To create data model and ETL architecture for data processing.
• Managed data loads from database to Hadoop using Sqoop.
• Analysis, design, coding, and development of various Java/J2EE/Hadoop-based ETL components to extract and load data. Built Hadoop-based data processing software to perform analytics on data sets.
Results
• Effective storage environment and creation of big- data based analytics.
• Complete data warehousing of huge datasets.
• Channeled the data pipeline and consolidated reports from different sources.
• Built ETL components to extract and migrate data from legacy systems to Amazon S3 data lake.
• Achieved a one shop visualization interface with Tableau to get a better picture about the analysis.
