Skip to content
Media & Entertainment · Cloud + Big Data
Americas Cloud migration & automation

Moving the Analytics Environment to Hadoop on AWS for a Global Sports-Entertainment Company

Web, social and internal data for a global sports-entertainment company, moved onto Hadoop on AWS — landed in S3, processed with EMR, served from Redshift — at a scale the previous environment could not hold.

Client
Global sports-entertainment company
Region
Americas
Engagement
Cloud migration & automation
BIG DATA PIPELINE ON AWS · TENS OF TB PER DAY SOURCESLANDINGPROCESSINGANALYSIS Web Social media Internal systems Amazon S3 Elastic MapReduce Python automation Amazon Redshift 360° views NO PROPRIETARY TOOLS · AUTOMATED ERROR RECOVERY

Tens of TB

Of data per day

100%

Analytics environment on cloud

0

Proprietary ETL tools needed

360°

Customer touch-point analysis

Requirement

The challenge

  • Complex and heterogeneous source systems.
  • Huge volumes of data (tens of terabytes per day).
  • A wide variety of data from web, social media and internal systems.
  • Long ETL and reporting cycles.
  • No predictive or advanced-analytics capabilities.
  • High maintenance due to the legacy code base and analytics environment.
Solution

What we did

  • Collected a large variety of data from different sources into AWS S3.
  • Processed, cleansed and aggregated the data for analytics using Elastic MapReduce on AWS.
  • Loaded the data into Amazon Redshift for ad-hoc analysis.
  • Automated the complete process in Python and removed the need for proprietary tools.
Tools Amazon S3Amazon EMRAmazon RedshiftPython

Outcomes

What changed for the client

  • The complete analytics environment is migrated to the cloud.
  • Cost savings by eliminating the need for proprietary tools.
  • Real-time analytics, avoiding time-consuming ETL and report-scheduling activities.
  • 360-degree analysis of customer touch-points and customer profiles.
  • Low maintenance with automated error recovery.

Related insights

All articles
DATA ENGINEERING vs DATA SCIENCE PLUMBERS + ARTISTS Data Engineering THE PLUMBERS · DESIGN THE DATA FLOW Pipelines & architectureC++ · Python · Scala · JavaAPIs · middlewareGovernance · quality checks Data Science THE ARTISTS · TELL THE DATA STORY Modelling & storytellingMaths · statistics · CSAI · machine learningBusiness insights ALLIANCE NOT "DO IT ALL" 85% OF BIG DATA PROJECTS FAIL (GARTNER 2017) · 70% OF DIGITAL TRANSFORMATIONS FAIL (McKINSEY) THE LINCHPIN: ENGINEERS + SCIENTISTS TOGETHER

Data Engineering & Data Science

Data Engineering vs Data Science: Which One Saves the Business?
MANUFACTURING · 7 OPERATIONS DATA ENGINEERING + DATA SCIENCE DATA WAREHOUSE · SENSORS 1 Predictive maintenance 2 Real-time monitoring 3 Demand & inventory 4 Price optimisation 5 Supply chain 6 Product development 7 Worker safety

Manufacturing Analytics

7 Manufacturing Operations that Data Engineering & Data Science will Revamp
DATA ENGINEERING & DATA SCIENCE FIVE TRENDS · ADOPTION ↑ 01 NLP + data science VOICE · TEXT · MARKET INTEL 02 AI-powered IoT PREDICT FAILURES · UPTIME 03 ML automation CLEANING · PREDICTIVE ANALYTICS 04 Privacy & security ANOMALY DETECTION 05 Cloud data warehouses FLEXIBLE · CONNECTED · CHEAPER ADOPTION 2016 → 2020 → CLOUD-ERA DATA ENGINEERING ON-PREMISES WAREHOUSES CLOUD DW

Data Engineering & Data Science

Five Hottest Trends that are Reshaping Data Engineering & Data Science