Why "Big Data" is a Big Deal ?

Why "Big Data" is a Big Deal ?




By : Shwet Mishra
Review By : Utsav Srivastav


Consider a day in your life,  What is the best road to take? , Would there be any bad weather? , What is the best way to invest money? , Which product should I buy? To answer these questions, firstly, we need to understand Big Data. Lets dig deeper into this, What is Big Data? Now a days Data flows in our daily life from phones and credit cards ; televisions and computers; From sensor-equipped buildings, trains, buses, planes, bridges and factories. This shows that we are living in a digital age where we are collecting and converting huge data into digital form 
Giving some perspectives:
  • 90% of the data in the world today was created within the past two years.
  • Web does millions of activities per second and so much server logs are created.
  • Social Networking Sites e.g Facebook, has over 1 billion active users per month and collects around 80 billion photos from them.
  • Walmart handles more than 1 million customer transactions every hour which are imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data the equivalent of 167 times the information contained in all the books in the US Library of Congress.
Phew ! So, clearly there is a lot of data populating overtime.  So where does the problem lie?
  •  Storing Data : It takes 1000 computers to store 1 peta byte.(assuming on an average a single computer stores 1 tera byte of data) 
  •  Moving Data: It takes 2 hours to copy 1 TB and 83 days to copy 1 PB.(assuming 10 GB network) 
  •  Searching: Assuming each record is 1 KB, and one machine can process 1000 records per second, it needs 277 CPU days to process 1 TB and 785 CPU years to process 1 PB. 
How Data Science helps us resolve them?
  • Storage of data: Historically, we used databases where scalability was a challenge, Now, NoSQL databases such as CASSANDRA, HADOOP(for structured data), and Distributed file system e.g HDFS/Hadoop Distributed File System(for unstructured data) are used.
  • Movement of data: With the help of distributed computing, there's no need to transfer huge amount of data, instead we can divide it into small chunks that are distributed over networks of computers in a geographical area.
  • Making Sense Of Data : From performing basic analytics (i.e. min, max, average, histograms, distribution) to complex data visualization, it is now possible to answer some of the most fascinating questions. 
Usecases Of Big Data 
  • Targeted Marketing Making personalized deals and recommendation based on the types of pages you have liked on Facebook. Now , who would have thought that? 
  • Travel By analysing traffic data it is now possible to tackle traffic and avoid congestion on road
  •  FIFA World Cup 2014 Systems Applications Products (SAP) a European firm partnered with the German team in FIFA World Cup 2014 
    • SAP used Big Data, to help the German coaching staff make smart decisions on tactics player fitness, scouting , preparation as well as game management . 
    • SAP has introduced a new concept called SAP "Match Insights" which analyses vast amount of data about members of the German team and their opponents, based on their on-field performance. This was made possible by the company's in-house HANA(High-PerformanceAnalyticAppliance) in- memory computing platform. 
    • With Match Insights, game data can be converted into simulations and graphs that can be viewed on tablet or smartphone , allowing trainers , coaches , and players to identify and assess key situations in each match. These insights can then be used during pre-match preparations to improve player and team performance. 
That's why Big Data is a big deal !! 
 

Article printed in Sept 2014 edition of Drishticone. By #Prashant Kumar Sharma & #Shwet kamal Mishra




Drishticone || Footer