<ul><li><p>Big Data Analysis Using Apache Hadoop </p><p>Jyoti Nandimath Dept. of Computer Engineering </p><p>SKNCOE, Pune, India jyotign@gmail.com </p><p>Ankur Patil Dept. of Computer Engineering </p><p>SKNCOE, Pune, India ankurpatil23@gmail.com </p><p>Ekata Banerjee Dept. of Computer Engineering </p><p>SKNCOE, Pune, India koool.ekata@gmail.com </p><p> Pratima Kakade Dept. of Computer Engineering </p><p>SKNCOE, Pune, India pratimakakade@gmail.com</p><p>Saumitra Vaidya Dept. of Computer Engineering </p><p>SKNCOE, Pune, India saumitravaidya28@gmail.com </p><p>Abstract The paradigm of processing huge datasets has been shifted from centralized architecture to distributed architecture. As the enterprises faced issues of gathering large chunks of data they found that the data cannot be processed using any of the existing centralized architecture solutions. Apart from time constraints, the enterprises faced issues of efficiency, performance and elevated infrastructure cost with the data processing in the centralized environment. </p><p>With the help of distributed architecture these large organizations were able to overcome the problems of extracting relevant information from a huge data dump. One of the best open source tools used in the market to harness the distributed architecture in order to solve the data processing problems is Apache Hadoop. Using Apache Hadoops various components such as data clusters, map-reduce algorithms and distributed processing, we will resolve various location-based complex data problems and provide the relevant information back into the system, thereby increasing the user experience. </p><p>Index TermsHadoop,Big data,Map Reduce, Data processing </p><p>I. INTRODUCTION </p><p>Amount of data generated every day is expanding in drastic manner.Big data is a popular term used to describe the data which is in zetta bytes. Government , companies many organisations try to acquire and store data about their citizens and customers in order to know them better and predict the customer behaviour . social networking websites generate new data every second and handling such a data is one of the major challenges companies are facing. Data which is stored in data warehouses is causing disruption because it is in a raw format ,proper analysis and processing is to be done in order to produce usable information out of it.</p><p> New tools are being used to handle such a large amount of data in short time.ApacheHadoop is java based programming framework which is used for processing large data sets in distributed computer environment. Hadoop is used in system where multiple nodes are present which can process terabytes of data.hadoop uses its own file system HDFS which facilitates fast transfer of data which can sustain node failure and avoid system failure as whole. Hadoop uses MapReduce algorithm which breaks down the big data into smaller chunks and performs the operations on it</p><p>Various technologies will come in hand-in-hand to accomplish this task such as Spring Hadoop Data Framework for the basic foundations and running of the Map-Reduce jobs, Apache Maven for distributed building of the code, REST Web services for the communication, and lastly Apache Hadoop for distributed processing of the huge dataset. </p><p>I.</p><p>II. RELATED WORKHadoop framework is used by many big companies like </p><p>Google, yahoo, IBM , for applications such as search engine, advertising and information gathering and processing </p><p>A. Apache Hadoop goes realtime at Facebook </p><p>At Facebook, Hadoop has traditionally been used in conjunction with Hive for storage and analysis of large data sets. Most of this analysis occurs in offline batch jobs and the emphasis has been on maximizing throughput and efficiency. These workloads typically read and write large </p><p>700IEEE IRI 2013, August 14-16, 2013, San Francisco, California, USA 978-1-4799-1050-2/13/$31.00 2013 IEEE</p></li><li><p>amounts of data from disk sequentially. As such, there has been less emphasis on making Hadoop performant for random access workloads by providing low latency access to HDFS. Instead, we have used a combination of large clusters of MySQL databases and caching tiers built using memcached[9]. In many cases, results from Hadoop are uploaded into MySQL or memcached for consumption by the web tier. </p><p>Recently, a new generation of applications has arisen at Facebook that require very high write throughput and cheap and elastic storage, while simultaneously requiring low latency and disk efficient sequential and random read performance MySQL storage engines areproven and have very good random read performance, but typically suffer from low random write throughput. It is difficult to scale up our MySQL clusters rapidly while maintaining good load balancing and high uptime. Administration of MySQL clusters requires a relatively high management overhead and they typically use more expensive hardware. Given our high confidence in the reliability and scalability of HDFS, we began to explore Hadoop and HBase for such applications. </p><p>B. Yelp : uses AWS and Hadoop </p><p>Yelp originally depended upon giant RAIDs to store their logs, along with a single local instance of Hadoop. When Yelp made the move Amazon Elastic MapReduce, they replaced the RAIDs with Amazon Simple Storage Service (Amazon S3) and immediately transferred all Hadoop jobs to Amazon Elastic MapReduce.</p><p>Yelp uses Amazon S3 to store daily logs and photos, generating around 100GB of logs per day. The company also uses Amazon Elastic MapReduce to power approximately 20 separate batch scripts, most of those processing the logs. Features powered by Amazon Elastic MapReduce include:</p><p>1. People Who Viewed this Also Viewed 2. Review highlights 3. Auto complete as you type on search 4. Search spelling suggestions 5. Top searches 6. AdsYelp uses MapReduce. MapReduce is about the </p><p>simplest way you can break down a big job into little pieces. Basically, mappers read lines of input, and spit out tuples of (key, value). Each key and all of its corresponding values are sent to a reducer. </p><p>III. THE PROPOSED SCHEMESAs mentioned earlier we overcome the problem of analysis </p><p>of big data using Apache Hadoop . The processing is done in </p><p>four steps which includes creating a servers of required configuration using Amazon web services using their own cloud formation language and configuring it using chef . Data on the cluster is stored using MongoDB which stores data in the form of key:value pairs which is advantageous over relational database for managing large amount of data. </p><p>Various languages allows writing scripts for importing data from collections in Mongodb to the json file which then can be processed in Hadoop as per users requirement .</p><p>Hadoop jobs are written in spring framework this jobs implement MapReduce algorithm for data processing. Six jobs are implemented in STS for a data processing in a location based social networking application. </p><p>The record of the whole session has to me maintained in log file using aspect (AOP) programming in spring. </p><p>The output produced after data processing in the hadoop job, has to be exported back to the database . the old values to the database has to be updated immediately after processing , in order to prevent loss of valuable data. </p><p>The whole process is automated by using java scripts and tasks written in ant build tool for executing executable JAR files. </p><p>A. Importing data from database to Hadoop Data is stored in mongo DB which is a NoSQL </p><p>database. Its stored in the form of key value pairs which makes it easier for latter updation and creating the index than using traditional relational database. JSON (JavaScript Object Notation ) is a data interchangeable format which is easy for humans to read and write and its easy for machines to parse and generate. Its a text format which is completely language independent but it is understandable for c, java programmer . </p><p>Using json files with Hadoop Is beneficial as it stores information in key value format. Map reduce is designed in such a way that it takes key value pairs as an input. Hence there is no additional cost of conversion . </p><p>Groovy scripts can be written to make a connection to remote database and copy collection to the json file which then will passed to HDFS to perform required operations .groovy scripts are called and run in an application using groovy class loader . </p><p>File system of hadoop is different than the file system the OS .it uses single name node and multiple data nodes to store data and perform jobs. hence json file is to be transferred the HDFS from local file system . java script is included in batch file to automate this process </p><p>B. Performing jobs in Hadoop Map/Reduce is a programming paradigm that was made popular by Google where a task is divided into small portions and distributed to a large number of nodes for processing (map), and the results are then summarized into the final answer (reduce). Hadoop also uses Map/Reduce for data processing. Hence different functions for the processing are written in the form of Hadoop job. A Hadoop job consists of mapper and reducer functions. These jobs are written in spring </p><p>701</p></li><li><p>framework i.e. STS is used as the integrated development environment (IDE). Six Hadoop jobs are implemented in spring framework. These jobs are required to process data for a location based social networking application. The input to these jobs is a JSON file containing different information fields about various locations. </p><p>The jobs and their implementation are explained below: Hadoop jobs are written for calculating: </p><p>1.Average Rating:- Ratings for a location are added from different records and average is calculated. Higher rating specifies that a particular location is more popular. </p><p>2.Total Recommendations:- Total number of recommendations given by user for a particular location are calculated. </p><p>3.Total Votes:- Total number of votes given by user for a particular location are calculated. </p><p>4.Total Suggestions:- A user can suggest a location to other users who visit the web site. Total suggestions are calculated. </p><p>5.Unique Tags:- Unique tags corresponding to a location are calculated. </p><p>C. Exporting data back to the database The output produced after data processing in the </p><p>hadoop job, has to be exported back to the database. The old values in the database has to be updated immediately after processing, in order to prevent loss of valuable data. Exporting of data is done in the following steps: </p><p>1.The output of the hadoop job is in a key value format. The key contains the location_id and value is the specific value pertaining to the data processed. </p><p>2.There are 6 different jobs, each job will have its own export program, that will copy the specific output to the mongo database. Consider the following example: Calculating the average rating for a particular location(say barbeque nation) is a hadoop job that we have implemented. Now members from all around the globe will post their ratings and comments regarding barbeque nation. We need to give a final absolute rating to barbeque nation </p><p>for our users interest. This comes under the data processing part which will be implemented in the hadoop job. </p><p>The final output of the average ratings of all places is produced in a comma separated format output file. This file in turn is used in the Groovy script to update the records in the database. </p><p>The key value pairs are extracted from the output file and a query is generated to update the database. Hadoop and mongodb are connected using the hadoop-mongodb connector. </p><p>D.Logging and automation The whole session of such a large process has to be </p><p>maintained and logged in to a file for a record. Aspect(AOP) Programming allows us to put an aspect on a class(ie to intercept a class) </p><p>1. @Before - Run before the method execution 2. @After - Run after the method returned a result 3. @Around - Run around the method execution, </p><p>combine all three advices above. </p><p>Aspect can be implemented using following steps </p><p>1.To add Project Dependencies To enable AspectJ, you need aspectjrt.jar, aspectjweaver.jar and spring-aop.jar in pom.xml file 2.Spring Bean Normal bean, with few methods, later intercept it via AspectJ annotation. 3.Enable AspectJ In Spring configuration file, put , and define your Aspect (interceptor) and normal bean. </p><p>Aspect Types : </p><p> AspectJ @Before The logBefore() method will be executed before the execution of a class method </p><p>AspectJ @After The logAfter() method will be executed after the execution of class method. </p><p>AspectJ @Around The logAround() method will be executed before the class method. To overcome the shortage of time and to run such a large </p><p>process manually the whole process has to be automated. Ant is a build tool where task can be written to automated various task. </p><p>1)Making excuted JAR file 2)Compliation of java class 3) Clean Ant </p><p>IV. CONCLUSIONThe application performs the operation on big data like counting average ratings, total recommendations, unique tags etc, in optimal time and producing an output with </p><p>702</p></li><li><p>minimum utilization of resources . The data analysis and processing is used in a social networking application. Thus providing the necessary information to the application users with least effort. </p><p>V. ACKNOWLEDGMENTWe would like to thank Prof. Jyoti Nandimath for guiding </p><p>us throughout the selection and implementation of the topic .we would also like to thank Mr.Divyansh Chaturvedi for his guidance and providing the resources for making this paper possible. </p><p>VI. REFERENCES[1.] Hadoop in Action by Chuck Lam[2.]Hadoop the definitive guide by Tom White publisher: O</p><p>REILLY [3.] Pro Hadoop- build scalable distributed applications in the </p><p>cloud by Jason Venner [4.] Michael G Noll tutorials Applied Research. Big Data. </p><p>Distributed Systems. website: http://www.michael-noll.com[5.] Expert One-on-One J2EE Design and Development by Rod</p><p>Johnson. [6.] MK Yong Spring Tutorials [7.] Groovy in Action by D-ierk Konig with Andrew Glover, Paul </p><p> KingguiIllaume Laforge, and Jon Skeet. [8.] MapReduce: Simplified Data Processing on Large </p><p>Clusters.Available at http://labs.google.com/papers/mapreduce- osdi04.pdf</p><p>[9.] Oreilly MongoDB The Definitive Guide Sep 2010[10.] Spring Hadoop reference manual by Costin Leau</p><p>703</p></li></ul>