Work Location: Sydney
Experience: More than 5 years
· Over all 5 years’ experience, with at least 2 years’ experience on Hadoop technologies.
· Install, configure and maintain enterprise hadoop environment.
· Loading data from different datasets and deciding on which file format is efficient for a task. Hadoop developers source large volumes of data from diverse data platforms into Hadoop platform.
· Understanding the requirements of input to output transformations.
· Defining Hadoop Job Flows.
· Build distributed, reliable and scalable data pipelines to ingest and process data in real-time. Hadoop developer deals with fetching impression streams, transaction behaviours, clickstream data and other unstructured data.
· Design and implement column family schemas of Hive and HBase within HDFS.
· Assign schemas and create Hive tables.
· Managing and deploying HBase clusters.
· Develop efficient pig and hive scripts with joins on datasets using various techniques.
· Assess the quality of datasets for a hadoop data lake.
· Apply different HDFS formats and structure like Parquet, Avro, etc. to speed up analytics.
· Build new hadoop clusters
· Maintain the privacy and security of hadoop clusters.
· Fine tune hadoop applications for high performance and throughput.
If interested and have the skills required or know of someone who may be interested please contact me at firstname.lastname@example.org for details or kindly share me your Resume.