Hadoop engineer

  • Attractive
  • Singapore
  • Permanent, Full time
  • Non-disclosed
  • 11 Dec 17 2017-12-11

The Hadoop engineer is a key role in the Transformation project of the bank. He's an expert in Hadoop stacks (Spark, Scala, Sqoop...) and in data related problems and how to solve them.

Role:

The Hadoop Engineer will be a part of the Data Lake team. He understands the modern data platform ecosystem & is expert in applying tools in solving the common data pipeline patterns in enterprise data lake projects covering sourcing, ingestion, transformation, consumption & presentation tiers.
The candidate for this position must be capable of playing different roles within the team, from technical lead on technologies around big data & data fabric to highly skilled developer.
The candidate must have a sound understanding of Hadoop and related tools but also very good communication skills, with the ability to adapt to different needs on project from idea to implementation.

 

Qualification & Experience:

  • 10+ years of total IT experience
  • 4+ years on Data Sourcing, Quality, Warehousing, Mining & ETL tools
  • 3+ years into Big Data platform like Hadoop like Hortonworks HDP 2.5+
  • Strong critical thinker with problem solving aptitude.
  • Excellent written and oral communication skills
  • Hands on design and development experience on Hadoop Data Platform preferably Hortonworks (HDP) covering sourcing, ingestion, processing & consumption workloads
  • Hands on programming experience on Big data ETL workloads using Sqoop, Flume, Spark Core/SQL API (Scala), Hive (Tez/LLAP) and Phoenix.
  • Hands on experience on Storage covering Schema layout, data modeling, partitioning & Read/Write API for Relational, MPP & NoSQL stack including File Format (Avro, Parquet), Hive, HBase, MongoDB & Oracle.
  • Strong experience in setting up data governance, data security, metadata management, lineage tracking on Hadoop Platform using Kerberos, Ranger Policy, Atlas & Ambari
  • Job Monitoring, debugging, Scheduling and Performance Tuning using Data Platform Operations Tools like Oozie, YARN Settings, Spark tuning, Ambari Config & Grafana
  • Experience in implementing Informatica Data Fabric Tools covering Enterprise Information Catalog, Intelligent Data Lake, Big data management tools on top of Hadoop