Principal Big Data Engineer
At Great Eastern, we are building a data platform to realize the full value of our data. We seek an experienced Principal Big Data Engineer to join our growing data engineering and analytics team. The right person for the job will have strong knowledge of Big Data Engineering and proven ability to strategize the implementation of data and analytics capabilities on a data platform (on premise and cloud) - from conception through release and production. The ideal candidate will be a strong data engineer; be willing to wrangle data, optimize data systems and products, and build them from the ground up. Experience with AWS, Snowflake, and migration to public could would be an advantage. The Job
The Principal Big Data Engineer will work with cross-functional teams like Data & Analytics (Data Engineer, Data Visual Analyst and Data Scientist), Data Management and Governance, IT and Digital Platform to support data curation and analytics.
- Assess and implement robust and scalable Big Data Technologies/Architecture to enable optimal data extraction, ingestion, transformation and storage from a wide variety of data sources.
- Architect end-to-end solution for Business Analytics Product (Dashboards or Statistical Model) from the curation of data, contextualizing data for business analytics and integrating of Product with Business process.
- Build and maintain an efficient, scalable and future-proof deployment infrastructure to enable the development and deployment of production quality analytics and AI applications.
- Engineer, optimize, fine-tune and maintain efficient, secure and reliable data pipelines to ingest, clean and consolidate data sources into the analytics systems and solutions.
- Provide technical guidance to Junior Data Engineers/Analysts on complex data issues, handling of big data sets and use of advanced methodologies.
- Recommend and implement ways to improve data reliability, efficiency and quality through the use of programming languages and big data processing/manipulation tools.
- Conduct research on emerging Big Data Architecture, Technologies and Systems to ensure that they continue to support the requirements of the data scientists and the business stakeholders in the mid to long-term.
- Work with data scientists to operationalize and maintain analytics/ artificial intelligence solutions (e.g. applications, models) by integrating them into business processes. This includes converting proof-of-concepts developed by data scientists into production-grade products and converting machine learning models in Application Program Interfaces (APIs).
- Takes accountability in considering business and regulatory compliance risks and takes appropriate steps to mitigate the risks.
- Maintains awareness of industry trends on regulatory compliance, emerging threats and technologies in order to understand the risk and better safeguard the company.
- Highlights any potential concerns /risks and proactively shares best risk management practices.