Senior Data Engineer - Oil & Gas

About the role

Insight Enterprises, Inc., a Fortune 500 solutions integrator with expertise in cloud, data, AI, cybersecurity and intelligent edge, is hiring a Senior Data Engineer in Moldova. The role combines data engineering, data modelling, data architecture and integration knowledge with business understanding, and works with a team of data engineers and data analysts to deliver projects on time using industry best practices.

Key responsibilities

• Act as an integration expert and data solution architect, delivering end-to-end solutions and influencing key technical decisions with pros, cons and risks.
• Define and implement a global data platform strategy that centralises, normalises and democratises the client's data stores, and become an expert on EPC, global asset management and oil field operations datasets.
• Design data flows for big data and data lake use cases, and coach data engineers so they follow architecture principles, design standards and operational requirements.
• Work with analytics and business stakeholders to improve data models feeding BI tools, and estimate cluster size, cores, monitoring and troubleshooting of the data bricks cluster and analysis server.
• Deliver master data cleansing and improvement, including automated solutions for processing, cleansing and verifying data integrity.
• Secure the big data environment (encryption, tunnelling, access control, secure isolation) and build efficient OLAP cubes to overcome Power BI limitations.
• Deploy and maintain CI/CD DevOps pipelines across dev, staging and production, following a Scrum-based agile approach.

Required profile

• Experience defining, implementing and maintaining a global data platform.
• Hands-on experience onboarding data sources via real-time, batch or scheduled loads from cloud, on-premise, SQL, NoSQL or API sources.
• Experience with data extraction via JSON, OData, REST APIs, web services and XML.
• Experience with data ingestion platforms such as Apache Sqoop, Apache Flume, Amazon Kinesis, Fluent and Logstash.
• Experience supporting cross-functional teams in a dynamic environment.

Required skills

• Data extraction, transformation and loading from Oracle, Hadoop HDFS, flat files, JSON, Avro, Parquet and ORC.
• Databricks, Pig, Scala, Hive, Azure Data Factory, Python and R.
• Presto, Spark, Hive and Hadoop environments.
• Azure SQL DB, Oracle, MySQL, Cosmos DB and MongoDB.
• Data visualisation solutions.