Senior Data Engineer

We are seeking a Senior Data Engineer to lead the development and optimization of data pipelines, data movement tools, and data preparation services to support advanced analytics and enterprise integration. This role requires deep expertise in handling both structured and unstructured data, designing scalable data architectures, and developing reliable ETL solutions. The ideal candidate has significant experience with both cloud-native and on-premise technologies, and thrives in collaborative, mission-driven environments.

Clearance Requirement: TS/SCI with Polygraph

• Design, implement, and maintain scalable data pipelines using tools such as Spark, Apache Iceberg, Trino, OpenSearch, EMR, NiFi, and Kubernetes

• Perform data ingestion, extraction, formatting, and transformation for a wide range of data types and formats

• Clean, preprocess, and format data for exploration, analysis, and operational use

• Develop ETL tools and scripts and enhance existing pipelines using coding best practices

• Work with Data Scientists, analysts, and mission partners to align data pipelines with mission goals

• Collaborate with software engineers to configure and maintain data services and back-end components

• Ensure data quality, standardization, and provenance through collaboration with testing and data quality teams

• Produce comprehensive documentation including ETL mappings, data lineage, code usage guides, and access instructions

• Participate in enterprise working groups to define and promote data standards across systems

• Support both one-time and ongoing data extraction and transformation efforts across various repositories

• 10+ years of experience in data lifecycle engineering

• Deep expertise in ETL development, pipeline design, and data architecture

• Strong coding skills in Python, SQL, Spark, and other data engineering tools

• Experience with COTS and open-source tools including ElasticSearch and Apache NiFi

• Strong understanding of cloud and on-premise data storage and processing solutions

• Experience working in Agile development environments

• Proven track record of integrating diverse data sources into enterprise systems

• Experience deploying solutions on cloud platforms (e.g., AWS EMR, S3, RDS, EKS)

• Familiarity with data governance, data lineage tracking, and security requirements

• Hands-on experience with container orchestration using Kubernetes

• Knowledge of data lake architectures and streaming data processing

• Excellent communication skills and experience collaborating with cross-functional teams

• Languages & Frameworks: Python, SQL, Spark

• Data Tools: Apache NiFi, Apache Iceberg, Trino, ElasticSearch, OpenSearch

• Cloud & Infrastructure: AWS EMR, Kubernetes, Docker, On-Prem Systems

• DevOps & CI/CD: Git, Agile methodologies, automated testing frameworks

• Tackle large-scale data engineering challenges with national impact

• Work with cutting-edge cloud and open-source data tools

• Collaborate with a multidisciplinary team of engineers and analysts

• Flexible, mission-driven environment with opportunities for leadership

M1 Technology is an equal opportunity employer and values diversity. We do not discriminate in hiring on the basis of race, color, religion, sex, national origin, age, disability, veteran status, or any other characteristic protected by federal, state, or local law.