Senior /Staff Data & Cloud Platform Engineer

Responsibilities 

• Data Pipeline Engineering: Build scalable ingestion pipelines for structured, semi-structured, and engineering data sources. 

• Workflow Orchestration: Develop reliable batch, scheduled, and event-driven workflows using Apache Airflow and cloud-native services. 

• ETL / ELT Frameworks: Implement reusable pipeline patterns, transformations, incremental ingestion, and data quality checks. 

• Schema Management: Implement schema registry, schema versioning, schema evolution, and compatibility controls. 

• Delta Detection: Build change detection and incremental refresh mechanisms for efficient large-scale data synchronization. 

• Entity Extraction: Extract and normalize key identifiers such as Lot, Die, Wafer, Test, Flow, Product, and Step. 

• Cloud Landing Zones: Set up secure AWS, GCP, and on-prem landing targets for validation and analytics pipelines. 

• Infrastructure Automation: Provision infrastructure using Terraform, CI/CD, and Infrastructure-as-Code practices. 

• Platform Operations: Support containerized workloads using Kubernetes, Docker, monitoring, logging, and operational controls. 

• Security & Cost Optimization: Implement IAM, RBAC, hybrid connectivity, access controls, and cost optimization for TB-scale data. 

• Cross-Functional Delivery: Partner with IT, TPG AI, SMAI, Data Science, Product Engineering, and platform teams. 

Expertise 

• Data Pipeline Engineering: Design and operation of scalable ingestion pipelines, ETL/ELT frameworks, metadata-driven processing, and data quality validation. 

• Workflow Orchestration: Development and management of batch, event-driven, and scheduled workflows using Apache Airflow. 

• Cloud Data Platforms: Building data solutions using GCP and AWS services for ingestion, processing, storage, and analytics. 

• Schema & Metadata Management: Implementation of schema registries, schema evolution, version control, metadata capture, and lineage readiness. 

• Incremental Data Processing: Development of delta detection, CDC-style processing, watermarking, and efficient refresh strategies. 

• Manufacturing Data Modeling: Extraction and normalization of Lot, Die, Wafer, Test, Flow, Product, Step, and validation identifiers. 

• Multi-Cloud Architecture: Secure and scalable data platforms across AWS, GCP, and on-premises environments. 

• Infrastructure Automation: Cloud provisioning using Terraform, automation frameworks, reusable modules, and CI/CD practices. 

• Container & Platform Engineering: Operation of containerized applications and platform workloads using Kubernetes and Docker. 

• Cloud Security & Governance: IAM, RBAC, data protection, service accounts, secrets management, and enterprise access controls. 

• Hybrid Cloud Connectivity: Networking, VPN, private connectivity, and secure integration between cloud and on-prem environments. 

• Reliability & Cost Management: Monitoring, observability, performance tuning, troubleshooting, and optimization for TB-scale workloads. 

Qualifications 

• Education: Bachelor's degree in Computer Science, Data Engineering, Cloud Engineering, Information Systems, or related technical field. 

• Experience: 5+ years in data engineering, cloud engineering, platform engineering, DevOps, or infrastructure engineering. 

• Programming: Strong hands-on experience with Python, SQL, automation scripts, and cloud-native data services. 

• Cloud Platforms: Practical experience with GCP services such as Dataflow, Pub/Sub, Cloud Functions, BigQuery, Cloud Storage, IAM, and AWS services such as Glue, Lambda, S3, Redshift, IAM. 

• Infrastructure & DevOps: Experience with Terraform, Kubernetes, Docker, CI/CD, cloud networking, IAM, and operational automation. 

• Collaboration: Ability to partner with IT, AI, Data Science, Engineering, and manufacturing stakeholders to deliver reliable platform capabilities. 

• Proven ability to leverage AI‑assisted (vibe) coding techniques to improve efficiency or automate design and analysis methodologies

• Leverage AI tools to automate the tools and workflow

  Applying Artificial Intelligence in workflows to improve build efficiency

Preferred Domain Exposure 

• Industrial / Engineering Context: Semiconductor, NAND, product engineering, validation, manufacturing, reliability, quality, test, yield, or RCA data environments. 

• AI / Analytics Platforms: Data foundations for AI/ML, analytics, knowledge platforms, digital thread, traceability, or governed data products. 

• Certifications: AWS, GCP, Kubernetes, Terraform, or data engineering certifications are preferred but not mandatory.