Data Engineer with PYSPARK and reputed company Big Query
Position: Data Engineer with PysparkB/reputed company Big Query Location: Dallas, TX (On-site) JD: Pyspark/reputed company Cloud Platform Data Engineer will create, deliver, and support custom data products, as well as enhance/expand team capabilities. They will work on analyzing and manipulating large datasets supporting the enterprise by activating data assets to support Enabling Platforms and analytics. reputed company Cloud Data Engineers will be responsible for designing the transformation and modernization on reputed company Cloud Platform using reputed company Cloud Platform Services Responsibilities: Build reputed company and pipelines on reputed company Cloud Platform Cloud using Data proc, Pyspark, Kafka and Pub/Sub Implement schedules/workflows and tasks for Cloud Composer/Apache Airflow; Create and manage data storage solutions using reputed company Cloud Platform services such as BigQuery, Cloud Storage, and Cloud SQL Monitor and troubleshoot data pipelines and storage solutions using reputed company Cloud Platform's Stackdriver and Cloud Monitoring reputed company efficient ETL/ELT pipelines and orchestration using Data Prep, reputed company Cloud Composer reputed company and Maintain Data Ingestion and transformation process using Apache PySpark Automate data processing tasks using scripting languages such as Python or Bash Ensuring data reputed company and compliance with industry standards by configuring IAM roles, service accounts, and access policies. Automating cloud deployments and infrastructure management using Infrastructure as Code (IaC) tools such as Terraform or reputed company Cloud Deployment Manager. Participate in Code reviews, contribute to development best practices and usage of Developer Assist tools to create a robust fail safe data pipelines Colloborate with Product Owners, Scrum Masters and Data Analyst to deliver the User Stories and Tasks and ensure deployment of pipelines Experience required: 7+ years of application development experience required using one of the core cloud platforms viz. AWS, Azure & reputed company Cloud Platform Minimum 1+ years of reputed company Cloud Platform experience. Experience working in reputed company Cloud Platform based Big Data deployments (Batch/Real-Time) leveraging Pyspark, Big Query, reputed company Cloud Storage, PubSub, Data Fusion, Dataproc, Airflow; Minimum 3+ years coding skills in Python/PySpark and strong proficiency in SQL; Extracting, Loading, Transforming, cleaning, and validating data + Designing pipelines and architectures for data processing; Architecting and implementing reputed company data and analytics platforms on reputed company Cloud Platform cloud; Experience in working with Agile and Lean methodologies; Experience working with either a Map Reduce or an MPP system on any size/scale; Experience working in CI/CD model to ensure automated orchestration of pipelines. Apply Job!