Sr. Big Data Engineer

🌍 Remote, USA 🚀 Full-time 🕐 Posted Recently

Job Description

RedCloud Consulting is a global business and technology consulting firm supporting clients across North America, Europe, Asia-Pacific, and the Middle East. For more than two decades, RedCloud has partnered with high-growth mid-market companies and global enterprises to solve complex business and operational challenges—leveraging modern data platforms, artificial intelligence, and secure digital and physical infrastructure to drive smarter decisions and scalable outcomes.

Anchored in a foundation of integrity-based consulting, RedCloud brings senior-level experts together with client teams to deliver durable, high-impact solutions across Data & AI, Delivery Excellence, Digital Solutions, Marketing & Sales, Privacy, Security, Data Center and Critical Infrastructure Security, and Strategy & Transformation. Recognized for both performance and culture, RedCloud has been named a top place to work and a fast-growing firm, and was recognized by Consulting Magazine as a 2024 Leader in Technology Consulting.

RedCloud seeks a Senior Big Data Engineer to support immediate client operations.

Overview:
The Senior Big Data Engineer will support core data platform delivery while driving engineering productivity through the development of internal accelerators and reusable tooling. This role is focused on real production impact rather than conceptual or research work. Ideal candidates are hands on engineers who thrive in high scale environments, care deeply about reliability, and enjoy building systems and tools that make other engineers more effective.

Execute scoped data engineering workstreams across areas such as
Technology migrations and consolidations across fragmented systems
Security remediation for platforms and pipelines
Data pipeline optimization to improve performance and reliability

Design and build internal tools and automation that improve engineering velocity, including
Tools for DDL creation and schema definition
Airflow DAG scaffolding and workflow templates
Spark configuration helpers and optimization utilities
Standardized templates, generators, and reusable patterns to reduce repetitive work

Build and maintain high volume batch and streaming data pipelines
Ensure reliability through monitoring, retries, and adherence to SLAs
Operate effectively within enterprise environments with strong controls and security standards

Clear examples of building production grade ETL or ELT pipelines end to end
Deep knowledge of Spark tuning including partitions, shuffle, joins, caching, executor sizing, and skew mitigation
Strong understanding of streaming fundamentals including ordering, backpressure, idempotency, replay handling, and schema evolution
Operational excellence in Airflow including operators, sensors, dependencies, SLAs, and backfills
Strong DevOps hygiene including Git workflows, CI CD exposure, and code review participation
Candidates should have experience creating tools used across engineering teams such as:
Pipeline scaffolding generators or starter kits
Standardized repository templates and recommended workflow patterns
Automated DDL or schema generation tooling
Configuration or Spark tuning automation frameworks
Developer tools that eliminate repetitive tasks and increase throughput
Candidates should be prepared to speak to what they automated, who used it, and what measurable impact it had.
Core Technologies:
Apache Spark including batch processing and performance tuning
Apache Kafka including streaming and event driven pipeline patterns
Apache Airflow including DAG authoring and orchestration best practices
Programming Languages:
Strong proficiency with Scala
Experience with Java is also valuable
Production and Systems Experience:
High throughput data pipelines and distributed systems engineering
Production grade reliability practices
Strong understanding of data modeling, schema evolution, and workflow orchestration
Cloud Environment:
Ability to work confidently in cloud platforms outside of Azure such as AWS or GCP

Compensation range for position is $ 156,000 – 250,000 DOE.
Benefits and bonus information can be found at https://www.redcloudconsulting.com/careers

About Us:
RedCloud Consulting is a global business and technology consulting firm supporting clients across North America, Europe, Asia-Pacific, and the Middle East. For more than two decades, RedCloud has partnered with organizations ranging from high-growth mid-market companies to global enterprises, helping them solve complex business and operational challenges—leveraging modern data platforms, artificial intelligence, and secure digital foundations to drive smarter decisions and scalable outcomes.

Anchored in a foundation of integrity-based consulting, RedCloud brings together senior-level experts who work side by side with client teams to design and implement durable, high-impact solutions. Our approach emphasizes trust, accountability, and long-term value—balancing stability, growth, and innovation to help organizations operate securely and confidently in an increasingly complex global environment.

RedCloud delivers a comprehensive range of services across its core practices: Data & AI, Delivery Excellence, Digital Solutions, Marketing & Sales, Privacy, Security, Data Center and Critical Infrastructure Security, and Strategy & Transformation. As our clients expand globally, so do we—providing consistent, expert support across regions while remaining agile, collaborative, and deeply invested in client success.

Recognized for both performance and culture, RedCloud has been named #1 Best Company to Work For in Washington (Mid-Sized Businesses) by Seattle Business Magazine, a “Fastest Growing Company” multiple times on the Inc. 500/5000 list, and recognized by Consulting Magazine as a Leader in Technology Consulting.

Visit http://www.redcloudconsulting.com/ for more info.
#LI-Remote

Apply tot his job

Apply To this Job