Big Data Engineer for Hadoop Ecosystem

3 days ago


Warszawa, Mazovia, Czech Republic Capco Poland Full time

Capco Poland is a leading global technology and management consultancy, dedicated to driving digital transformation across the financial services industry.

We are seeking a highly skilled and motivated Big Data Administrator/Engineer with 5+ years of experience in Hadoop administration and expertise in automation tools like Ansible, shell scripting, or Python scripting.

The ideal candidate will have strong DevOps skills and proficiency in coding, particularly in Python. This is a dynamic role focused on managing and engineering Big Data solutions across multiple open-source platforms such as Hadoop, Kafka, HBase, and Spark.

Main Responsibilities:
  • Big Data Administration:
  • Administer and manage Hadoop clusters, ensuring high availability, performance, and scalability.
  • Maintain and troubleshoot Hadoop ecosystem components such as HDFS, MapReduce, YARN, and related tools.
  • Ensure Kafka, HBase, and Spark systems are optimized and running smoothly.
  • Implement monitoring and alerting for Big Data infrastructure.
Automation and Scripting:
  • Develop and automate scripts using tools such as Ansible, Shell scripting, or Python to streamline Big Data administration tasks.
  • Create reusable automation frameworks to reduce manual efforts and improve operational efficiency.
  • Work on CI/CD pipelines for deployment automation and system integration.
DevOps Practices:
  • Apply DevOps principles to the Big Data environment, focusing on continuous integration and continuous delivery (CI/CD).
  • Build and manage automated deployment processes for Big Data clusters and services.
  • Collaborate with development teams to integrate automation in Big Data workflows.
Troubleshooting and Debugging:
  • Identify, troubleshoot, and resolve issues related to Big Data platforms, including system performance, resource utilization, and service failures.
  • Work with logs, monitoring tools, and other debugging techniques to diagnose and resolve complex issues.
Collaboration and Support:
  • Work closely with other teams to support data pipelines, data quality checks, and performance optimizations.
  • Provide ongoing technical support to ensure that Big Data systems are stable, secure, and aligned with business objectives.

Requirements:

  • Big data
  • DevOps
  • Hadoop
  • HDFS
  • Yarn
  • Kafka
  • HBase
  • Spark
  • Performance tuning
  • Ansible
  • Shell
  • Python
  • Terraform
  • Docker
  • Kubernetes
  • Degree
  • Cloud platform
  • Azure
  • GCP
  • Apache
  • NiFi
  • Airflow

Benefits:

  • Employment contract and/or Business to Business - whichever you prefer
  • Possibility to work remotely
  • Speaking English on daily basis, mainly in contact with foreign stakeholders and peers
  • Multiple employee benefits packages (MyBenefit Cafeteria, private medical care, life-insurance)
  • Access to 3.000+ Business Courses Platform (Udemy)
  • Access to required IT equipment
  • Paid Referral Program
  • Participation in charity events e.g. Szlachetna Paczka
  • Ongoing learning opportunities to help you acquire new skills or deepen existing expertise
  • Being part of the core squad focused on the growth of the Polish business unit
  • A flat, non-hierarchical structure that will enable you to work with senior partners and directly with clients
  • A work culture focused on innovation and creating lasting value for our clients and employees


  • Warszawa, Mazovia, Czech Republic Capco Poland Full time

    5+ years of hands-on experience in Big Data administration, automation, and DevOps practices, including Hadoop ecosystem tools such as HDFS, YARN, and MapReduce.Proficient in managing and troubleshooting Kafka, HBase, and Spark.Experience with performance tuning, cluster optimization, and high-availability configurations.Strong experience with automation...


  • Warszawa, Mazovia, Czech Republic 7N Full time

    Praca dla Specjalisty Danych Big DataTwój zwiększony potencjał czeka w 7N, firmy doradztwa IT, która buduje najbardziej zaawansowane rozwiązania big data. Pracując u nas, zdobędziesz doświadczenie przy stworzeniu autorskich systemów detekcji anomalii i będziesz miał okazję wykorzystać narzędzia do budowy zaawansowanych modeli uczenia...


  • Warszawa, Mazovia, Czech Republic dotData Full time

    Minimum Qualifications:3+ years professional experience building data science-driven solutions including data processing, feature engineering/selection, model training/tuning, and post-deployment validationStrong knowledge of data analytics and machine learning, and familiarity with common use cases across different industriesStrong hands-on coding skills...


  • Warszawa, Mazovia, Czech Republic Pragmile Full time

    We are looking for a skilled Distributed Data Engineer to join our team at Pragmile. Our company is dedicated to supporting solar farm operations through an analytical platform that processes various types of PV plant data.The ideal candidate will have experience with designing and implementing scalable data architectures, capable of processing large amounts...


  • Warszawa, Mazovia, Czech Republic Global Payments Full time

    Minimum QualificationsBS in Computer Science, Information Technology, Business / Management Information Systems or related fieldExperience in lieu of degree will be consideredTypically minimum of 4 years of strong development background in ETL tools like GCP-Data Flow , PySpark , SSISExperience in Google cloud platform - GCP Pub/Sub, Datastore, BigQuery,...

  • Data Architect @

    2 days ago


    Warszawa, Mazovia, Czech Republic ITDS Full time

    You're ideal for this role if you have:Minimum 5 years of experience working with data architectureExpertise in SQL Server, including modeling, indexing, and performance tuningHands-on experience with ETL processes and tools like Informatica Power CenterKnowledge of cloud storage platforms and their integrationFamiliarity with big data technologies such as...


  • Warszawa, Mazovia, Czech Republic Global Payments Full time

    As a Senior Software Data Engineer at Global Payments, you will play a crucial role in designing, implementing, and testing our data pipelines. Your expertise in dimensional modeling and data engineering best practices will be instrumental in architecting data models, data warehouse/datalake architecture, and optimizing data pipelines for high-performance...


  • Warszawa, Mazovia, Czech Republic T-Mobile Polska Full time

    Extensive experience (4+ years) in designing and delivering cloud or hybrid data platformsStrong background in data engineering and analyticsProficiency in cloud technologies (AWS, GCP)Experience with Data Warehousing, Big Data technologies and NoSQL databasesStrong leadership and communication skillsAbility to translate complex technical concepts for...


  • Warszawa, Mazovia, Czech Republic T-Mobile Polska Full time

    Job Description:T-Mobile Polska is seeking a highly skilled Senior Cloud Data Solutions Engineer to join our team. As a key member of our data platform infrastructure, you will be responsible for designing, developing, and managing scalable, robust, and secure data solutions.Key Responsibilities:Design and Develop Data Platforms: Lead the design and...


  • Warszawa, Mazovia, Czech Republic Asana Full time

    About youDegree in Computer Science, Engineering or equivalent technical field experience6+ years of hands-on experience in Data Engineering or Software Engineering.Fluent in SQL and proficient in at least one programming language (e.g., Python, Java, Scala etc.)Strong expertise in Databricks, AWS S3, Spark, Snowflake, and AirflowKnowledge of system and...


  • Warszawa, Mazovia, Czech Republic Asana Full time

    About AsanaWe are a rapidly growing technology company that helps teams orchestrate their work, from small projects to strategic initiatives.Millions of teams around the world rely on Asana to achieve their most important goals, faster.Our Enterprise Data & Intelligence (EDI) team is tasked with building powerful decision-making data products, integrations,...


  • Warszawa, Mazovia, Czech Republic SambaTV Full time

    5+ years of experience in Data Engineering, Software Engineering, or a related field.Must-have expertise in Apache Airflow, Databricks, and PySpark for orchestration, scalable data processing, and transformation.Expertise in Python or strong proficiency in another modern programming language with a willingness to master Python.Deep understanding of...


  • Warszawa, Mazovia, Czech Republic Pragmile Full time

    What we expect: Experience of working with big data, data engineering tools for data ingestion, transformations, querying Strong understanding of data structures and algorithms Strong understanding of object design and integration patterns Proven experience of using python for building data processing platforms/applications Experience in building distributed...


  • Warszawa, Mazovia, Czech Republic Asana Full time

    About youExpertise in programming, distributed systems design, and infrastructureExperience building and operating scalable, reliable, and highly-available services4+ years designing and implementing production code for backend, infrastructure, and/or data systems2+ years mentoring/coaching other team members on design and execution of projectsEagerness to...


  • Warszawa, Mazovia, Czech Republic Asana Full time

    About YouWe are seeking a highly skilled Senior Data Engineer to join our team. This individual will be responsible for designing, building, and maintaining scalable data pipelines using modern cloud-based architectures.The ideal candidate will have 6+ years of hands-on experience in data engineering or software engineering, with a strong expertise in...


  • Warszawa, Mazovia, Czech Republic dotData Full time

    Company Overview:dotData is a leader in automating data science solutions for businesses. Our team is passionate about empowering organizations to unlock the full potential of their data.Job Description:We are seeking an experienced Data Scientist to join our customer-facing team. As a trusted advisor, you will work closely with clients to understand their...


  • Warszawa, Mazovia, Czech Republic SambaTV Full time

    Samba TV is a leading provider of data and technology solutions for the media industry.We are seeking an experienced Tech Lead - Data Engineering to join our team. As a key member of our data engineering team, you will be responsible for designing and implementing scalable, high-performance data pipelines that power Samba TV's analytics and insights.The...


  • Warszawa, Mazovia, Czech Republic SambaTV Full time

    Bachelor's or Master's degree in a quantitative field (e.g., Computer Science, Statistics, Mathematics, Data Science, or similar).5+ years of experience in data analysis, data processing, or related fields.Proficiency in at least one programming language (e.g., Python, R, or SQL) with a strong foundation in analytical techniques.Practical experience with...


  • Warszawa, Mazovia, Czech Republic Digital Turbine Full time

    2+ years of experience in software engineeringBachelor Degree in Computer Science, Mathematics, or related fieldProficiency in any JVM-based (Java, Kotlin, Scala)Self motivation to achieve something remarkableProficient communication in English is a mustNice to have skills:Experience with Apache Kafka, knowledge of stream processing framework like Apache...


  • Warszawa, Mazovia, Czech Republic monday Full time

    Proven working experience and deep understanding of RDBMS (MySQL / PostgreSQL/ Oracle / SQL Server / etc).Experience of at least 5+ years with performance optimization and scaling out data worldwide.Coding experience and understanding of code and software environment.Experience and understanding of cloud environments and toolsExperience in leading projects,...