Hadoop Big Data Engineering Specialist

1 day ago


Warszawa, Mazovia, Czech Republic Capco Poland Full time

About Capco Poland

Capco Poland is a leading global technology and management consultancy dedicated to driving digital transformation across the financial services industry. Our passion lies in helping clients navigate the complexities of the financial world, and our expertise spans banking and payments, capital markets, wealth, and asset management.

We pride ourselves on maintaining a nimble, agile, and entrepreneurial culture, and we are committed to growing our business by hiring top talent.

Job Overview

We are seeking a highly skilled and motivated Hadoop Big Data Engineer with 5+ years of experience in Hadoop administration and expertise in automation tools like Ansible, shell scripting, or Python scripting.

The ideal candidate will have strong DevOps skills and proficiency in coding, particularly in Python. This is a dynamic role focused on managing and engineering Big Data solutions across multiple open-source platforms such as Hadoop, Kafka, HBase, and Spark.

Main Responsibilities

  • Hadoop Administration: Administer and manage Hadoop clusters, ensuring high availability, performance, and scalability.
  • Maintenance and Troubleshooting: Maintain and troubleshoot Hadoop ecosystem components such as HDFS, MapReduce, YARN, and related tools.
  • Optimization and Monitoring: Ensure Kafka, HBase, and Spark systems are optimized and running smoothly.
  • Automation and Scripting: Develop and automate scripts using tools such as Ansible, Shell scripting, or Python to streamline Big Data administration tasks.
  • DevOps Practices: Apply DevOps principles to the Big Data environment, focusing on continuous integration and continuous delivery (CI/CD).
  • Troubleshooting and Debugging: Identify, troubleshoot, and resolve issues related to Big Data platforms, including system performance, resource utilization, and service failures.

Required Skills and Qualifications

We are looking for a highly skilled and motivated individual with the following qualifications:

  • 5+ years of experience in Hadoop administration and automation tools like Ansible, shell scripting, or Python scripting.
  • Strong DevOps skills and proficiency in coding, particularly in Python.
  • Experience with cloud platforms (AWS, Azure, GCP) for Big Data solutions.
  • Knowledge of containerization and orchestration tools (Docker, Kubernetes).
  • Familiarity with data pipeline frameworks and tools like Apache NiFi, Airflow, or similar.
  • Bachelor's degree in Computer Science, Engineering, or related field.

Benefits

We offer a range of benefits to support your career growth and well-being:

  • Employment contract and/or Business to Business - whichever you prefer.
  • Possibility to work remotely.
  • Speaking English on daily basis, mainly in contact with foreign stakeholders and peers.
  • Multiple employee benefits packages (MyBenefit Cafeteria, private medical care, life-insurance).
  • Access to 3.000+ Business Courses Platform (Udemy).
  • Access to required IT equipment.
  • Paid Referral Program.
  • Participation in charity events e.g. Szlachetna Paczka.
  • Ongoing learning opportunities to help you acquire new skills or deepen existing expertise.
  • Being part of the core squad focused on the growth of the Polish business unit.


  • Warszawa, Mazovia, Czech Republic Capco Poland Full time

    5+ years of hands-on experience in Big Data administration, automation, and DevOps practices, including Hadoop ecosystem tools such as HDFS, YARN, and MapReduce.Proficient in managing and troubleshooting Kafka, HBase, and Spark.Experience with performance tuning, cluster optimization, and high-availability configurations.Strong experience with automation...


  • Warszawa, Mazovia, Czech Republic Axiom software solutions Full time

    Axiom software solutions is seeking a seasoned Big Data Senior Developer to join our team.We are looking for an expert in Big Data technologies, specifically in Spark and Scala. With at least 2 years of experience working with these tools, you will be responsible for designing, developing, and maintaining large-scale data processing systems.As a key member...


  • Warszawa, Mazovia, Czech Republic Quontex Ltd Full time

    Welcome to Quontex Ltd, a leading programmatic media company specializing in ingesting large volumes of data and offering a range of products and services across Media, Analytics, and Technology.Our team is expanding rapidly, with over 700 employees and 15 global offices spanning four continents. We are seeking an experienced Data Engineer to join our team...

  • Data Engineer @

    2 days ago


    Warszawa, Mazovia, Czech Republic Quontex Ltd Full time

    3+ years of overall experience in Data Warehouse development and database designDeep understanding of distributed computing principlesExperience with AWS cloud platform, and big data platforms like EMR, Databricks, EC2, S3, RedshiftExperience with Scala, Spark, Hive, Yarn/Mesos, etc.Experience in SQL and NoSQL databases, as well as experience with data...


  • Warszawa, Mazovia, Czech Republic 7N Full time

    Opis stanowiskaJesteśmy poszukiwani przez naszego klienta, firmę z sektora bankowego, których oczekuje od nas współpracy w dalszym rozwoju autorskiego systemu do detekcji anomalii za pomocą narzędzi do budowy modeli uczenia maszynowego.Oczekujemy doświadczonego specjalisty ds. danych big data, kandydata na stanowisko Data Scientist / Big Data...


  • Warszawa, Mazovia, Czech Republic Axiom software solutions Full time

    At Axiom Software Solutions, we are looking for a skilled Backend Spark developer to join our team.About the JobWe are seeking a highly motivated and experienced professional to develop, test and deploy technical and functional specifications from our Solution Designers / Business Architects / Business Analysts, ensuring correct operability and compliance...


  • Warszawa, Mazovia, Czech Republic 7N Full time

    Nasza firma, 7N, poszukuje doświadczonego specjalisty w dziedzinie inżynierii inteligencji maszynowej na stanowisko AI Engineer / Data Scientist. Praca będzie obejmowała rozwój i powiększanie obecnego zespołu naszego klienta - lidera innowacyjnych rozwiązań bankowych.O projekcieW ramach projektu będziemy pracować nad autorskim systemem do detekcji...


  • Warszawa, Mazovia, Czech Republic Pragmile Full time

    At Pragmile, we are on a mission to maximize green energy production and reduce O&M costs through the use of artificial intelligence models.Solar Spy is an analytical platform dedicated to support solar farm operations. It processes various types of PV plant data, including image data from aerial inspections, weather data, data from photovoltaic installation...


  • Warszawa, Mazovia, Czech Republic T-Mobile Polska Full time

    Extensive experience (4+ years) in designing and delivering cloud or hybrid data platformsStrong background in data engineering and analyticsProficiency in cloud technologies (AWS, GCP)Experience with Data Warehousing, Big Data technologies and NoSQL databasesStrong leadership and communication skillsAbility to translate complex technical concepts for...


  • Warszawa, Mazovia, Czech Republic Reckitt Full time

    SAP project implementations in a global environmentSAP project delivery experience as Data Migration Lead in a global environmentSAP broad technical and functional expertiseAdvanced project management skills in complex governed environmentsExperience in managing multi-cultural teams of consultantsExperience on both commercial and manufacturing projects is a...


  • Warszawa, Mazovia, Czech Republic Asana Full time

    About youExpertise in programming, distributed systems design, and infrastructureExperience building and operating scalable, reliable, and highly-available services4+ years designing and implementing production code for backend, infrastructure, and/or data systems2+ years mentoring/coaching other team members on design and execution of projectsEagerness to...


  • Warszawa, Mazovia, Czech Republic SambaTV Full time

    Samba TV is revolutionizing the viewing experience with our innovative data and technology. Our mission is to transform the way media companies connect with audiences and advertisers engage with viewers.We are seeking an experienced Tech Lead - Data Engineering to lead the development of scalable, high-performance data pipelines that power our analytics and...


  • Warszawa, Mazovia, Czech Republic T-Mobile Polska Full time

    About UsT-Mobile Polska is a leading telecommunications company in Poland, providing innovative solutions to customers across the country.Job DescriptionWe are seeking an experienced Data Platforms Technical Lead - Cloud Architect to join our team. As a key member of our data platforms infrastructure team, you will be responsible for designing, developing,...


  • Warszawa, Mazovia, Czech Republic monday Full time

    We are seeking an experienced Database Reliability Engineer to join our rapidly growing DBRE team at monday.com. As a key member of our team, you will be responsible for ensuring the reliability and performance of our hundreds of production and pre-production database environments.The ideal candidate will have a deep understanding of RDBMS and experience...

  • Data Analyst

    4 days ago


    Warszawa, Mazovia, Czech Republic 7N Full time

    About the RoleWe are seeking a skilled Data Analyst to join our team and contribute to the development of a laboratory data platform integration project. As a Data Analyst, you will play a crucial role in building tools for bio samples research and ordering.Key ResponsibilitiesCollect, clean, and organize data to ensure its accuracy and usability.Design data...


  • Warszawa, Mazovia, Czech Republic monday Full time

    Proven working experience and deep understanding of RDBMS (MySQL / PostgreSQL/ Oracle / SQL Server / etc).Experience of at least 5+ years with performance optimization and scaling out data worldwide.Coding experience and understanding of code and software environment.Experience and understanding of cloud environments and toolsExperience in leading projects,...


  • Warszawa, Mazovia, Czech Republic Asana Full time

    About youExpertise in programming and computer science, and strong interest in distributed systems5+ years of software development and infrastructure experience2+ years of experience building and scaling high-volume online data systems (e.g. search infrastructure, data stores), and operating highly available, user-facing production servicesExperience with...


  • Warszawa, Mazovia, Czech Republic 7N Full time

    Company OverviewWe are 7N, a Swedish fintech company expanding in Poland. Our mission is to revolutionize investment decision-making by leveraging machine learning and strategic data.Job DescriptionWe are seeking an experienced Fullstack Developer to join our team. As a Fullstack Engineer, you will design, develop, and maintain scalable solutions,...


  • Warszawa, Mazovia, Czech Republic Reckitt Full time

    About the RoleThe SAP Data Migration Specialist is a key position within Reckitt, responsible for supporting local business teams in deploying data migration solutions.Key ResponsibilitiesData Migration Support: Provide end-to-end support for all activities required to operate in the target ERP system (SAP ECC 6.0)Vertical Expertise: Focus on a specific...


  • Warszawa, Mazovia, Czech Republic AVENGA Full time

    Highly competent with SIEM Engineering and Detection EngineeringGood understanding and knowledge of common industry cyber securityframeworks, standards and methodologies, including; OWASP, MITRE ATT&CK and NIST is essentialAble to work in fast paced environmentsGreat written and oral communication skillsPassion for security and love to learn and grow...