Hadoop Big Data Administrator/Engineer

1 day ago

Warszawa, Czech Republic Capco Poland Full time
  • 5+ years of hands-on experience in Big Data administration, automation, and DevOps practices, including Hadoop ecosystem tools such as HDFS, YARN, and MapReduce.
  • Proficient in managing and troubleshooting Kafka, HBase, and Spark.
  • Experience with performance tuning, cluster optimization, and high-availability configurations.
  • Strong experience with automation tools such as Ansible, Shell scripting, and Python scripting.
  • Ability to automate deployment, monitoring, and administration tasks to increase operational efficiency
  • Solid understanding of DevOps concepts and practices, including CI/CD, version control, and deployment pipelines.
  • Hands-on experience with automation of infrastructure provisioning and management using tools like Terraform, Docker, and Kubernetes (optional but desirable).
  • Proficiency in at least one programming language, with a preference for Python.
  • Ability to write efficient, maintainable code for automation and integration tasks.
  • Strong debugging skills and experience in resolving complex issues across distributed systems.
  • Ability to think critically and provide practical solutions in high-pressure situations.
  • Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience).

Nice to Have:

  • Experience with cloud platforms (AWS, Azure, GCP) for Big Data solutions.
  • Knowledge of containerization and orchestration tools (Docker, Kubernetes).
  • Familiarity with data pipeline frameworks and tools like Apache NiFi, Airflow, or similar.

*We are looking for Poland based candidate. 

Joining Capco means joining an organisation that is committed to an inclusive working environment where you’re encouraged to #BeYourselfAtWork. We celebrate individuality and recognize that diversity and inclusion, in all forms, is critical to success. It’s important to us that we recruit and develop as diverse a range of talent as we can and we believe that everyone brings something different to the table – so we’d love to know what makes you different. Such differences may mean we need to make changes to our process to allow you the best possible platform to succeed, and we are happy to cater to any reasonable adjustments you may require. You will find the section to let us know of these at the bottom of your application form or you can mention it directly to your recruiter at any stage and they will be happy to help.

Capco Poland is a global technology and management consultancy specializing in driving digital transformation across the financial services industry. We are passionate about helping our clients succeed in an ever-changing industry.

We also are experts in Java, Python, Spring, Hadoop, Angular, React, Android, Google Cloud, Selenium, SQL, Docker, Kubernetes focused on development, automation, innovation, and long-term projects in financial services. In Capco, you can code, write, create, and live at your maximum capabilities without getting dull, tired, or foggy.

Capco Poland is a leading global technology and management consultancy, dedicated to driving digital transformation across the financial services industry. Our passion lies in helping our clients navigate the complexities of the financial world, and our expertise spans banking and payments, capital markets, wealth, and asset management. We pride ourselves on maintaining a nimble, agile, and entrepreneurial culture, and we are committed to growing our business by hiring top talent.

We are seeking a highly skilled and motivated Big Data Administrator/Engineer with 5+ years of experience in Hadoop administration and expertise in automation tools like Ansible, shell scripting, or Python scripting. The ideal candidate will have strong DevOps skills and proficiency in coding, particularly in Python. This is a dynamic role focused on managing and engineering Big Data solutions across multiple open-source platforms such as Hadoop, Kafka, HBase, and Spark.

You will be responsible for performing critical Big Data administration, troubleshooting, debugging, and ensuring the seamless operation of various data processing frameworks. If you are a hands-on, results-driven individual with a passion for Big Data technologies, this is the role for you.

WHY JOIN CAPCO?

  • Employment contract and/or Business to Business - whichever you prefer
  • Possibility to work remotely
  • Speaking English on daily basis, mainly in contact with foreign stakeholders and peers
  • Multiple employee benefits packages (MyBenefit Cafeteria, private medical care, life-insurance)
  • Access to 3.000+ Business Courses Platform (Udemy)
  • Access to required IT equipment
  • Paid Referral Program
  • Participation in charity events e.g. Szlachetna Paczka
  • Ongoing learning opportunities to help you acquire new skills or deepen existing expertise
  • Being part of the core squad focused on the growth of the Polish business unit
  • A flat, non-hierarchical structure that will enable you to work with senior partners and directly with clients
  • A work culture focused on innovation and creating lasting value for our clients and employees

ONLINE RECRUITMENT PROCESS STEPS

  • Screening call with Recruiter
  • Technical interview
  • Meeting with Hiring Manager
  • Feedback/Offer


,[Big Data Administration:, Administer and manage Hadoop clusters, ensuring high availability, performance, and scalability., Maintain and troubleshoot Hadoop ecosystem components such as HDFS, MapReduce, YARN, and related tools., Ensure Kafka, HBase, and Spark systems are optimized and running smoothly., Implement monitoring and alerting for Big Data infrastructure., Automation and Scripting:, Develop and automate scripts using tools such as Ansible, Shell scripting, or Python to streamline Big Data administration tasks., Create reusable automation frameworks to reduce manual efforts and improve operational efficiency., Work on CI/CD pipelines for deployment automation and system integration., DevOps Practices:, Apply DevOps principles to the Big Data environment, focusing on continuous integration and continuous delivery (CI/CD)., Build and manage automated deployment processes for Big Data clusters and services., Collaborate with development teams to integrate automation in Big Data workflows., Troubleshooting and Debugging:, Identify, troubleshoot, and resolve issues related to Big Data platforms, including system performance, resource utilization, and service failures., Work with logs, monitoring tools, and other debugging techniques to diagnose and resolve complex issues., Collaboration and Support:, Work closely with other teams to support data pipelines, data quality checks, and performance optimizations., Provide ongoing technical support to ensure that Big Data systems are stable, secure, and aligned with business objectives.] Requirements: Big data, DevOps, Hadoop, HDFS, yarn, Kafka, HBase, Spark, Performance tuning, Ansible, Shell, Python, Terraform, Docker, Kubernetes, Degree, Cloud platform, Azure, GCP, Apache, NiFi, Airflow Additionally: Private healthcare, Employee referral bonus, MyBenefit, Udemy for business.

  • Senior Data Engineer for Big Data Projects

    7 days ago

    Warszawa, Mazovia, Czech Republic Axiom software solutions Full time

    We are seeking a highly skilled Senior Data Engineer to join our team at Axiom software solutions. In this role, you will be responsible for designing and developing scalable data processing systems using Spark and Scala.Key Responsibilities:Data Processing: Design and implement efficient data processing pipelines using Spark and Scala.Big Data Expertise:...

  • Big Data Solutions Architect

    7 days ago

    Warszawa, Mazovia, Czech Republic Axiom software solutions Full time

    Company OverviewAxiom software solutions is a leading provider of innovative big data solutions.We are seeking an experienced Big Data Senior Developer to join our team and contribute to the design, development, and implementation of cutting-edge big data projects.Job DescriptionThe ideal candidate will have at least 2 years of experience working with Spark...

  • Senior Data Engineer @ Data Science UA

    1 day ago

    Remote, Lisbon, Frankfurtammain, Heidelberg, Warszawa, Czech Republic Data Science UA Full time

    - Bachelor degree in Computer Science, similar technical field of study or equivalent practical experience.- Commercial experience developing Spark Jobs using Scala and Java- Experience in data processing using traditional and distributed systems (Hadoop, Spark, AWS - S3) and designing data models and data warehouses.- Strong understanding and application of...

  • Big Data Developer with Spark and Agile Expertise

    4 days ago

    Warszawa, Mazovia, Czech Republic Axiom software solutions Full time

    Axiom software solutions is seeking a seasoned Big Data Senior Developer to join our team.We are looking for an expert in Big Data technologies, specifically in Spark and Scala. With at least 2 years of experience working with these tools, you will be responsible for designing, developing, and maintaining large-scale data processing systems.As a key member...

  • Data Engineer

    7 days ago

    Remote, Lisbon, Frankfurtammain, Heidelberg, Warszawa, Czech Republic Data Science UA Full time

    About Data Science UA:Data Science UA is a leading service company with a strong focus on data science and AI expertise.We are passionate about fostering the largest Data Science Community in Eastern Europe, and our journey began in 2016 with the organization of the first Data Science UA conference.Our Mission:We aim to deliver innovative solutions that...

  • Big Data Architect for Scalable Data Processing

    11 hours ago

    Warszawa, Mazovia, Czech Republic Quontex Ltd Full time

    Welcome to Quontex Ltd, a leading programmatic media company specializing in ingesting large volumes of data and offering a range of products and services across Media, Analytics, and Technology.Our team is expanding rapidly, with over 700 employees and 15 global offices spanning four continents. We are seeking an experienced Data Engineer to join our team...

  • Data Engineer @

    14 hours ago

    Warszawa, Mazovia, Czech Republic Quontex Ltd Full time

    3+ years of overall experience in Data Warehouse development and database designDeep understanding of distributed computing principlesExperience with AWS cloud platform, and big data platforms like EMR, Databricks, EC2, S3, RedshiftExperience with Scala, Spark, Hive, Yarn/Mesos, etc.Experience in SQL and NoSQL databases, as well as experience with data...

  • Backend Developer for Big Data Solutions

    7 days ago

    Warszawa, Mazovia, Czech Republic Axiom software solutions Full time

    Axiom Software Solutions is seeking a highly skilled Backend Spark developer to join our team. Our company specializes in delivering innovative software solutions, and as a key member of our team, you will be responsible for developing, testing, and deploying technical specifications.Job DescriptionAs a Backend Spark developer, your mission will be to...

  • AI Engineer

    1 day ago

    Warszawa, Czech Republic 7N Full time

    Profil Kandydata/-ki Wykształcenie wyższe na kierunku informatyka, fizyka, matematyka, lub pokrewnym (warunek konieczny) Min. 3 lata doświadczenia komercyjnego w zakresie tworzenia rozwiązań Data Science/Big Data Wiedza z zakresu tworzenia modeli uczenia maszynowego oraz ich ewaluacji Doświadczenie z dużymi modelami językowymi (LLM), embedingami...

  • Machine Learning Engineer/Data Engineer @ Lumicode Sp. z o.o.

    1 day ago

    Gdańsk, Łódź, Warszawa, Gdynia, Czech Republic Lumicode Sp. z o.o. Full time

    Wymagane doświaczenie: Ponad 5 lat doświadczenia w programowaniu w języku Python 4+ lat doświadczenia z technologiami bazodanowymi, najlepiej z obszarem Big Data (Hadoop, Spark, przetwarzanie potokowe danych) 3+ lat zaawansowanej znajomości SQL (T-Sql, PL/SQL, Spark SQL) Doświadczenie w projektach uczenia maszynowego i modelach nauki o danych...

  • Senior Data Scientist @ dotData

    1 day ago

    Warszawa, Czech Republic dotData Full time

    Minimum Qualifications: 3+ years professional experience building data science-driven solutions including data processing, feature engineering/selection, model training/tuning, and post-deployment validation Strong knowledge of data analytics and machine learning, and familiarity with common use cases across different industries Strong hands-on coding...

  • Data Engineer @ Devire

    1 day ago

    Warszawa, Czech Republic Devire Full time

    Developing, optimizing & maintaining big data (ELT/ETL) pipelines for business intelligence and statistical modeling purposes Ensuring & monitoring data quality and integrity Setting up and configuring data storage systems (e.g., SQL databases, data lakes) Managing and supporting production use of connections between key elements of data...

  • Big Data Architect @ Connectis_

    1 day ago

    Remote, Warszawa, Czech Republic Connectis_ Full time

    🔍 CZEGO OCZEKUJEMY OD CIEBIE? Minimum 5 lat komercyjnego doświadczenia na stanowisku Big Data Architect lub podobnym. Znajomość zagadnień związanych z projektowaniem i wdrażaniem złożonych systemów IT wraz z integracją. Doświadczenie w opracowywaniu modeli oraz analiz architektonicznych, tworzeniu architektury biznesowej. Znajomość branży...

  • Strong Middle Data Engineer

    22 hours ago

    Remote, Warszawa, Cracow, Wrocław, Czech Republic N-iX Full time

    4+ years of hands-on experience in data engineering.Strong expertise with Python, Pyspark and Azure Databricks.Solid experience in Cloud-based Big Data integration, specifically within the Azure ecosystem.Strong data engineering mindset with experience building and maintaining production-level pipelines.Solid knowledge of Azure, GitLab, and OOP...

  • Specjalista ds. Danych Big Data

    20 hours ago

    Warszawa, Mazovia, Czech Republic 7N Full time

    Opis stanowiskaJesteśmy poszukiwani przez naszego klienta, firmę z sektora bankowego, których oczekuje od nas współpracy w dalszym rozwoju autorskiego systemu do detekcji anomalii za pomocą narzędzi do budowy modeli uczenia maszynowego.Oczekujemy doświadczonego specjalisty ds. danych big data, kandydata na stanowisko Data Scientist / Big Data...

  • Scalable Data Engineer for ETL Processes

    4 days ago

    Warszawa, Mazovia, Czech Republic Axiom software solutions Full time

    At Axiom Software Solutions, we are looking for a skilled Backend Spark developer to join our team.About the JobWe are seeking a highly motivated and experienced professional to develop, test and deploy technical and functional specifications from our Solution Designers / Business Architects / Business Analysts, ensuring correct operability and compliance...

  • Senior Data Engineer @ TechHunt

    1 day ago

    Remote, Kraków, Warszawa, Kyiv, Czech Republic TechHunt Full time

    Qualifications: Strong proficiency in Scala and the ability to design and build robust backend services.  Experience with Python for data pipeline development, ETL processes, or data integration Solid understanding of database systems (SQL/NoSQL) and experience working with large datasets.  Experience with K8S, writing and maintaining Helm...

  • Specjalista ds. Inżynierii Inteligencji Maszynowej

    1 day ago

    Warszawa, Mazovia, Czech Republic 7N Full time

    Nasza firma, 7N, poszukuje doświadczonego specjalisty w dziedzinie inżynierii inteligencji maszynowej na stanowisko AI Engineer / Data Scientist. Praca będzie obejmowała rozwój i powiększanie obecnego zespołu naszego klienta - lidera innowacyjnych rozwiązań bankowych.O projekcieW ramach projektu będziemy pracować nad autorskim systemem do detekcji...

  • Data Engineer @ PKO Finat

    1 day ago

    Warszawa, Czech Republic PKO Finat Full time

    UMIEJĘTNOŚCI WYMAGANE DO SPEŁNIENIA:1. Co najmniej 5-letnie doświadczenie jako Developer ETL w obszarze procesów integracji danych na on-prem :- Ekspercka znajomość relacyjne bazy danych Oracle, PostgeSQL (projektowanie, budowa i tiuning baz danych)- Biegłe posługiwanie się narzędziami ETL na on-prem na poziomie zaawansowanym : Informatic Power...

  • Data Architect for Scalable Data Platform

    2 days ago

    Warszawa, Mazovia, Czech Republic Pragmile Full time

    At Pragmile, we are on a mission to maximize green energy production and reduce O&M costs through the use of artificial intelligence models.Solar Spy is an analytical platform dedicated to support solar farm operations. It processes various types of PV plant data, including image data from aerial inspections, weather data, data from photovoltaic installation...