Big Data Engineer for Hadoop Ecosystem
3 days ago
Capco Poland is a leading global technology and management consultancy, dedicated to driving digital transformation across the financial services industry.
We are seeking a highly skilled and motivated Big Data Administrator/Engineer with 5+ years of experience in Hadoop administration and expertise in automation tools like Ansible, shell scripting, or Python scripting.
The ideal candidate will have strong DevOps skills and proficiency in coding, particularly in Python. This is a dynamic role focused on managing and engineering Big Data solutions across multiple open-source platforms such as Hadoop, Kafka, HBase, and Spark.
Main Responsibilities:- Big Data Administration:
- Administer and manage Hadoop clusters, ensuring high availability, performance, and scalability.
- Maintain and troubleshoot Hadoop ecosystem components such as HDFS, MapReduce, YARN, and related tools.
- Ensure Kafka, HBase, and Spark systems are optimized and running smoothly.
- Implement monitoring and alerting for Big Data infrastructure.
- Develop and automate scripts using tools such as Ansible, Shell scripting, or Python to streamline Big Data administration tasks.
- Create reusable automation frameworks to reduce manual efforts and improve operational efficiency.
- Work on CI/CD pipelines for deployment automation and system integration.
- Apply DevOps principles to the Big Data environment, focusing on continuous integration and continuous delivery (CI/CD).
- Build and manage automated deployment processes for Big Data clusters and services.
- Collaborate with development teams to integrate automation in Big Data workflows.
- Identify, troubleshoot, and resolve issues related to Big Data platforms, including system performance, resource utilization, and service failures.
- Work with logs, monitoring tools, and other debugging techniques to diagnose and resolve complex issues.
- Work closely with other teams to support data pipelines, data quality checks, and performance optimizations.
- Provide ongoing technical support to ensure that Big Data systems are stable, secure, and aligned with business objectives.
Requirements:
- Big data
- DevOps
- Hadoop
- HDFS
- Yarn
- Kafka
- HBase
- Spark
- Performance tuning
- Ansible
- Shell
- Python
- Terraform
- Docker
- Kubernetes
- Degree
- Cloud platform
- Azure
- GCP
- Apache
- NiFi
- Airflow
Benefits:
- Employment contract and/or Business to Business - whichever you prefer
- Possibility to work remotely
- Speaking English on daily basis, mainly in contact with foreign stakeholders and peers
- Multiple employee benefits packages (MyBenefit Cafeteria, private medical care, life-insurance)
- Access to 3.000+ Business Courses Platform (Udemy)
- Access to required IT equipment
- Paid Referral Program
- Participation in charity events e.g. Szlachetna Paczka
- Ongoing learning opportunities to help you acquire new skills or deepen existing expertise
- Being part of the core squad focused on the growth of the Polish business unit
- A flat, non-hierarchical structure that will enable you to work with senior partners and directly with clients
- A work culture focused on innovation and creating lasting value for our clients and employees
-
Hadoop Big Data Administrator/Engineer
4 days ago
Warszawa, Mazovia, Czech Republic Capco Poland Full time5+ years of hands-on experience in Big Data administration, automation, and DevOps practices, including Hadoop ecosystem tools such as HDFS, YARN, and MapReduce.Proficient in managing and troubleshooting Kafka, HBase, and Spark.Experience with performance tuning, cluster optimization, and high-availability configurations.Strong experience with automation...
-
Specjalista Danych Big Data
5 days ago
Warszawa, Mazovia, Czech Republic 7N Full timePraca dla Specjalisty Danych Big DataTwój zwiększony potencjał czeka w 7N, firmy doradztwa IT, która buduje najbardziej zaawansowane rozwiązania big data. Pracując u nas, zdobędziesz doświadczenie przy stworzeniu autorskich systemów detekcji anomalii i będziesz miał okazję wykorzystać narzędzia do budowy zaawansowanych modeli uczenia...
-
Senior Data Scientist @
1 week ago
Warszawa, Mazovia, Czech Republic dotData Full timeMinimum Qualifications:3+ years professional experience building data science-driven solutions including data processing, feature engineering/selection, model training/tuning, and post-deployment validationStrong knowledge of data analytics and machine learning, and familiarity with common use cases across different industriesStrong hands-on coding skills...
-
Warszawa, Mazovia, Czech Republic Pragmile Full timeWe are looking for a skilled Distributed Data Engineer to join our team at Pragmile. Our company is dedicated to supporting solar farm operations through an analytical platform that processes various types of PV plant data.The ideal candidate will have experience with designing and implementing scalable data architectures, capable of processing large amounts...
-
Senior Data Software Engineer @
5 days ago
Warszawa, Mazovia, Czech Republic Global Payments Full timeMinimum QualificationsBS in Computer Science, Information Technology, Business / Management Information Systems or related fieldExperience in lieu of degree will be consideredTypically minimum of 4 years of strong development background in ETL tools like GCP-Data Flow , PySpark , SSISExperience in Google cloud platform - GCP Pub/Sub, Datastore, BigQuery,...
-
Data Architect @
2 days ago
Warszawa, Mazovia, Czech Republic ITDS Full timeYou're ideal for this role if you have:Minimum 5 years of experience working with data architectureExpertise in SQL Server, including modeling, indexing, and performance tuningHands-on experience with ETL processes and tools like Informatica Power CenterKnowledge of cloud storage platforms and their integrationFamiliarity with big data technologies such as...
-
Warszawa, Mazovia, Czech Republic Global Payments Full timeAs a Senior Software Data Engineer at Global Payments, you will play a crucial role in designing, implementing, and testing our data pipelines. Your expertise in dimensional modeling and data engineering best practices will be instrumental in architecting data models, data warehouse/datalake architecture, and optimizing data pipelines for high-performance...
-
Cloud Data Architect
6 days ago
Warszawa, Mazovia, Czech Republic T-Mobile Polska Full timeExtensive experience (4+ years) in designing and delivering cloud or hybrid data platformsStrong background in data engineering and analyticsProficiency in cloud technologies (AWS, GCP)Experience with Data Warehousing, Big Data technologies and NoSQL databasesStrong leadership and communication skillsAbility to translate complex technical concepts for...
-
Senior Cloud Data Solutions Engineer
6 days ago
Warszawa, Mazovia, Czech Republic T-Mobile Polska Full timeJob Description:T-Mobile Polska is seeking a highly skilled Senior Cloud Data Solutions Engineer to join our team. As a key member of our data platform infrastructure, you will be responsible for designing, developing, and managing scalable, robust, and secure data solutions.Key Responsibilities:Design and Develop Data Platforms: Lead the design and...
-
Senior Data Engineer @
5 days ago
Warszawa, Mazovia, Czech Republic Asana Full timeAbout youDegree in Computer Science, Engineering or equivalent technical field experience6+ years of hands-on experience in Data Engineering or Software Engineering.Fluent in SQL and proficient in at least one programming language (e.g., Python, Java, Scala etc.)Strong expertise in Databricks, AWS S3, Spark, Snowflake, and AirflowKnowledge of system and...
-
Data Engineer for Scalable Data Solutions
2 days ago
Warszawa, Mazovia, Czech Republic Asana Full timeAbout AsanaWe are a rapidly growing technology company that helps teams orchestrate their work, from small projects to strategic initiatives.Millions of teams around the world rely on Asana to achieve their most important goals, faster.Our Enterprise Data & Intelligence (EDI) team is tasked with building powerful decision-making data products, integrations,...
-
Tech Lead Data Engineer @
6 days ago
Warszawa, Mazovia, Czech Republic SambaTV Full time5+ years of experience in Data Engineering, Software Engineering, or a related field.Must-have expertise in Apache Airflow, Databricks, and PySpark for orchestration, scalable data processing, and transformation.Expertise in Python or strong proficiency in another modern programming language with a willingness to master Python.Deep understanding of...
-
Senior Python Engineer @
7 days ago
Warszawa, Mazovia, Czech Republic Pragmile Full timeWhat we expect: Experience of working with big data, data engineering tools for data ingestion, transformations, querying Strong understanding of data structures and algorithms Strong understanding of object design and integration patterns Proven experience of using python for building data processing platforms/applications Experience in building distributed...
-
Warszawa, Mazovia, Czech Republic Asana Full timeAbout youExpertise in programming, distributed systems design, and infrastructureExperience building and operating scalable, reliable, and highly-available services4+ years designing and implementing production code for backend, infrastructure, and/or data systems2+ years mentoring/coaching other team members on design and execution of projectsEagerness to...
-
Senior Data Engineer and Architect
5 days ago
Warszawa, Mazovia, Czech Republic Asana Full timeAbout YouWe are seeking a highly skilled Senior Data Engineer to join our team. This individual will be responsible for designing, building, and maintaining scalable data pipelines using modern cloud-based architectures.The ideal candidate will have 6+ years of hands-on experience in data engineering or software engineering, with a strong expertise in...
-
Data Science Automation Specialist
7 days ago
Warszawa, Mazovia, Czech Republic dotData Full timeCompany Overview:dotData is a leader in automating data science solutions for businesses. Our team is passionate about empowering organizations to unlock the full potential of their data.Job Description:We are seeking an experienced Data Scientist to join our customer-facing team. As a trusted advisor, you will work closely with clients to understand their...
-
Data Engineering Team Lead Position
5 days ago
Warszawa, Mazovia, Czech Republic SambaTV Full timeSamba TV is a leading provider of data and technology solutions for the media industry.We are seeking an experienced Tech Lead - Data Engineering to join our team. As a key member of our data engineering team, you will be responsible for designing and implementing scalable, high-performance data pipelines that power Samba TV's analytics and insights.The...
-
Senior Data Scientist @
6 days ago
Warszawa, Mazovia, Czech Republic SambaTV Full timeBachelor's or Master's degree in a quantitative field (e.g., Computer Science, Statistics, Mathematics, Data Science, or similar).5+ years of experience in data analysis, data processing, or related fields.Proficiency in at least one programming language (e.g., Python, R, or SQL) with a strong foundation in analytical techniques.Practical experience with...
-
Machine Learning Software Engineer @
2 hours ago
Warszawa, Mazovia, Czech Republic Digital Turbine Full time2+ years of experience in software engineeringBachelor Degree in Computer Science, Mathematics, or related fieldProficiency in any JVM-based (Java, Kotlin, Scala)Self motivation to achieve something remarkableProficient communication in English is a mustNice to have skills:Experience with Apache Kafka, knowledge of stream processing framework like Apache...
-
Database Reliability Engineer @ .com
5 days ago
Warszawa, Mazovia, Czech Republic monday Full timeProven working experience and deep understanding of RDBMS (MySQL / PostgreSQL/ Oracle / SQL Server / etc).Experience of at least 5+ years with performance optimization and scaling out data worldwide.Coding experience and understanding of code and software environment.Experience and understanding of cloud environments and toolsExperience in leading projects,...