Data Engineer for Large Scale Data Processing

11 hours ago


Wrocław, Województwo dolnośląskie, Czech Republic Shelf Full time
About the Role

We are seeking a skilled Data Engineer to join our team and contribute to building robust backend services for large-scale data processing. As a key member of our engineering team, you will be responsible for designing, implementing, and optimizing our distributed ETL pipeline, focusing on background processing logic, data transformation, and scalability.

Our ideal candidate has experience with Python (and Node.js) to create data pipelines and handle data from diverse storage solutions. You will work closely with our Data Scientists to implement ML model integrations within the data pipeline and develop clean, maintainable code in Python, adhering to best practices in observability, cost-efficiency, and robust error handling.

This is an exciting opportunity to build truly robust and accurate systems, working with cutting-edge technologies and contributing to the development of our innovative product. If you enjoy crafting efficient, testable code and want to be part of the engine behind advanced data processing, we encourage you to apply.

About Us

At Shelf, we empower humanity with better answers everywhere. We provide the core infrastructure that enables GenAI to be deployed at scale, helping companies deliver more accurate GenAI answers by eliminating bad data in documents and files before they go into an LLM and create bad answers.

We have high velocity growth powered by our innovative product, 3X growth for 3 years in a row. We now have over 100 employees in multiple U.S. states and European countries, and we have ambitious hiring goals over the next few months.

What You Will Do
  • Design, implement, and optimize our distributed ETL pipeline, focusing on background processing logic, data transformation, and scalability.
  • Develop modular and composable components capable of efficiently processing large-scale data across a diverse range of storage solutions, including S3, RDS/PostgreSQL, Elasticsearch, DynamoDB, data warehouses, and data lakes.
  • Implement ML model integrations within the data pipeline, working closely with Data Scientists on model deployment and data flow.
  • Develop clean, maintainable code in Python, adhering to best practices in observability, cost-efficiency, and robust error handling.
  • Proactively identify and address performance bottlenecks and inefficiencies in current systems, proposing solutions to improve scalability and reliability, while ensuring continuous production stability through thorough testing and monitoring practices.
  • Share your knowledge, participate in code reviews, and advocate for best practices to advance our backend development standards.
Requirements and Qualifications
  • Python, SQL, NoSQL, CQRS, AWS, ETL, NLP, Node.js
  • Stock options, GitHub Copilot subscription, LLM credits


  • Wrocław, Województwo dolnośląskie, Czech Republic Brightech Full time

    Job Title: Data Integration Expert with API and ETL KnowledgeAbout Us:Brightech is a leading American IT company offering innovative solutions for data exchange between different systems. We are committed to providing our clients with the highest level of business cooperation and positive atmosphere.Job Description:We are seeking an experienced Data...


  • Wrocław, Województwo dolnośląskie, Czech Republic CloudBusiness Sp. Zo. o. Full time

    About CloudBusiness Sp. Zo. o.We are seeking a skilled Back-end Java engineer to help us scale our next-gen consumer finance platform. Our product has tremendous traction, assisting people to manage their finances better, smarter, and faster.Key Responsibilities:Implement well-designed, testable, and reliable REST APIs to support our growing user...


  • Wrocław, Województwo dolnośląskie, Czech Republic WIPRO IT SERVICES POLAND Sp. z o.o. Full time

    At Wipro IT Services Poland Sp. z o.o., we're seeking a seasoned Senior Software Quality Assurance Engineer to join our team and drive quality excellence in complex workflows.We're a global organization with 900+ employees in Poland supporting over 45 clients, leveraging our holistic portfolio of capabilities in consulting, design, engineering, operations,...


  • Wrocław, Województwo dolnośląskie, Czech Republic FIS Techlology Services (Poland) Sp. z o.o. Full time

    Company OverviewFIS Technology Services (Poland) Sp. z o.o. is a leading global provider of financial technology solutions, empowering companies to innovate and excel in the rapidly changing financial landscape.


  • Wrocław, Województwo dolnośląskie, Czech Republic Scalo Full time

    Scalo szuka doświadczonego fachowca, który sprawi, że nasze platformy będą jeszcze bardziej elastyczne i dostępne. Jeśli masz pasję do pracy z oprogramowaniem Adobe AEM, jesteśmy dla Ciebie idealną firmą.Zakres obowiązkówTworzenie platformy dla dostawców, która umożliwi centralizację wymiany danych oraz współpracę w jednym...


  • Wrocław, Województwo dolnośląskie, Czech Republic RST Software Full time

    We are seeking a skilled Backend Developer to join our team at RST Software. As a key member of our development team, you will be responsible for designing and implementing the backend infrastructure of our early childhood education management application.Key ResponsibilitiesBackend Development: Develop scalable web applications using Next.js, focusing on...


  • Kraków, Wrocław, Czech Republic Unit8 SA Full time

    About You MSc level in the field of Computer Science, Machine Learning, Applied Statistics, Mathematics, or equivalent work experience. You are a proficient software engineer who knows the fundamentals of computer science and has experience in applying a blend of software engineering, machine learning, and statistical methods to solve real-world...


  • Kraków, Wrocław, Czech Republic Unit8 SA Full time

    As a member of agile project teams, your mission will be to build solutions and infrastructure aiming at solving the business problems of our clients. You are a proficient software engineer who knows the fundamentals of computer science and you master at least one widely adopted programming language (Python, Java, C#, C++). You know how to write...


  • Remote, Wrocław, Warszawa, Kraków, Czech Republic Holisticon Connect Full time

    We are seeking a highly skilled Senior Bioinformatics Data Engineer to support the operation and engineering needs of cBioPortal.Key Responsibilities:Develop, execute, and maintain ETL pipelines for extracting, transforming, and loading data for use in cBio and other bioinformatics analysis and visualizations.Ensure the reliability, scalability, and...


  • Remote, Wrocław, Warszawa, Kraków, Czech Republic Holisticon Connect Full time

    Holisticon Connect is a division within NEXER GROUP, a custom software development company. We started in Poland and are now a team of over 140 people with offices in Wrocław, Warsaw, and Cracow.Job DescriptionWe are looking for a Senior Bioinformatics Data Engineer to support the operation and engineering needs of cBioPortal. In this role, you will impact...


  • Remote, Wrocław, Czech Republic AVENGA Full time

    At Avenga, we are seeking a talented Data Expert to join our team and contribute to the development of modern cloud data solutions. As a key member of our data science team, you will have the opportunity to work on a wide range of projects and collaborate with experienced professionals from various business domains.About the RoleThis is an exciting...


  • Remote, Wrocław, Czech Republic Comscore (via CC) Full time

    Comscore (via CC) is a global leader in media analytics, revolutionizing insights into consumer behavior, media consumption, and digital engagement.We are looking for a highly skilled Java/Linux Engineer to join our team in Poland and around the world. As a key member of our technical staff, you will be responsible for designing, implementing, and...

  • Data Architect

    7 days ago


    Kraków, Wrocław, Czech Republic Unit8 SA Full time

    About UsFounded in 2017, Unit8 SA is a fast-growing Swiss AI and data analytics consulting and services company dedicated to solving complex problems of traditional industries.We work with some of the biggest organisations in Europe to solve the challenges that directly affect their business – be it operations, finance, manufacturing or R&D. Since our...


  • Remote, Kraków, Wrocław, Warsaw, Czech Republic N-iX Full time

    Programming: Minimum of 3-4 years as data engineer, or in a relevant field Python Proficiency: Advanced experience in Python, particularly in delivering production-grade data pipelines and troubleshooting code-based bugs. Data Skills: Structured approach to data insights Cloud: Familiarity with cloud platforms (preferably Azure) Data Platforms: Experience...


  • Remote, Wrocław, Czech Republic AVENGA Full time

    **About Us**We are Avenga, a global technology and business services company committed to helping our clients succeed in a world of continuous change.A Career as a Cloud Migration Data Engineer with AvengaAre you looking for a challenging role that involves migrating on-premises solutions to the cloud in Azure? As a Cloud Migration Data Engineer at Avenga,...


  • Remote, Kraków, Wrocław, Warsaw, Czech Republic N-iX Full time

    We are looking for a skilled Senior Data Engineer to join our team at N-iX. As a Senior Data Engineer, you will be responsible for designing and building data pipelines using Python.Our client is a Scandinavian-based company established with the mission to fundamentally transform the execution of capital projects and operations. The company's platform...


  • Remote, Gdynia, Wrocław, Gdańsk, Trójmiasto, Czech Republic N-iX Full time

    About the Project:N-iX is merging with a similar company in Canada, creating a leading online car market in North America.As a Senior Data Engineer, you will play a pivotal role in shaping the future of online car markets by enhancing the user experience for millions of car buyers and sellers.Main Responsibilities:Collaborate with a team of Senior Engineers,...


  • Kraków, Wrocław, Czech Republic Unit8 SA Full time

    About YouWe are seeking a highly skilled AI and Data Science Consultant to join our team at Unit8 SA. As a consultant, you will be responsible for working with our clients to understand their complex problems and designing solutions using cutting-edge technologies.Requirements:MSc level in Computer Science, Machine Learning, Applied Statistics, or...


  • Remote, Białystok, Wrocław, Kraków, Czech Republic Grape Up Full time

    Company OverviewGrape Up is a technology-driven company that leverages innovation to drive software advancements.We partner with leading providers in the automotive industry to build cutting-edge Data & Analytics platforms, serving as central hubs for R&D activities.

  • Cloud Data Engineer

    4 days ago


    Remote, Wrocław, Gdańsk, Rzeszów, Czech Republic Xebia sp. z o.o. Full time

    About Xebia sp. z o.o.Xebia is a leading digital solutions provider, renowned for its expertise in cloud-based technologies and data engineering. With a strong partnership with Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP), we've established ourselves as authorities in our field. Our dedication to innovation and technological...


  • Kraków, Wrocław, Czech Republic Unit8 SA Full time

    About Unit8Unit8 is a Swiss AI and data analytics consulting and services company dedicated to solving complex problems of traditional industries. We work with some of the biggest organisations in Europe to solve the challenges that directly affect their business.Job DescriptionWe are seeking an experienced Cloud Engineer to join our team in Krakow or...


  • Remote, Wrocław, Czech Republic KUBO Full time

    Transforming Raw Logs into Actionable InsightsWe are seeking a highly skilled Information Security Data Analyst Specialist to join our team at KUBO.About the RoleThe ideal candidate will possess strong data-handling skills, with the ability to extract insights and solve problems effectively. Proficiency in KQL, Regex, and Grok for data transformation and...


  • Remote, Wrocław, Warsaw, Cracow, Czech Republic Holisticon Connect Full time

    Company Overview">Holisticon Connect is a division within NEXER GROUP - a custom software development company.We are a team of over 140 people with offices in Wrocław, Warsaw, and Cracow.Job DescriptionWe are looking for a talented Data Warehouse Engineer to join our team working with a renowned global organization.This is a fantastic opportunity to be part...


  • Remote, Wrocław, Czech Republic AVENGA Full time

    About the RoleWe are seeking an experienced Cloud Data Solutions Architect to join our VCE Sales & Finance Analytics team. As a key member of our team, you will play a crucial role in designing and implementing data solutions that meet the evolving needs of our customers worldwide.


  • Remote, Warsaw, Wrocław, Białystok, Kraków, Gdańsk, Czech Republic Addepto Full time

    About AddeptoAddepto is a leading consulting and technology company specializing in AI and Big Data, helping clients deliver innovative data projects. We partner with top-tier global enterprises and pioneering startups.


  • Remote, Warszawa, Wrocław, Kraków, Czech Republic Bank of Montreal, przez Vistulo Sp. z o.o. Full time

    Job Description:We are seeking a highly skilled Java software engineer to join our team at Bank of Montreal, through Vistulo Sp. z o.o., and work on building and maintaining our low-latency trading system.About the Role:Design and implement robust software solutions using core Java (17 and 21) for the bank's trading systems.Understand, develop, and improve...