Data Engineer with Spark @ Comscore

The candidate must have: Solid understanding of Spark basics, building blocks, and mechanics (the deeper the knowledge, the higher the value) Strong knowledge of Python, Java, or Scala (with the ability to expand to expert level) 1+ years of experience with Spark (commercial not required, deep understanding matters more than years) Good SQL skills - not necessarily writing complex queries by hand, but strong knowledge of available tools and approaches to solve data problems Understanding of data quality issues in large datasets (inconsistencies, missing data, imbalanced sets, etc.) 1+ years of experience with Linux (power-user skills are a big plus; deployment is not required, but Linux knowledge makes your life easier) Professional working proficiency in English (oral and written) Understanding of HTTP API communication patterns (HTTP/REST/RPC) and protocol itself Good software debugging skills (beyond print - using debuggers effectively) Deep understanging of at least one technical area (be ready to share your biggest "battle story" about it) Solid Git understanding Strong communication skills (ability to drive end-to-end projects and mentor team members) If you don't have all the qualifications, but you're interested in what we do and you have a solid Linux understanding -> let's talk Correct Context is looking for a Data Engineer with Spark for Comscore in Poland and around. Comscore is a global leader in media analytics, revolutionizing insights into consumer behavior, media consumption, and digital engagement. Comscore leads in measuring and analyzing audiences across diverse digital platforms. Thrive on using cutting-edge technology, play a vital role as a trusted partner delivering accurate data to global businesses, and collaborate with industry leaders like Facebook, Disney, and Amazon. Contribute to empowering businesses in the digital era across media, advertising, e-commerce, and technology sectors. We have multiple Java + Spark, Scala + Spark, Python + Spark teams and we may try to match you to multiple teams or just find you single best fit depends on your skills and experience. We offer: Real big data projects (PB scale) 🚀 An international team (US, PL, IE, CL) 🌎 A small, independent team working environment 🧑‍💻 High influence on working environment Hands on environment Flexible work time ⏰ Fully remote or in-office work in Wroclaw, Poland 🏢 12,000 - 22,000 PLN net/month B2B 💰 Private healthcare (PL) 🏥 Multikafeteria (PL) 🍽️ Free parking (PL)🚗 The recruitment process for the Data Engineer position has following steps: Technical survey - 10min Technical screening - 30 min video call Technical interview - 60-90min video call - this step can be multiplied if we speak to multiple teams (we have multiple teams that you may want to join, your choice) Final Interview - Technical/Managerial - 30 min video call ,[ Design, implement, and maintain petabyte-scale big data pipelines using Spark (Java, Python, or Scala - depending on the team), Apache Airflow, Kubernetes, and other technologies , Optimize performance - working with big data is highly specific: sometimes IO-bound, sometimes CPU-bound. You’ll help figure out the most efficient approaches , Collaborate closely with other big data teams , Work with technologies such as AWS, Kubernetes, Airflow, EMR, Hadoop, Linux/Ubuntu, Kafka, and Spark  ] Requirements: Spark, Java, Scala, Python, Big data, AWS, API Tools: Jira, Confluence, Bitbucket, GIT, Jenkins, Agile, Kanban. Additionally: Remote work, Flexible working hours, Sport subscription, Flat structure, Small teams, International projects, Free parking, Free coffee, Playroom, Modern office, Startup atmosphere, No dress code.