Senior Data Engineer

Open Innovation AI is a global technology company that specializes in developing advanced solutions for managing AI workloads. Its flagship product, the Open Innovation Cluster Manager (OICM), orchestrates complex AI tasks efficiently across diverse infrastructures. The platform is hardware-agnostic, optimized for various GPUs and accelerators hardware, and facilitates seamless integration and scalability for enterprise AI applications. Open Innovation AI focuses on optimizing and simplifying AI workload management and making AI technologies accessible to organizations of all sizes. With its innovative solutions, companies can reduce operational costs, accelerate time to value, and maximize their return on investment, ensuring that their AI strategies contribute directly to enhanced business outcomes.

Role Overview:

Open Innovation AI is seeking an experienced Senior Data Engineer to contribute to the development of our MLOps platform and to work on deploying Large Language Models (LLMs). This role involves both maintaining scalable data infrastructure and actively participating in the enhancement of our machine learning operations.

Role Requirements:

• Design, construct, install, test, and maintain scalable data management systems

• Actively contribute to the development and enhancement of our MLOps platform

• Work on deploying and managing Large Language Models, ensuring efficiency and scalability

• Develop data set processes for data modeling, mining, and production

• Integrate new data management technologies and software engineering tools into existing structures

• Collaborate with the engineering team and other stakeholders in automating the deployment of machine learning models

• Recommend improvements to data reliability, efficiency, and quality

• Research industry trends and apply this knowledge to data infrastructure projects

• Utilize data to identify tasks that can be automated and streamline processes

Required Qualifications

• Bachelor’s or master’s degree in computer science, Engineering, or a related field

• Hands-on experience with Apache Airflow and ClickHouse (database), along with proficiency in programming languages such as Python, Java, C++, or Scala

• Proven experience as a Data Engineer with a focus on MLOps platforms and big data technologies

• Experience in deploying and managing Large Language Models (LLMs)

• Strong proficiency in SQL and NoSQL databases, data warehousing solutions

• Skilled in data pipeline, data modeling, ETL and workflow management tools

• Knowledge of cloud services (AWS, Azure, GCP) and their application in data engineering

• Excellent problem-solving skills and ability to work in a collaborative environment

• Strong project management and organizational skills

Requirements added by the job poster

• Bachelor's Degree

• 3+ years of work experience with ClickHouse

• 4+ years of work experience with Apache Airflow

• 4+ years of work experience with Extract, Transform, Load (ETL)