Data & AI Engineer
✨ AI Summary
As a Data & AI Engineer, you will enhance the context for large language models (LLMs) by developing AI platforms tailored for institutional clients in sectors such as private equity, agribusiness, and financial services. Your primary responsibility will be to transform raw data—financial metrics, operational data, and qualitative insights—into formats that LLMs can effectively utilize.
Main Responsibilities:
- Develop retrieval pipelines including ingestion, chunking strategies, embedding generation, indexing, and hybrid retrieval techniques.
- Create structured data layers using SQL schemas that enable precise access to quantitative data.
- Integrate various client data sources, ensuring data remains accurate and up-to-date without excessive human oversight.
- Design LLM tool interfaces to guide models toward producing accurate answers.
- Conduct rigorous evaluations of retrieval quality and LLM output, using metrics such as recall and precision.
- Oversee data quality through validation and deduplication processes.
- Select tools based on evidence to optimize embedding models and vector storage solutions.
- Contribute to system observability through performance tracking and usage analytics.
Requirements:
- Strong understanding of retrieval-augmented generation and experience building production-level retrieval systems.
- Proficiency in Python, SQL, and asynchronous I/O.
- Experience with embedding models and vector search technologies.
- Ability to quantitatively assess retrieval and LLM outputs.
- Familiarity with financial and analytical data where precision is critical.
