Similar to other people, the first interview was with the Talent team, mostly motivational and very pleasant. Next stage was a take home test that allegedly takes 4h. I don't think it's a 4h. Similar to what other people have said it's a really lengthy one. Basically, you are asked to build a Q&A bot (agentic and whatnot) based on the CONVFINQA paper from 2023, but avoid using the same methodology. When you consider pydantic validation, API costs (which are not covered), expected to use OpenAI API, and the rest of a proper agentic solution, it adds up. For £135k offered (max), not bad, but agree that might not select good candidates, rather people with time.

Initial 45m talk with a senior engineer followed by a take home exercise. I would also like to echo a lot of the negative experience comments. The task per se was quite easy: an LLM-based conversational agent over some financial data. As instructed, it should take a few hours, solution should not be complicated, and also they stated that not all ideas need to be implemented. After all, they instructed to spend 25% of time on writing a report that included solution description, 'Future work' and Error analysis. Putting everything together to have a well structured code, run small scale eval to analyse results, do prompt improvements, re-run eval, and conclude with detailed report easily adds up to a few hours. However, the feedback seemed to imply that far more were expected than either necessary for this problem (LLMs+tools or Agentic style solution), or requested in the description or could be finished in a few hours. The feedback also implied using CoT which was again unnecessary for the complexity of the problem and which I also described in detail in the Future work section, but apparently it was ignored. Other negative aspects: - It is the first time I am asked to use LLMs for home exercise and they don't provide with a temp api_key. - In their orig. instructions email, they mentioned it would be possible that I send them one of my projects as opposed to doing the exercise. However, when I offered to opensource a project of mine that included RAG, LLMs, prompting, they declined. In retrospect it seems clear they want you to work on their problem.