You studied economics, then learned to code and moved through product, growth, and now hands-on AI constructing. What perspective does that generalist path provide you with that specialists sometimes miss?
I’m undecided.
People see generalists as having shallow knowledge, but generalists may also dig deep.
I see generalists as individuals with multiple interests and a drive to know the entire, not only one part. As a generalist you have a look at the tech, the shopper, the info, the market, the associated fee of the architecture, and so forth. It gives you an edge to maneuver across topics and still do good work.
I’m not saying specialists can’t do that, but generalists are likely to adapt faster because they’re used to picking things up quickly.
You’ve been writing rather a lot about agentic systems currently. When do “agents” actually outperform simpler LLM + RAG patterns, and when are we overcomplicating things?
It depends upon the use case, but typically we throw AI into a number of things that probably don’t need it. Should you can control the system programmatically, you need to. LLMs are great for translating human language into something a pc can understand, but additionally they introduce unpredictability.
As for RAG, adding an agent means adding costs, so doing it only for the sake of getting an agent isn’t an ideal idea. You possibly can work around it through the use of smaller models as routers (but this adds work). I’ve added an agent to a RAG system once because I knew there can be questions on constructing it out to also “act.” So again, it depends upon the use case.
Once you say Agentic AI needs “evaluations” what’s your list of go-to metrics? And the way do you choose which one to make use of?
I wouldn’t say you usually need evals, but corporations will ask for them, so it’s good to know what teams measure for product quality. If a product will probably be utilized by a number of people, be sure that you’ve some in place. I did quite a number of research here to know the frameworks and metrics which have been defined.
Generic metrics are probably not enough though. You wish a couple of custom ones in your use case. So the evals differ by application.
For a coding copilot, you may track what percent of completions a developer accepts (acceptance rate) and whether the total chat reached the goal (completeness).
For commerce agents, you would possibly measure whether the agent picked the best products and whether answers are grounded in the shop’s data.
Safety and security related metrics are vital too, akin to bias, toxicity, and the way easy it’s to interrupt the system (jailbreaks, data leaks).
For RAG, see my article where I break down the standard metrics. Personally, I actually have only arrange metrics for RAG to this point.
It may very well be interesting to map how different AI apps arrange evals in an article. For instance, Shopify Sidekick for commerce agents and other tools akin to legal research assistants.
In your Agentic RAG Applications article, you built a Slack agent that takes company knowledge under consideration (with LlamaIndex and Modal). What design alternative ended up mattering greater than expected?
The retrieval part is where you’ll get stuck, specifically chunking. Once you work with RAG applications, you split the method into two. The primary part is about fetching the right information, and getting it right is essential because you’ll be able to’t overload an agent with an excessive amount of irrelevant information. To make it precise the chunks must be quite small and relevant to the search query.
Nonetheless, for those who make the chunks too small, you risk giving the LLM too little context. With chunks which might be too large, the search system may grow to be imprecise.
I arrange a system that chunked based on the kind of document, but immediately I actually have an idea for using context expansion after retrieval.
One other design alternative you could be mindful is that although retrieval often advantages from hybrid search, it will not be enough. Semantic search can connect things that answer the query without using the precise wording, whereas sparse methods can discover exact keywords. But sparse methods like BM25 are token-based by default, so plain BM25 won’t match substrings.
So, for those who also want to look for substrings (a part of product IDs, that form of thing), you could add a search layer that supports partial matches as well.
There may be more, but I risk this becoming a whole article if I keep going.
Across your consulting projects over the past two years, what problems have come up most frequently in your clients, and the way do you address them?
The problems I see are that almost all corporations are searching for something custom, which is great for consultants, but constructing in-house is riddled with complexities, especially for individuals who haven’t done it before. I saw that 95% number from the MIT study about projects failing, and I’m not surprised. I believe consultants should get good at certain use cases where they’ll quickly implement and tweak the product for clients, having already learnt do it. But we’ll see what happens.
You’ve written on TDS about so many various topics. Where do your article ideas come from? Client work, tools you desire to try, or your individual experiments? And what topic or problem is top of mind for you immediately?
A little bit of every thing, frankly. The articles also help me ground my very own knowledge, filling in missing pieces I could not have researched myself yet. Right away I’m researching a bit on how smaller models (mid-sized, around 3B–7B) might be utilized in agent systems, security, and specifically improve RAG.
Zooming out: what’s one non-obvious capability teams should cultivate in the subsequent 12–18 months (technical or cultural) to grow to be genuinely AI-productive somewhat than simply AI-busy?
Probably learn to construct within the space (especially for business people): just getting an LLM to do something consistently is a option to understand how unpredictable LLMs are. It makes you a bit more humble.
To learn more about Ida‘s work and stay up-to-date together with her latest articles, you’ll be able to follow her on TDS or LinkedIn.
