Automate Pinecone Day by day Upsert Task with Celery and Slack monitoring

It’s been some time since my last LLM post and I’m excited to share that my prototype has been successfully productionized as Outside’s first LLM-powered chatbot, Scout. In case you are an Outside+ member, you may test it out over at https://scout.outsideonline.com/.

This journey began as my weekend curiosity project back in March 2023. I had the concept to construct a Q&A chatbot using OpenAI’s LLMs and Outside’s content as a knowledge base. Later I shared my prototype at our internal product demo day and I used to be thrilled by the interest it managed to spark. Scout quickly became an official project. On November twenty eighth 2023, we launched Scout to limited Outside+ members. Fast forward to today, April twelfth, 2024, over 28.3k unique users have already utilized this Outdoor Companion AI tool.

I couldn’t be more grateful for this moonstruck experience and I’ve been planning to write down a mini-series to share some behind-the-scenes insights into what it takes to bring LLM & RAG powered apps to life. To this point I’ve planned to cover the next three parts:

• 🦦 Part 1: Automate Pinecone Day by day Upserts with Celery and Slack monitoring
• 🦦 Part 2: Constructing an LLM Websocket API in Django with Postman Testing
• 🦦 Part 3: Monitoring LLM Apps with Datadog: synthetic tests, OpenAI, and Pinecone usage tracking

This post will dive into Part 1, organising scheduled tasks with Celery Beat to mechanically upsert embeddings into the Pinecone vector database. And we’ll arrange slack updates for straightforward monitoring. Let’s start!

LLMs typically have training data cut off date, the present gpt-4-turbo was cut off at 2023-Dec (to my writing day -2024-April). The promise of using RAG is that we are able to equip LLMs with more fresh and domain specific data to cut back hallucinations and improve user experience. Thus the query: how can we keep the knowledge base fresh and up to this point? The reply is — using Celery and Celery Beat to schedule a periodical task (day by day or weekly) to embed newly published…