Listed here are the rough steps you’ll follow to understand this project:

1. Download the video or podcast transcript and cargo into documents
2. Split long documents into chunks
3. Summarize the transcript with an LLM
4. Optional: Wrap all of it in a user-friendly command line interface and even an online application.

Project Idea 4: Information Extraction

One other useful use case of LLMs is . For instance, you may provide an LLM with a couple of examples that contain text and the data you wish it to extract.

Rember the cover letter generator from earlier? You would extend it with a component to extract the relevant information from a job posting directly:

prompt = """
This program will extract relevant information from a job posting. 
Listed here are some examples:
Job posting: 
Lead engineer for software integration (distant possible)
At XYZ Co. we're making the world a greater place.
To achieve this we're on the lookout for a lead engineer with experience in Python and JIRA.
Extracted Text:
Role: Lead engieer for software integration.
Company: XYZ Co.
Requirements: Python, JIRA
--
Job posting: 
Senior software engineer - Autonomous Mobility
ABC Inc. is an awesome company.
We're on the lookout for someone with great ability to put in writing complex C code.
Extracted Text:
"""

Listed here are the rough steps you’ll follow to understand this project:

1. Load job description from job posting right into a document
2. Extract the relevant information with the LLM by prompt engineering a prompt using examples

Project Idea 5: Web Scraper

LLMs are exceptional at (transforming) texts, similar to

• rewriting text in a selected style (e.g., the form of “The Economist” or “Latest Yorker”)
• rewriting text in a selected reading level (e.g., level grade 6 for easier readability)
• reformatting information from any format to some other format
• text correction (e.g., spelling and grammar)
• translations

It is vitally common to make use of LLMs to convert text from one form to a different.

A creative idea to use this rewriting capability is to make use of it for web scraping. If you’ve got ever written an online scraper, how tedious it’s. What in the event you could use LLMs to construct a more generic solution to extract data from unstructured web sites?

This is precisely what mangotree has done:

Listed here are the rough steps you’ll follow to understand this project:

1. Scrape the web site’s source code and cargo right into a document
2. Split long documents into chunks
3. Extract the relevant data from the source code using the LLM (see extraction)
4. Reformat the extracted data into the specified format with the LLM by prompt engineering a prompt using examples

The project ideas up to now were based on the concept of generating recent text. But one other use case of LLMs relies on the concept of text representations. You possibly can input the text to an embeddings model and extract the numerical representation of this text — the “text embeddings”.

These text embeddings enable you to perform mathematical operations, including similarity calculations, or apply Machine Learning algorithms.

On this section, we are going to discuss some project ideas based on use cases related to them:

1. Search and similarity: searchable database of your documents
2. Query answering: query answering over documents or code base
3. Clustering: clustering social media posts and podcast episodes into topics
4. Classification: classify business inquiries from e-mails

Project Idea 6: Searchable Database of Your Documents

Embeddings will help us seek for content based on . In contrast to keyword-based search engines like google and yahoo, we are able to calculate the similarity of a document’s embeddings to the embeddings of a search query.

For instance, you possibly can turn your personal documents right into a searchable database:

One other neat project is Andrej Karpathy’s weekend hack that allows you to seek for a selected movie:

Listed here are the rough steps you’ll follow to understand a project like these:

1. Load the files into documents
2. Split long documents into chunks
3. Generate and store the embeddings from the documents with an embeddings model
4. Define the index query to retrieve the relevant files

Project Idea 7: Query Answering over Documents

will be viewed as a mix of search (see search) and summarization (see summarization). It could help work through any document in a more intuitive way.

You need to use it to chat along with your documents or any code base:

Listed here are the rough steps you’ll follow to understand this project:

1. Load source code into documents
2. Split long documents into chunks
3. Generate and store the embeddings from the documents with an embeddings model
4. Define the index query to retrieve context and prompt the LLM on it

Project Idea 8: Clustering Documents into Topics

Other than querying documents or information from said documents, you can even use embeddings to place documents into categories through the use of (unsupervised learning).

For instance, you need to use clustering to search out topics in a podcast episode.

Or you may cluster posts on a web based forum into topics.