Module 06: API & RAG Projects

← Previous | 🏠 Home | Next →

The Power of Programmatic AI

Until now, we have interacted with AI through a chat interface built by someone else (like the ChatGPT web app). However, the true power of AI unlocks when you build your own applications using AI Application Programming Interfaces (APIs).

By using APIs, you can programmatically send text, images, or audio to an AI model and receive structured data back. This allows you to integrate AI into existing software, build autonomous agents, or create specialized internal tools for your company.

Deep Dive: AI Providers and APIs

Choosing the right API provider depends on your specific use case, budget, and privacy requirements.

Provider / Model	Strengths	Weaknesses	Best Used For
OpenAI (GPT-4o)	Industry standard, highly reliable, excellent tool-calling (function calling), multimodal vision.	Strict content filtering, high cost for heavy volume.	General-purpose chatbots, complex reasoning, structured JSON output.
OpenAI (GPT-4o-mini)	Extremely fast, very cheap, solid reasoning.	Struggles with highly complex logic compared to 4o.	High-volume tasks, simple classification, sentiment analysis.
Anthropic (Claude 3.5 Sonnet)	Unmatched coding ability, large 200k context window, nuanced human-like writing tone.	API rate limits can be strict for new tiers.	Code generation, analyzing large documents, creative writing.
Google (Gemini 1.5 Pro)	Massive context window (up to 2 Million tokens). Can process entire books or hours of video at once.	Sometimes hallucinates slightly more than GPT-4 on logic puzzles.	Document synthesis, RAG over massive datasets, video analysis.
Open-source (Ollama/Llama-3)	100% free, runs locally, total data privacy.	Requires powerful hardware (GPUs) to run fast.	Healthcare, finance, air-gapped environments.

Essential API Concepts for Beginners

Before writing code, you need to understand the terminology:

1. API Key: A long string of characters (e.g., sk-proj-12345...) that acts as your username and password combined. CRITICAL SECURITY RULE: Never upload your API key to GitHub. Hackers scan GitHub 24/7. If they find your key, they will run up a $50,000 bill in hours. Always use .env files to store keys locally.
2. Endpoint: The specific URL you send data to. For OpenAI chat, it’s https://api.openai.com/v1/chat/completions.
3. JSON (JavaScript Object Notation): The universal language of APIs. It’s a structured way to send data using key-value pairs (e.g., {"role": "user", "content": "Hello"}).
4. Tokens: AI doesn’t read words; it reads tokens. A token is about 3/4 of an English word (e.g., “Hamburger” = “Ham”, “bur”, “ger” = 3 tokens). You are billed per 1,000 or 1,000,000 tokens. Output tokens (generating text) cost more than input tokens (reading text).
5. Temperature: A parameter you send in your request (ranging from 0.0 to 2.0).
   • Temperature = 0.0: Completely deterministic, analytical, factual. Best for coding or data extraction.
   • Temperature = 0.8: Creative, varied, slightly unpredictable. Best for brainstorming or creative writing.

Tutorial: Making API Calls with Python

Let’s look at practical code examples for the top three providers.

1. OpenAI Setup

First, install the SDK: pip install openai python-dotenv
Create a .env file in your directory: OPENAI_API_KEY=sk-your-key-here

import os
from dotenv import load_dotenv
from openai import OpenAI

# Load the environment variables from the .env file
load_dotenv()

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

response = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.5,
    messages=[
        {"role": "system", "content": "You are a sarcastic but helpful coding tutor."},
        {"role": "user", "content": "Explain what a 'while' loop is."}
    ]
)

print(response.choices[0].message.content)

2. Anthropic (Claude) Setup

pip install anthropic

import os
from dotenv import load_dotenv
from anthropic import Anthropic

load_dotenv()
client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))

message = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1000,
    temperature=0,
    system="You are an expert editor.",
    messages=[
        {"role": "user", "content": "Proofread this sentence: Their going to the store, over their."}
    ]
)

print(message.content[0].text)

Demystifying RAG (Retrieval-Augmented Generation)

If you ask ChatGPT, “What is our company’s refund policy?”, it will hallucinate an answer because it hasn’t read your employee handbook. You cannot train a new model from scratch every time your policy changes—that costs millions of dollars.

The solution is RAG. Instead of retraining the brain (the LLM), we give the brain an open book to read before it answers.

The RAG Architecture (Step-by-Step)

Phase 1: Ingestion (Preparing the Data)

1. Document Loading: You load your PDF (e.g., a 100-page HR manual).
2. Chunking: You cannot send 100 pages to an AI easily. You split the document into “chunks” of 500 words each.
3. Embedding: You send each chunk to an Embedding Model (like OpenAI’s text-embedding-3-small). The model reads the chunk and converts the meaning of the text into a massive array of floating-point numbers (a Vector).
   • Analogy: Imagine mapping words on a graph. “Dog” and “Puppy” will have coordinates very close to each other. “Dog” and “Spaceship” will be far apart.
4. Vector Database: You store these number arrays in a specialized Vector DB (like Pinecone, ChromaDB, or Qdrant).

Phase 2: Retrieval & Generation (Answering the User)

1. User Query: The user asks, “How many vacation days do I get?”
2. Query Embedding: The RAG system converts the user’s question into a Vector using the exact same Embedding Model.
3. Semantic Search (Cosine Similarity): The system searches the Vector Database for the chunks mathematically closest to the question’s vector. It retrieves the top 3 most relevant paragraphs.
4. Augmentation & Prompting: The system builds a prompt behind the scenes:
   • System Prompt: “Answer the user’s question using ONLY the provided context.”
   • Context: [Pasted text from the 3 retrieved paragraphs]
   • User Question: “How many vacation days do I get?”
5. Generation: The LLM reads the context, finds the answer, and generates a natural language response.

Chunking Strategies

How you cut your text matters immensely.

• Naive Chunking: Cutting exactly every 500 characters. Problem: You might cut a sentence in half, destroying the meaning.
• Sentence-Split Chunking: Using libraries like LangChain or NLTK to split cleanly at periods.
• Semantic Chunking: Using an AI to group text based on topic changes.

Full Python RAG Tutorial (Using LangChain and Chroma)

This script shows a basic RAG setup using a local, open-source vector database (Chroma) and OpenAI.

Prerequisites: pip install langchain langchain-openai chromadb tiktoken

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import CharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma

load_dotenv()

# 1. Load the document
# Assume we have a text file called 'company_policy.txt'
loader = TextLoader("company_policy.txt")
documents = loader.load()

# 2. Chunk the document
text_splitter = CharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = text_splitter.split_documents(documents)

# 3. Create Embeddings and store in Chroma Vector DB
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
# This creates a local SQLite-based vector database in the directory
db = Chroma.from_documents(chunks, embeddings)

# 4. Create a retriever
retriever = db.as_retriever(search_kwargs={"k": 2}) # Get top 2 results

# 5. The User's Query
query = "What is the policy on working from home?"

# 6. Retrieve relevant chunks
relevant_docs = retriever.invoke(query)
context = "\n\n".join([doc.page_content for doc in relevant_docs])

# 7. Ask the LLM to generate the final answer
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
prompt = f"""
You are a helpful HR assistant. Answer the question based ONLY on the context below.
If the answer is not in the context, say "I don't know based on the company policy."

Context:
{context}

Question: {query}
"""

response = llm.invoke(prompt)
print("AI Response:\n", response.content)

No-Code / Low-Code RAG Alternatives

If you don’t want to write Python, you can build enterprise-grade RAG systems visually:

1. Flowise / LangFlow: These are visual node builders. You drag a “PDF Loader” box, connect it to a “Text Splitter” box, connect that to a “Chroma” box, and connect an “OpenAI Chat Model” to a “Conversational Retrieval Chain”. You can build complex RAG in 10 minutes and host it on your own server.
2. Dify.ai: A powerful open-source platform that handles document parsing, vector storage, and provides a beautiful web UI for your users to chat with the bot.
3. Custom GPTs: The easiest RAG. Just upload a PDF to OpenAI’s Custom GPT interface. However, you lack control over the chunking and embedding process, meaning accuracy on very large documents can degrade.

API Cost Management & Production Tips

• Implement Caching: If your app asks the same question multiple times, cache the response in Redis or a local database. Tools like langchain-cache make this easy.
• Streaming Responses: When building UIs, users hate waiting 10 seconds for an answer. Use the stream=True parameter in the API to send text chunk-by-chunk to the frontend, creating that “typing” effect seen in ChatGPT.
• Structured Outputs: If you need the API to return data for another system (e.g., a database), do not rely on prompt engineering alone. Use OpenAI’s response_format={"type": "json_object"} or the new Structured Outputs feature using Pydantic models to guarantee valid JSON 100% of the time.

Common Pitfalls to Avoid

• Blindly Trusting RAG: Just because RAG uses your documents doesn’t mean it’s flawless. If the retriever fails to find the right chunk in the database, the LLM will confidently say “The document does not mention this.” Always evaluate your retrieval accuracy.
• Lost in the Middle: LLMs suffer from a phenomenon where they pay strong attention to the beginning and end of the context provided to them, but ignore information buried in the middle of a massive block of text. Keep your retrieved context concise.
• Infinite Loops in Agents: If you build autonomous agents (using LangChain or AutoGPT), ensure you set a strict max_iterations limit. Otherwise, an agent that gets confused might call the API in a loop 50,000 times overnight.

Curated Resources

• OpenAI Python Library Documentation
• LangChain Documentation (RAG basics)
• Flowise AI – Visual RAG Builder
• Pinecone: What is a Vector Database?
• PromptingGuide.ai – Advanced RAG Techniques

← Previous | 🏠 Home | Next →