What Is a Vector Database? The AI Search Tool Behind Every RAG App

Why AI Coders Need to Know This

Here's the project that sends every vibe coder down this rabbit hole: "I want to build a chatbot that answers questions about my company docs, my contracts, my product manual." You ask AI to build it. AI immediately starts talking about embeddings, vector stores, and semantic search — and suddenly you're looking at infrastructure you've never seen before.

This isn't AI overcomplicating things. This is a genuinely different kind of database problem — and once you understand why, the whole setup clicks into place.

Think about how a regular database search works. You type "refund" into a search box, and the database scans every record for the exact word "refund." It's like having a filing cabinet with an alphabetical index. Fast and reliable — but only if you use the exact right word. Your filing cabinet doesn't know that "returns," "money back," and "cancel my order" all mean the same thing.

Now think about how AI understands language. When you ask ChatGPT "can I get my money back?" it knows you mean refund. It understands meaning, not just words. A vector database brings that same understanding to your own data. Instead of searching by exact word match, it searches by meaning.

As a builder putting AI at the center of your projects, you will run into vector databases constantly. They power:

• RAG chatbots — the "chat with your documents" apps everyone's building
• Semantic search — search that finds what you mean, not just what you typed
• Recommendation engines — "you might also like" features based on content similarity
• Knowledge bases — internal tools that let teams ask questions across hundreds of documents
• Customer support automation — bots that pull accurate answers from a policy library

You don't need to understand the math. You need to know what it is, what it does, and what to tell your AI when something breaks. That's exactly what this covers.

Real Scenario: You Asked AI to Build a Document Chatbot

You expected maybe a simple API that reads a PDF. Instead, Claude came back with a multi-step setup: install ChromaDB, generate embeddings with OpenAI, chunk the documents, store the chunks, then query them at search time. It might have felt like AI was solving a bigger problem than the one you asked about.

It wasn't. Here's why all of those steps are necessary, and what the code actually does.

What AI Generated

Here's the kind of code Claude or GPT-4 generates for a document chatbot backend. This example uses ChromaDB (runs locally, no account needed) and the OpenAI embedding API.

Step 1 — Ingest Your Documents (Run Once)

This script reads your documents, breaks them into chunks, converts each chunk to an embedding, and stores everything in ChromaDB:

# ingest.py — run this once to load your documents into the vector database

import chromadb
from openai import OpenAI
import os

# Connect to ChromaDB (stores data locally in a folder called chroma_db)
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(name="company_docs")

# Connect to OpenAI for generating embeddings
openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def get_embedding(text: str) -> list[float]:
    """Convert a piece of text into a list of numbers (an embedding)."""
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    """Split a long document into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap  # overlap keeps context across chunk boundaries
    return chunks

# Load and ingest your documents
documents_folder = "./documents"
for filename in os.listdir(documents_folder):
    if filename.endswith(".txt"):
        filepath = os.path.join(documents_folder, filename)
        with open(filepath, "r") as f:
            full_text = f.read()

        chunks = chunk_text(full_text)
        print(f"Ingesting {filename}: {len(chunks)} chunks")

        for i, chunk in enumerate(chunks):
            embedding = get_embedding(chunk)
            collection.add(
                documents=[chunk],
                embeddings=[embedding],
                ids=[f"{filename}_chunk_{i}"],
                metadatas=[{"source": filename}]
            )

print("Done! All documents loaded into the vector database.")

Step 2 — Query at Chat Time (Runs on Every Question)

When a user asks a question, this code converts the question to an embedding, finds the most similar document chunks, and passes them to GPT-4 as context:

# query.py — runs every time a user asks a question

import chromadb
from openai import OpenAI
import os

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(name="company_docs")
openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def get_embedding(text: str) -> list[float]:
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def answer_question(user_question: str) -> str:
    # Step 1: Convert the question to an embedding
    question_embedding = get_embedding(user_question)

    # Step 2: Find the 5 most relevant document chunks
    results = collection.query(
        query_embeddings=[question_embedding],
        n_results=5
    )

    # Step 3: Pull out the matching text
    relevant_chunks = results["documents"][0]
    context = "\n\n---\n\n".join(relevant_chunks)

    # Step 4: Ask GPT-4 to answer using only that context
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant. Answer the user's question "
                    "using only the context provided below. If the answer is "
                    "not in the context, say so honestly.\n\n"
                    f"Context:\n{context}"
                )
            },
            {
                "role": "user",
                "content": user_question
            }
        ]
    )

    return response.choices[0].message.content

# Example usage
question = "What is our policy on returning damaged items?"
answer = answer_question(question)
print(answer)

That's a complete RAG pipeline in under 80 lines of Python. The vector database is the middle layer — it's what makes the question-to-answer connection possible without sending every document to GPT-4 on every request.

Understanding Each Part

Let's walk through what every key concept in that code actually means for you — not the math, just the mental model.

What Is an Embedding?

An embedding is what you get when you take a piece of text and convert it into a long list of numbers. For OpenAI's text-embedding-3-small model, that list is 1,536 numbers long. For the larger model, it's 3,072 numbers.

Here's the crucial insight: similar-meaning text produces similar numbers. The embedding for "refund policy" and the embedding for "returns and exchanges" will produce number lists that are close to each other — even though the words are completely different. The embedding for "recipe for lasagna" will produce a number list that's far away from both of them.

Think of it like GPS coordinates. "Downtown" and "city center" might be described differently, but they'll have nearly the same coordinates. "The woods outside town" will have very different coordinates. A vector database works the same way — it measures the distance between coordinate sets to find what's nearby.

You don't generate embeddings yourself. You call an API (OpenAI, Cohere, or a local model like Ollama), hand it text, and get back a list of numbers. That's it. The model does all the work.

What Is Chunking?

You can't embed an entire 50-page policy document as one unit. That would be like trying to describe an entire city with a single GPS pin. Instead, you break documents into chunks — pieces of 300–500 words — and embed each chunk separately.

The overlap parameter in the code (set to 50 characters) means consecutive chunks share a little text at their borders. This prevents an answer from getting cut in half between two chunks and being missed entirely. It's like cutting boards with a slight overlap on each mark so no measurement gets lost in the saw kerf.

Chunk size is one of the main things you'll tune when a chatbot gives bad answers. Too large and each chunk contains too many topics. Too small and chunks lose their context. AI will help you find the right balance for your documents.

What Is Similarity Search?

When a user asks a question, you convert that question to an embedding — a list of numbers — and then ask the vector database: "Which of the stored embeddings are mathematically closest to this one?"

The database returns the top N closest chunks. Those are the chunks most likely to contain the answer. You pass them to GPT-4 as context, and GPT-4 generates a coherent answer from them.

This is why it works even when words don't match. The question "can I return something I bought last month?" produces an embedding that's close to the chunk containing "items may be returned within 30 days of purchase" — even though there isn't a single word in common. The model understands meaning, and the numbers reflect that.

What Is a Collection?

In ChromaDB (and most vector databases), a collection is the container for your embeddings — like a table in a regular database. You might have one collection for your policy documents, another for your product catalog, and another for customer feedback. Each collection is searched separately.

In Pinecone, the equivalent is called an index. In Weaviate, it's called a class. The concept is the same across all of them: it's the named bucket your vectors live in.

What Is RAG?

RAG stands for Retrieval Augmented Generation. It's the pattern you just saw in the code:

1. Retrieval — find the most relevant chunks from your data (using the vector database)
2. Augmented — add those chunks to the AI's context window (the system prompt)
3. Generation — let the AI generate an answer based on that grounded context

RAG is how you get an AI to answer questions about your specific data without hallucinating. The AI can't make up answers about your refund policy if you give it the actual policy text as context. It's like the difference between asking a new hire to answer from memory versus handing them the employee handbook and asking them to look it up.

What AI Gets Wrong About Vector Databases

AI generates vector database code well, but it makes a handful of predictable mistakes. These are the ones that will cost you the most time.

1. Suggesting a Dedicated Vector DB When pgvector Would Do

This is the most common mistake, and it matters. If you're already using PostgreSQL (or Supabase, which runs on PostgreSQL), you do not need a separate vector database. PostgreSQL has a plugin called pgvector that adds vector search right inside your existing database.

AI often defaults to recommending Pinecone or ChromaDB because those are the most-discussed vector databases in its training data. But if your database is already PostgreSQL, adding a separate vector database means:

• Another service to manage and pay for
• Keeping two databases in sync
• More complex deployment

With pgvector, you store vectors right next to your regular data in the same database you already have. Supabase even enables pgvector by default. If you're on Supabase or PostgreSQL, tell your AI that explicitly and ask it to use pgvector instead of a separate service.

2. Not Explaining Embedding Dimensions — Then Getting the Mismatch Error

Every embedding model produces a fixed-size list of numbers. OpenAI's text-embedding-3-small produces 1,536 numbers. OpenAI's text-embedding-3-large produces 3,072. A local model like nomic-embed-text produces 768.

The problem: when you set up your vector database, you have to tell it the dimension size upfront. If you later switch embedding models (say, from OpenAI to a local model), the dimensions won't match and everything breaks.

AI often sets up the database without making this dimension dependency explicit. Then you swap models to save money, and suddenly you're getting errors that make no sense. The fix: always ask AI to document which embedding model the dimension count is tied to, and re-ingest all documents if you change models.

3. Skipping the Re-Ingestion Step When Documents Change

A vector database is not live. It stores a snapshot of your documents at the time you ran the ingest script. If you update a policy document, add new FAQs, or delete outdated content, the vector database doesn't know. It still has the old embeddings.

AI often builds the ingest step but doesn't build an update workflow. For small document sets, re-running the full ingest script is fine. For larger or frequently-changing content, you need a pipeline that detects changes and updates only the affected chunks.

When you see your chatbot giving outdated answers, this is almost always why. Tell AI: "My documents change regularly. Build the ingestion with an update workflow that replaces embeddings when source documents change."

4. Over-Engineering the Setup for a Simple Use Case

If you have 20 PDF documents that rarely change and fewer than 100 users, you do not need Pinecone's managed cloud service with multiple replicas. ChromaDB running on your existing server is fine. If you're on Supabase, pgvector is even simpler.

AI sometimes reaches for the production-grade, paid solution when a local or built-in option would work perfectly well. Ask it: "I have [X documents], [Y users], and [Z update frequency]. What's the simplest vector database setup that works at this scale?" You'll often get a much simpler recommendation.

How to Debug Vector Database Issues With AI

Vector database problems fall into a small number of categories. Here's how to identify them and what to tell your AI.

Problem: "The chatbot gives wrong or irrelevant answers"

This is the most common complaint, and it has several causes. Work through these in order:

1. Check if the right chunks are being retrieved. Add a debug print to your query function that shows you the top 5 retrieved chunks before they're sent to GPT-4. If the retrieved chunks don't contain the answer, the problem is in retrieval — not in the AI's response generation.

2. Adjust chunk size. If retrieved chunks are too short, they lose context. If they're too long, they contain too many topics and dilute the relevance score. Try 400–600 word chunks for general documents, smaller for Q&A-style content.

3. Increase the number of retrieved chunks. If n_results=5 isn't capturing the answer, try n_results=10. You're giving the model more material to work with.

Problem: DimensionalityError or embedding size mismatch

ValueError: Collection expects embeddings with 1536 dimensions, got 768

You changed embedding models without re-ingesting your documents. The stored embeddings are a different size than the new ones you're generating.

Problem: OPENAI_API_KEY errors or rate limits

openai.AuthenticationError: Incorrect API key provided

Your OpenAI API key isn't set, or the environment variable isn't being loaded. This is an environment variable issue, not a vector database issue.

Problem: Ingestion is slow or times out for large document sets

You're calling the embedding API once per chunk sequentially. For large document libraries this takes a long time and may hit API rate limits.

Quick Reference: Vector Database Options at a Glance

Here's how the main options compare for a vibe coder building a real project:

The practical decision tree:

• Already using Supabase or PostgreSQL? → Use pgvector
• Just prototyping or building locally? → Use ChromaDB
• Need a managed cloud service for production? → Use Pinecone
• Need keyword + semantic search combined? → Use Weaviate

What to Learn Next

Vector databases don't live in isolation — they sit between your regular data layer and your AI layer. These articles fill in the surrounding context:

• What Is a Database? Start here if you're still fuzzy on what a database actually is. Understanding regular databases makes vector databases much easier to reason about — they're the same idea, just with a different kind of data inside.
• What Is PostgreSQL? PostgreSQL is the database most serious projects end up on — and with the pgvector extension, it can do vector search too. If you're choosing a stack, PostgreSQL + pgvector is often the simplest all-in-one answer.
• What Is Supabase? Supabase is a hosted PostgreSQL service with pgvector enabled by default. It's often the fastest path to a production-ready RAG app — database, auth, storage, and vector search in one platform.
• What Is an API? Your vector database query code runs as part of an API that your frontend calls. If the concept of an API is still fuzzy, this is the essential background for understanding how the moving parts connect.