Get All Documents from ChromaDB: LangChain and Native Client

How do I get all documents from ChromaDB in Python?

Both the LangChain wrapper and the native client expose a get() method that reads records by id or filter without running a similarity search. Called with no arguments it returns every record in the collection.

The LangChain wrapper delegates to the native collection, so the two return the same dictionary shape: ids, documents, metadatas, embeddings, plus uris and data in recent versions. The unrequested fields are None.

# LangChain wrapper (pip install langchain-chroma)
from langchain_chroma import Chroma

vectorstore = Chroma(
    collection_name="my_docs",
    embedding_function=embeddings,          # same one used at ingestion
    persist_directory="./chroma_db",
)
data = vectorstore.get(include=["documents", "metadatas", "embeddings"])
print(data.keys())
print(len(data["ids"]), len(data["documents"]))

# Native client
import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_collection("my_docs")
data = collection.get(include=["documents", "metadatas", "embeddings"])

# Escape hatch from the wrapper to the native collection
data = vectorstore._collection.get()

Why does Chroma get() not return embeddings?

Embeddings are large (for example 1,536 floats per record for many OpenAI models), so Chroma leaves them out unless you ask. The default include is documents and metadatas, which is why a plain get() shows embeddings as None.

Ask for them with include. The ids are always returned, and passing 'ids' inside include raises a validation error, so do not list it there.

Another common surprise: the returned object is a dictionary, not a list of documents. Use data['documents'][i], data['metadatas'][i] and data['ids'][i] for record i; the lists are aligned by position.

How do I fetch a large collection in batches with limit and offset?

Without limit, get() tries to return everything in one call. That can exhaust memory with embeddings included, and hosted or server deployments may cap the page size. Batches of 500 to 5000 records are a sensible range; smaller if embeddings are wide.

Loop until a batch comes back empty. This works with filters too, where collection.count() would overstate the total because it counts the whole collection.

import numpy as np

def fetch_all(collection, batch=1000, where=None, where_document=None):
    ids, docs, metas, embs = [], [], [], []
    offset = 0
    while True:
        r = collection.get(
            limit=batch,
            offset=offset,
            where=where,
            where_document=where_document,
            include=["documents", "metadatas", "embeddings"],
        )
        if len(r["ids"]) == 0:
            break
        ids.extend(r["ids"])
        docs.extend(r["documents"])
        metas.extend(r["metadatas"])
        embs.extend(list(r["embeddings"]))
        offset += len(r["ids"])
    return ids, docs, metas, np.array(embs)

ids, docs, metas, embs = fetch_all(collection)
print(len(ids), collection.count(), embs.shape)

How do I filter while fetching (by source, text or id)?

get() accepts the same filters as a query, so you rarely need to download everything and filter in Python. Metadata filters use where, text filters use where_document, and specific records use ids.

Combine conditions with $and or $or. Metadata filters match exact stored values, so a source saved as an int will not match a string.

# exact metadata match
collection.get(where={"source": "report.pdf"})

# several conditions
collection.get(where={"$and": [{"source": "report.pdf"}, {"page": {"$gte": 3}}]})

# substring in the document text
collection.get(where_document={"$contains": "invoice"})

# specific records
collection.get(ids=["64f1a...", "64f1b..."], include=["documents", "embeddings"])

# LangChain equivalent
vectorstore.get(where={"source": "report.pdf"}, limit=100, offset=0)

Why is my collection empty or smaller than what I ingested?

Four causes account for most mismatches. Check them in this order and compare collection.count() against the number of records you ingested.

How do I keep MongoDB ids aligned and feed everything into BERTopic?

The cleanest mapping is to make the Chroma id the Mongo _id as a string at ingestion. ObjectId is not a valid id type, so convert it with str(). Metadata values must be str, int, float or bool; None, nested dicts and (in many versions) lists are rejected, so convert or drop them first.

At read time, pass the stored embeddings to BERTopic so it does not re-embed. docs must be a list of strings, and embeddings a 2D numpy array in the same order.

# --- ingestion: keep the Mongo _id ---
texts = [d["text"] for d in mongo_docs]
ids = [str(d["_id"]) for d in mongo_docs]
metas = [{"source": d.get("source") or "unknown"} for d in mongo_docs]  # no None values
vectorstore.add_texts(texts=texts, metadatas=metas, ids=ids)

# --- export + topic modeling ---
from bertopic import BERTopic
ids, docs, metas, embs = fetch_all(collection)

topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs, embeddings=embs)

# --- write back to MongoDB ---
from pymongo import UpdateOne
from bson import ObjectId

ops = [
    UpdateOne({"_id": ObjectId(i)}, {"$set": {"topic": int(t)}})
    for i, t in zip(ids, topics)
]
for start in range(0, len(ops), 1000):
    mongo_collection.bulk_write(ops[start:start + 1000], ordered=False)

# --- or write back to Chroma metadata ---
for start in range(0, len(ids), 1000):
    collection.update(
        ids=ids[start:start + 1000],
        metadatas=[{"topic": int(t)} for t in topics[start:start + 1000]],
    )