LangChain document loader · TranscriptFetch docs

A LangChain document loader for video transcripts: one Document per video, or per chunk with a link to the second it starts.

LangChain document loader. langchain-transcriptfetch loads video transcripts as LangChain Documents. Package on PyPI, source and issues on GitHub. It is built on the Python SDK (transcriptfetch-sdk 2.5.0) and fetches transcripts with the SDK's transcripts.batch(), 50 at a time, asking the API for chunks when you set chunks=True.

Install

BASH

pip install -U langchain-transcriptfetch
export TRANSCRIPTFETCH_API_KEY="tf_live_..."

Create a key on the API keys page. A free account gets 50 free credits a month, with no card.

Load transcripts

PYTHON

from langchain_transcriptfetch import TranscriptFetchLoader

loader = TranscriptFetchLoader(videos=[
    "https://www.youtube.com/watch?v=aircAruvnKk",
    "jNQXAC9IVRw",
])
docs = loader.load()
for doc in docs:
    print(doc.metadata["title"], doc.metadata["transcript_source"])

TranscriptFetchLoader is a LangChain BaseLoader: .load() returns a list of Documents and .lazy_load() yields them one at a time.

Sources

Pass exactly one of these:

Argument	What it loads
`videos`	A video URL or ID, or a list of them: YouTube, TikTok, Instagram, or a direct audio or video file URL
`channel`	A YouTube channel (@handle, URL or ID), or a TikTok or Instagram profile URL
`playlist`	A YouTube or TikTok playlist
`search`	The videos a search returns, on YouTube unless you pass `search_platform="tiktok"` or `search_platform="instagram"`

channel, playlist and search load at most max_videos videos (default 25).

Documents and metadata

By default the loader returns one Document per video, with these metadata keys:

Key	Value
`source`	The video URL
`video_id`	The video's ID
`platform`	The platform the video is on
`title`	The video's title
`channel`	The channel or profile that posted it
`duration`	The video's length in seconds
`language`	The transcript's language
`transcript_source`	`"captions"` or `"audio"`

Chunks for RAG

With chunks=True the API splits each transcript between caption lines, never mid-word, with a small overlap between pieces, and the loader returns one Document per chunk. You do not need a text splitter. Each chunk's source is a link that opens the video at the chunk's start, such as https://www.youtube.com/watch?v=aircAruvnKk&t=4s, and its metadata adds video_url, chunk_index, start and end (in seconds) and tokens.

chunk_tokens sets the chunk size, from 100 to 2000 tokens (default 500), and chunk_overlap the overlap, from 0 to 200 tokens (default 50). Tokens are counted with OpenAI's o200k_base encoding.

PYTHON

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_transcriptfetch import TranscriptFetchLoader

loader = TranscriptFetchLoader(channel="@3blue1brown", max_videos=20, chunks=True)
docs = loader.load()

store = InMemoryVectorStore.from_documents(docs, OpenAIEmbeddings())
for doc in store.similarity_search("what does a neuron hold?", k=3):
    print(doc.metadata["source"])

Options

Option	Default	What it does
`mode`	`"auto"`	`"auto"` transcribes the audio of videos that have no captions, which most TikToks and Reels need. `"captions"` skips those videos instead, and a skipped video costs nothing.
`wait_for_audio`	`True`	Wait for audio transcriptions to finish. With `False`, videos that need one are skipped.
`audio_timeout`	`900`	How long to wait for an audio transcription, in seconds
`continue_on_failure`	`True`	Log a video that fails and skip it
`api_key`	`TRANSCRIPTFETCH_API_KEY`	Your API key
`base_url`	The production API	The API base URL, to point at another environment

Credits

A transcript or a listing page costs 1 credit, audio transcription costs 1 credit per started 5 minutes of audio, and failed requests are free. See pricing and billing and credits.