LlamaIndex reader · TranscriptFetch docs

A LlamaIndex reader for video transcripts: one Document per video, or per chunk with a link to the second it starts.

LlamaIndex reader. llama-index-readers-transcriptfetch loads video transcripts as LlamaIndex Documents. Package on PyPI, source and issues on GitHub. It is built on the Python SDK (transcriptfetch-sdk 2.5.0) and fetches transcripts with the SDK's transcripts.batch(), 50 at a time, asking the API for chunks when you set chunks=True.

Install

BASH

pip install llama-index-readers-transcriptfetch
export TRANSCRIPTFETCH_API_KEY="tf_live_..."

Create a key on the API keys page. A free account gets 50 free credits a month, with no card.

Load transcripts

PYTHON

from llama_index.readers.transcriptfetch import TranscriptFetchReader

reader = TranscriptFetchReader()
docs = reader.load_data([
    "https://www.youtube.com/watch?v=aircAruvnKk",
    "jNQXAC9IVRw",
])
for doc in docs:
    print(doc.metadata["title"], doc.metadata["transcript_source"])

Options are set on the reader, and sources go to reader.load_data(...), or to reader.lazy_load_data(...) to get Documents one at a time.

Sources

Pass exactly one of these to load_data or lazy_load_data:

Argument	What it loads
`videos` (positional)	A video URL or ID, or a list of them: YouTube, TikTok, Instagram, or a direct audio or video file URL
`channel=`	A YouTube channel (@handle, URL or ID), or a TikTok or Instagram profile URL
`playlist=`	A YouTube or TikTok playlist
`search=`	The videos a search returns, on YouTube unless you pass `search_platform="tiktok"` or `search_platform="instagram"`

channel, playlist and search load at most max_videos= videos (default 25).

Documents and metadata

By default the reader returns one Document per video, with these metadata keys:

Key	Value
`source`	The video URL
`video_id`	The video's ID
`platform`	The platform the video is on
`title`	The video's title
`channel`	The channel or profile that posted it
`duration`	The video's length in seconds
`language`	The transcript's language
`transcript_source`	`"captions"` or `"audio"`

Links, IDs and timings are left out of the text that gets embedded (excluded_embed_metadata_keys) and kept in the metadata for citations.

Chunks for RAG

With chunks=True the API splits each transcript between caption lines, never mid-word, with a small overlap between pieces, and the reader returns one Document per chunk. You do not need a text splitter. Each chunk's source is a link that opens the video at the chunk's start, such as https://www.youtube.com/watch?v=aircAruvnKk&t=4s, and its metadata adds video_url, chunk_index, start and end (in seconds) and tokens.

chunk_tokens sets the chunk size, from 100 to 2000 tokens (default 500), and chunk_overlap the overlap, from 0 to 200 tokens (default 50). Tokens are counted with OpenAI's o200k_base encoding.

PYTHON

from llama_index.core import VectorStoreIndex
from llama_index.readers.transcriptfetch import TranscriptFetchReader

reader = TranscriptFetchReader(chunks=True)
docs = reader.load_data(channel="@3blue1brown", max_videos=20)

index = VectorStoreIndex.from_documents(docs)
response = index.as_query_engine().query("What does a neuron hold?")
print(response)
for node in response.source_nodes:
    print(node.metadata["source"])

Options

Option	Default	What it does
`mode`	`"auto"`	`"auto"` transcribes the audio of videos that have no captions, which most TikToks and Reels need. `"captions"` skips those videos instead, and a skipped video costs nothing.
`chunks`	`False`	One Document per chunk instead of per video
`chunk_tokens`	`500`	Chunk size, from 100 to 2000 tokens
`chunk_overlap`	`50`	Overlap between chunks, from 0 to 200 tokens
`wait_for_audio`	`True`	Wait for audio transcriptions to finish. With `False`, videos that need one are skipped.
`audio_timeout`	`900`	How long to wait for an audio transcription, in seconds
`continue_on_failure`	`True`	Log a video that fails and skip it
`api_key`	`TRANSCRIPTFETCH_API_KEY`	Your API key
`base_url`	The production API	The API base URL, to point at another environment

Credits

A transcript or a listing page costs 1 credit, audio transcription costs 1 credit per started 5 minutes of audio, and failed requests are free. See pricing and billing and credits.