Google Drive → Searchable Meeting Index
Keeps the Meeting Intelligence Chatbot's search index current: it syncs only new or updated files from a shared Google Drive folder, then cleans, chunks, embeds and indexes them so the chatbot can answer from them.
Workflow
From “App startup, a greeting, page load or the Sync button” to result
- 1. Trigger
Authenticate
The backend signs in to Google Drive with a service account. If sign-in fails, it logs a warning and the chat keeps working on the existing index.
- 2. Action
List the folder
A paged Drive listing returns each file's ID, name, type, size and modified time. The listing is cached for 60 seconds to avoid redundant API calls.
- 3. Action
Filter
Unsupported file types are skipped, and so are files modified outside the 90-day window.
- 4. Action
Compare with sync state
Each file's modifiedTime is checked against gdrive_sync_state.json. Unchanged files count as up to date; the rest are marked new or updated.
- 5. Action
Download
Changed files are downloaded. Google Docs, Sheets and Slides are exported to text formats first.
- 6. Action
Parse, clean & chunk
Each format has its own parser. Meeting transcripts lose their filler words and are cut into speaker-aware chunks of 800 words with 150 overlapping; other documents into 600 with 100.
- 7. Action
Embed & index
Chunks are embedded with all-MiniLM-L6-v2 in batches of 32, added to the FAISS index, and the BM25 keyword index is rebuilt. Duplicate content hashes are skipped.
- 8. Output
Save sync state
The index and the updated sync state are written to disk, and the sync reports files synced, new chunks, files up to date and files outside the date range.
Does this file need syncing?
flowchart TD
S([Sync starts]) --> AU{Drive authenticated?}
AU -- no --> X[Stop: not authenticated]
AU -- yes --> L[List folder, cached 60s]
L --> F{Supported type?}
F -- no --> SK[Skip]
F -- yes --> W{Modified in last 90 days?}
W -- no --> OUT[Outside date range]
W -- yes --> C{Same modifiedTime as last sync?}
C -- yes --> UTD[Up to date]
C -- no --> P[Process: new or updated]
P --> I[Index all new chunks]
I --> ST[(Save sync state)]Problem
The chatbot can only answer from what is in its index, and meeting transcripts keep arriving in a shared Drive folder. Re-processing every file on every sync would waste time and API calls, and a cold start meant loading a heavy embedding model on top of stale data. The automation has to notice what changed in the folder, process only that, and have the app ready with fresh data before the first person starts chatting.
Architecture
Four components do the work: a Google Drive service with a 60-second folder cache, a file processor with one parser per format (PDF, DOCX, XLSX, CSV, PPTX, TXT, Markdown and JSON), a transcript processor that cleans and chunks meetings, and a vector store manager that owns the FAISS and BM25 indexes. State lives in three files on a persistent volume: vector_store.faiss, metadata.pkl and gdrive_sync_state.json. There is no database. The same sync runs from several places. At startup, the app loads the embedding model in a background thread, loads the saved index, signs in to Drive and syncs only if any files need it, then waits up to 60 seconds for the model so the first request doesn't have to. A greeting in chat runs a silent sync before replying. The React app runs one on page load when Drive is connected, and the Sync button runs it on demand. A force-full-sync endpoint ignores the saved state.
Startup: ready with fresh data
- 1. Trigger
Launch script runs
Loads .env, stops the old process
- 2. Action
Load model in background
all-MiniLM-L6-v2 on CPU
- 3. Data store
Load saved index
FAISS + metadata from disk
- 4. Action
Sign in to Drive
Service account
- 5. Decision
Files changed?
Compare with sync state
- 6. Action
Incremental sync
Download, chunk, embed, index
- 7. Action
Wait for model
up to 60s - 8. Output
App ready
Saves index and state on shutdown
Input
A shared Google Drive folder of meeting-recorder JSON transcripts and PDF, DOCX, XLSX, PPTX, CSV, TXT and Markdown files.
Output
New chunks in the FAISS and BM25 indexes, an updated sync-state file, and counts of files synced, new chunks, files up to date and files outside the date range.
Integrations
Limitations
The greeting-triggered sync runs inside a chat request, which adds latency to that reply. The automation watches a single, fixed folder. Files trashed in Drive are skipped when listing, but their chunks stay in the index: there is no removal step yet. Startup fails if the embedding model doesn't load within 60 seconds.
Security Notes
The service-account JSON key sits on the server with chmod 600 and is excluded from git, and the Drive folder has to be shared with the service-account email. Other credentials are read from git-ignored .env files. The API endpoints that trigger a sync have no authentication of their own, which is one of the gaps the planned redesign addresses.
Learnings
Tracking each file's modifiedTime is enough to make sync incremental: only new or updated files are processed, and the rest are counted as up to date. Caching the folder listing cut redundant Drive API calls, and shortening the Drive timeout from 60s to 30s made failures show up faster. Loading the embedding model in the background at startup means the first user doesn't wait for it.