Sebastian: My Personal RAG Knowledge Assistant

The Concept

Have you ever thought of an idea, opened Instagram, and the exact same thing shows up on your feed? That eerie moment when algorithms seem to know you better than you know yourself.

Now imagine having that kind of intelligence, but working for you, not against you.

No data harvesting, no ads, no cloud logging, just an assistant that actually remembers your world.

Sebastian isn't a chatbot. He's a personal memory system (part calendar manager, part research assistant, part life archivist)

He doesn't try to replace me; he observes how I work and makes sure I never lose track of what matters.

Why I Built This

Sebastian is built around one principle:

He doesn't just respond to prompts, he uses context from my digital life to make sense of what I ask.

Real Examples:

• "Remind me to reply to Youssef next week"
  → He knows Youssef is in my Gmail threads
• "Find me time to finish the proposal"
  → He checks my Google Calendar
• "Summarize my last meeting notes"
  → He pulls data from Notion
• "Where did I park last Tuesday?"
  → He checks Google Maps location history

🔒 Privacy First: All of this works locally or through my own API keys, so my data never leaves my environment.

Sebastian integrates with my entire digital workspace

Architecture Overview

Sebastian runs as a local API layer built with FastAPI, powered by a RAG engine, and enhanced by several personal integrations.

┌──────────────────────────────┐
│        FastAPI Core          │
│   (Query Router + Auth)      │
└─────────────┬────────────────┘
              │
    ┌─────────┼──────────┐
    │         │          │
 Notion     Gmail     Calendar
  API        API        API
    │         │          │
    └─────────┼──────────┘
              ↓
       Context Aggregator
              ↓
   pgvector + OpenAI Embeddings
              ↓
         RAG Engine
              ↓
       GPT-4o-mini

🧰 Tech Stack

🧠 Memory System

Sebastian's long-term memory uses RAG (Retrieval-Augmented Generation).
All my data, from meeting notes to location logs, is vectorized and stored in PostgreSQL with pgvector.

Database Schema

The memory system stores embeddings alongside metadata, enabling semantic search across all my digital information sources.

Example Memory Entries

This means Sebastian can answer:

• "What did I discuss with Youssef last week?"
• "Show me notes related to my next presentation."
• "What's my busiest weekday this month?"

Example Workflow: "Summarize my week"

Let's say I ask Sebastian to summarize my week. Here's what happens internally:

1️⃣ Collect Context

Fetch events from Google Calendar, unread emails, and recent Notion notes.

2️⃣ Embed & Rank

Generate embeddings for all entries → use pgvector to find the most relevant ones.

3️⃣ Compose Prompt

Build a structured context summary (meetings, notes, tasks).

4️⃣ Generate Insight

Ask GPT-4o-mini to produce a weekly summary with actionable insights.

Sebastian's Response:

Hosting

Sebastian is self-hosted, meaning:

🚀 Next Steps

• Voice Interface: Add voice-to-text using Whisper + TTS for hands-free interaction
• Home Automation: Connect to Home Assistant for environment control
• Local Embeddings: Use nomic-embed-text for fully offline operation
• Daily Journaling: Auto-generate daily journal from my data patterns
• Notion Dashboard: Stream responses via WebSocket to live Notion dashboard
• Proactive Insights: Sebastian suggests optimizations before I ask