Domain Module

Core data models for notebooks, sources, notes, and settings with async SurrealDB persistence, auto-embedding, and relationship management.

Purpose

Two base classes support different persistence patterns: ObjectModel (mutable records with auto-increment IDs) and RecordModel (singleton configuration with fixed IDs).

Key Components

base.py

• ObjectModel: Base for notebooks, sources, notes

  • save(): Create/update with auto-embedding for searchable content
  • delete(): Remove by ID
  • relate(relationship, target_id): Create graph relationships (reference, artifact, refers_to)
  • get(id): Polymorphic fetch; resolves subclass from ID prefix
  • get_all(order_by): Fetch all records from table
  • Integrates with ModelManager for automatic embedding

• RecordModel: Singleton configuration (ContentSettings, DefaultPrompts)

  • Fixed record_id per subclass
  • update(): Upsert to database
  • Lazy DB loading via _load_from_db()

notebook.py

• Notebook: Research project container

  • get_sources(), get_notes(), get_chat_sessions(): Navigate relationships
  • get_delete_preview(): Returns counts of notes, exclusive sources, and shared sources that would be affected by deletion
  • delete(delete_exclusive_sources): Cascade deletion - always deletes notes, optionally deletes exclusive sources, always unlinks all sources

• Source: Content item (file/URL)

  • vectorize(): Submit async embedding job (returns command_id, fire-and-forget)
  • get_status(), get_processing_progress(): Track job via surreal_commands
  • get_context(): Returns summary for LLM context
  • add_insight(): Submit async insight creation via create_insight_command (fire-and-forget, returns command_id)

• Note: Standalone or linked notes

  • save(): Submits embed_note command after save (fire-and-forget)
  • add_to_notebook(): Link to notebook

• SourceInsight, SourceEmbedding: Derived content models

• ChatSession: Conversation container with optional model_override

• Asset: File/URL reference helper

• Search functions:

  • text_search(): Full-text keyword search. On a SurrealDB search::highlight "position overflow" (large/multi-byte chunks) it falls back to vector_search(); if that also fails it raises DatabaseOperationError (never silently returns an empty list)
  • vector_search(): Semantic search via embeddings (default minimum_score=0.2)

content_settings.py

• ContentSettings: Singleton for processing engines, embedding strategy, file deletion, YouTube languages

transformation.py

• Transformation: Reusable prompts for content transformation
• DefaultPrompts: Singleton with transformation instructions

credential.py

• Credential: Individual credential records for API keys and provider configuration

  • One record per credential: Each credential (e.g., "My OpenAI Key", "Work Anthropic") is a separate Credential record in SurrealDB
  • Fields: name, provider, modalities (list), api_key (SecretStr), base_url, endpoint, api_version, endpoint_llm/embedding/stt/tts, project, location, credentials_path
  • SecretStr protection: API key field uses Pydantic's SecretStr (values masked in logs/repr)
  • Encryption integration: Uses encrypt_value()/decrypt_value() from open_notebook.utils.encryption
    • Keys encrypted with Fernet before database storage
    • Requires OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable (warns if not set)
  • Key methods:
    • to_esperanto_config(): Builds config dict for Esperanto's AIFactory methods
    • get_by_provider(provider): Class method to fetch all credentials for a provider
    • get_linked_models(): Returns all Model records linked to this credential
  • Custom serialization: _prepare_save_data() extracts SecretStr values and encrypts before storage
  • Decryption on read: get() and get_all() overridden to decrypt api_key after fetch

• Note: provider_config.py still exists for legacy migration support (migrating old ProviderConfig records to Credential)

Important Patterns

• Async/await: All DB operations async; always use await
• Polymorphic get(): ObjectModel.get(id) determines subclass from ID prefix (table:id format)
• Fire-and-forget embedding: Models submit embed_* commands after save via submit_command() (non-blocking)
• Nullable fields: Declare via nullable_fields ClassVar to allow None in database
• Timestamps: created and updated auto-managed as ISO strings
• Fire-and-forget jobs: source.vectorize() returns command_id without waiting

Key Dependencies

• surrealdb: RecordID type for relationships
• pydantic: Validation and field_validator decorators
• open_notebook.database.repository: CRUD and relationship functions
• open_notebook.ai.models: ModelManager for embeddings
• surreal_commands: Async job submission (vectorization, insights)
• loguru: Logging

Quirks & Gotchas

• Polymorphic resolution: ObjectModel.get() fails if subclass not imported (search subclasses list)
• RecordModel singleton: new returns existing instance; call clear_instance() in tests
• Source.command field: Stored as RecordID; auto-parsed from strings via field_validator
• Text truncation: Note.get_context(short) hardcodes 100-char limit
• Auto-embedding behavior:
  • Note.save() → auto-submits embed_note command
  • Source.save() → does NOT auto-submit (must call vectorize() explicitly)
  • Source.add_insight() → submits create_insight_command which handles DB insert + embed_insight command (all fire-and-forget)
• Relationship strings: Must match SurrealDB schema (reference, artifact, refers_to)

How to Add New Model

1. Inherit from ObjectModel with table_name ClassVar
2. Define Pydantic fields with validators
3. Override save() to submit embedding command if searchable (use submit_command("embed_*", id))
4. Add custom methods for domain logic (get_X, add_to_Y)
5. Implement _prepare_save_data() if custom serialization needed

Usage

notebook = Notebook(name="Research", description="My project")
await notebook.save()

obj = await ObjectModel.get("notebook:123")  # Polymorphic fetch

# Search
await text_search("quantum", results=5)
await vector_search("quantum computing", results=10, minimum_score=0.3)

---

Source: lfnovo/open-notebook · 31809★ Repo: An Open Source implementation of Notebook LM with more flexibility and features