Auto-Retrain
Automatically re-crawl URL-based knowledge sources to keep your chatbot up to date
Auto-Retrain
Keep your chatbot's knowledge current by automatically re-crawling URL sources on a schedule.
How It Works
- A scheduled background task checks for URL-type sources with auto-retrain enabled (the schedule is attached in the background-job dashboard, typically hourly)
- Checks if enough time has passed since the last retrain (default: 24 hours; a source that has never been retrained runs immediately)
- Re-crawls the URL
- Compares word count — if content changed by more than 10 words, re-indexes the source
- If unchanged, skips re-indexing (saves processing)
Sources with auto-retrain enabled show a refresh icon in the Knowledge sources list.
Configuration
- Interval: 24 hours (configurable per source via
retrainInterval, in hours) - Content change detection: Compares word counts before and after crawl
- Skips unchanged content: Only re-indexes when content actually changed
- Applies to
url,website, andwebpagesource types
Last updated on