Commit Graph

25 Commits

Author SHA1 Message Date
sabaimran
f3045ab436 replace system prompt for full thoughts, not for datatrace 2025-02-03 23:23:33 -08:00
sabaimran
899e9a6c23 Add additional log lines 2025-02-03 13:16:07 -08:00
sabaimran
a35002e93b Use log level info for constructing the dataset. Temp: comment out error lines when reading in JSONL content 2025-02-03 13:04:19 -08:00
sabaimran
3db3d7bbb4 add print line about full thoughts 2025-02-03 07:17:42 -08:00
sabaimran
f870a2f7a2 Add more outputs about nature of dataset when uploading and configure custom system prompt 2025-02-03 07:11:34 -08:00
sabaimran
9815bbc8a1 Add method for loading the entire dataset, loading good evaluated rows, and uploading a merged dataset 2025-01-30 22:39:46 -08:00
sabaimran
06a900069f add line by line parsing logic for the dataset jsonl file 2025-01-29 11:58:32 -08:00
sabaimran
79f3f61b70 allow multiple eval_paths to combine eval results across multiple runs 2025-01-27 15:29:29 -08:00
sabaimran
6257d1bb62 Extract good rows before attempting to upload full thoughts df 2025-01-26 22:33:07 -08:00
sabaimran
ef794a78d5 Publish thoughts dataset to a separate repo from the subiteration traces 2025-01-26 21:50:03 -08:00
sabaimran
e5d5153fc4 Allow setting dynamic filepaths for full thought trace. Update data upload script to handle full thoughts. 2025-01-26 21:44:26 -08:00
sabaimran
7f001768c1 Update output and input formats for datasets
- intermediate trains of thought are outputted as jsonl in order to store nested / complex data
- evaluation results are stored as csv without references for simplicity
- the create dataset file reads both of them, combines and outputs the results ot huggingface, if possible
2025-01-26 21:42:06 +07:00
Debanjum
818239c3c7 Allow saving Khoj chat actor query, response traces to HF dataset
- Data tracer can be enabled by setting DATATRACE_PATH env var
- Save query, response pairs to a HF dataset locally in trainable
  format
- Currently only enabled for research mode apick_next_tool research mode
  chat actor
- Use prompt tracer like code execution hooks
2025-01-26 03:03:17 +07:00
Debanjum Singh Solanky
7d3a208f8b Update bump version script to bump new next.js web app version too 2024-08-05 16:20:47 +05:30
sabaimran
7e6b611a19 Fix typo for Obsidian 2024-08-05 15:55:06 +05:30
Debanjum Singh Solanky
7cb14ff07a Add dev setup script. Run prettier on web app pre-commit 2024-08-05 15:49:31 +05:30
Debanjum Singh Solanky
54132efd67 Fix Khoj Obsidian plugin build 2024-07-07 18:26:10 +05:30
Debanjum Singh Solanky
bbfd320ed4 Use Yarn instead of NPM to bump Desktop, Obsidian client versions 2024-06-26 10:37:58 +05:30
Debanjum Singh Solanky
57a36967bf Run Obsidian version script in bump_version.sh to write to versions.json
This handles updates from manifest.json minAppVersion field to the
versions.json file.

The minAppVersion field is for the minimum Obsidian app version
supported by a Khoj plugin version
2024-06-23 08:18:55 +05:30
Debanjum Singh Solanky
59edb99f04 Simplify, improve bump version development script
- Just use in-built `npm version' command to update desktop, obsidian version
- Upgrade by major, minor or patch version using new -t flag in script
  E.g bump_version -t minor
2024-06-22 18:19:38 +05:30
Debanjum Singh Solanky
f413dc62cd Upgrade Obsidian plugin dependencies. Add package lock file for it
Add it to bump_version script as well.
2024-06-22 17:38:52 +05:30
Debanjum Singh Solanky
e42111a8af Fix bump_version.sh to commit, clean-up after desktop app version bump 2023-12-22 21:42:03 +05:30
Debanjum Singh Solanky
96c0b21285 Sync desktop app package.json with other Khoj clients metadata
- Make `bump_version.sh' script set version for the Khoj desktop app too
- Sync Khoj desktop app authors, license, description and version with
  the other interfaces and server
- Update description in packages metadata to match project subtitle on Github
2023-10-13 20:43:55 -07:00
Debanjum Singh Solanky
bcc0bed9db Upgrade bump_version script to handle release and post-release commit
- Updates version in khoj.el and Obsidian manifest, package, versions
  json files under interface and project root
- Create and tag release commit with updated files
- Creates commit with post-release version upgrade in files
- Use flags to specify whether to create a release or post-release commit
2023-03-10 15:23:17 -06:00
Debanjum Singh Solanky
fada617faa Fix TOC links, Add how to auto start Khoj server to Readme
Rename tools directory to more standard scripts directory
2023-02-18 23:51:02 -06:00