feat: add docusaurus plugin for llmstxt #1570
Conversation
✅ Deploy Preview for oasisprotocol-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
matevz
left a comment
matevz
left a comment
There was a problem hiding this comment.
llms.txt doesn't seem to contain markdowns that are symlinked (oasis-sdk, cli, core, adrs). Can you debug it?
uniyalabhishek
force-pushed
the
uniyalabhishek/feature/llmstxt
branch
from
6d616ed to
e0af0b2
Compare
@matevz polished a lot now. please check it out.
|
uniyalabhishek
force-pushed
the
uniyalabhishek/feature/llmstxt
branch
2 times, most recently
from
00fef7b to
e0a6483
Compare
feat: symlink-aware llmstxt plugin feat: snippet inlining and external directory support refactor: update llmstxt build docs prioritization feat: enhance llms-sym plugin with cycle protection and markdown cleaning feat: enhance plugin with description derivation fix: enhance symlink error logging and image link fix: update docusaurus-plugin-llms version in yarn.lock feat: enhance llms plugin content cleaning, and description extraction
uniyalabhishek
force-pushed
the
uniyalabhishek/feature/llmstxt
branch
from
3ed5e80 to
ded20aa
Compare
matevz
left a comment
matevz
left a comment
There was a problem hiding this comment.
There are some major drawbacks:
- generic docusaurus plugin is not taking the Remark-parsed content as a base. This means we have duplicate code in llms-symlinks.js (almost 600 LOC + relying on a 3rd party plugin!) to support symlinks, for code snippets and cross-repo links.
- chapters hierarchy is not based on sidebars, currently everything is flat
- llms-full.txt also does not contain chapter hierarchy which results in duplicate chapter titles
I would suggest alternative approach in a new PR without duplicate code and 3rd party plugin - generate llms.txt and llms-full.txt inside a Remark plugin. This way we already have code snippets there and working cross-repo links for free as well as ordering and hierarchy based on side bars. The input for a remark plugin is a markdown code and some metadata (already correct filepath etc.). You would then add a chapter entry to llms.txt, shift the headings accordingly in the markdown tree and append it to llms-full.txt.
In the meantime, keep this PR as a backup for training AI models on our Build chapters.
closes: #1315
preview: https://deploy-preview-1570--oasisprotocol-docs.netlify.app/llms.txt
https://deploy-preview-1570--oasisprotocol-docs.netlify.app/llms-full.txt