Add support for llms.txt to pdoc

[salcc]

Problem Description

Developers are increasingly using AI coding assistants powered by LLMs (large language models) to write code, debug, and understand libraries. However, LLMs face two significant limitations:

• Their knowledge is often outdated, missing recent documentation changes.
• Context windows are too small to process entire API documentation sites during inference time.

As a result, LLMs often generate incorrect code, suggest deprecated APIs, or miss newer, more appropriate functionalities. This creates additional work for developers who must verify and correct AI-generated suggestions.

Proposal

Implement support for the llms.txt specification in pdoc by generating:

1. An /llms.txt file at the root of documentation
2. Markdown versions of all documentation pages (.md extension)

This would make pdoc-generated documentation more accessible to AI coding assistants through RAG approaches, enabling developers to get accurate, up-to-date help.

Alternatives

A related approach to making pdoc-generated docs more AI-friendly is discussed in #761. While that focuses on creating a single document for LLM context, the llms.txt approach provides a standardized way to organize and link to documentation specifically designed for LLM consumption.

[mhils]

I think it should be fairly straightforward to build something like this based on pdoc, see https://pdoc.dev/docs/pdoc.html#using-pdoc-as-a-library! 😃 I don't wan't this in pdoc itself because I feel it's scope creep and I don't want additional maintenance burden. But it sounds like a really cool standalone project!