feat: add Markdown formatter

[yordis]

Signed-off-by: Yordis Prieto yordis.prieto@gmail.com

[github-actions]

📦 Docs artifacts are ready: https://github.com/elixir-lang/ex_doc/actions/runs/20142372831/artifacts/4841057828

[yordis]

@josevalim any thoughts here thus far?

[josevalim]

Sorry, a bit busy with Elixir v1.19-rc and today I was out of focus. It is in my inbox and I will review it as soon as I can.

[leandrocp]

Hey @yordis that's amazing! Let me share some ideas that could be useful.

I was inspecting the generated Markdown files and thinking if we could apply some changes. See the differences here https://gist.github.com/leandrocp/ee4f0ba8325b410b8650ccd26b9b2351 (CompiledWithDocs.md vs CompiledWithDocs_PROPOSAL.md)

• Use frontmatter block to describe global values/notes
• Use the format "Summary / Functions" similar to HTML pages (so Functions become a level 2 heading)
• Include source links
• Reorganize metadata, for eg: add (deprecated) and doc in summary
• Remove links to hexdocs.pm because 1) it breaks the content and I guess that would make it harder for LLMs to parse; 2) I'm not sure it should link to html pages

You can see some examples on https://shopify.dev/docs/api/liquid/basics.md and https://vercel.com/docs/rest-api/reference/sdk.md

[yordis]

@leandrocp thanks for the help, about the formatting, I do not have any strong opinions of the final output, I will leave it to @josevalim and you to decide on that front, I can adjust it

[josevalim]

Just a heads up I am on holidays. This is still on top of my list but I cannot promise anything until I am back (next Wednesday). Sorry!

[leandrocp]

Hey @yordis no strong opinions either, just trying to figure out a format that presents all the info.

Regarding links, just to be clear it should contain links eventually but I believe that HexDocs must define how to serve Markdown pages first, for eg something like https://hexdocs.pm/elixir/1.18.4/Enum.md

[josevalim]

Some additional context: ExDoc is not tied to HexDocs and vice versa. We should design links with the assumptions it is served by any static host. I am thinking just using Enum.md with an anchor, same as html, is fine though? The anchor won’t actually work, but at least it is some context?

[leandrocp]

just using Enum.md with an anchor, same as html, is fine though?

Yeah it seems appropriate.

---

Shopify docs links to "canonical" URLs, for eg in https://shopify.dev/docs/api/liquid/tags/form you can see a bunch of anchor links, but in the .md docs (https://shopify.dev/docs/api/liquid/tags/form.md) they are rendered as [`cart`](https://shopify.dev/docs/api/liquid/tags/form#form-cart)

Vercel is not much different, they include relative links inside .md docs, for eg [Deployments Automation](/examples/deployments-automation)

Claude use relative links with anchors, for eg in https://docs.claude.com/en/docs/build-with-claude/prompt-caching you can see a link <a href="#pricing" class="link">at additional cost</a> which is rendered as [at additional cost](#pricing) in the .md docs (https://docs.claude.com/en/docs/build-with-claude/prompt-caching.md)

[josevalim]

I am back. My goal is to release the new hexdocs.pm, review #2055, and then focus on this.

[yordis]

I am around, once you folks aligned, please share a action-driven feedback, I am can put the work on

[yordis]

@josevalim could you give a second pass to the code review 🙏🏻

[yordis]

@eksperimental should I change the tests to do full equality of the content? Maybe read them from a file snapshot or somthing, I am having a hard time trying to follow the existing tests setup, I can't tell the final structure that easy but I did not want to change it that much neither

[eksperimental]

@eksperimental should I change the tests to do full equality of the content? Maybe read them from a file snapshot or somthing, I am having a hard time trying to follow the existing tests setup, I can't tell the final structure that easy but I did not want to change it that much neither

I would focus on the final structure first and then add/update tests accordingly.

[yordis]

What should I do about that .build-markdown file? Should that be merged into the existing .build, since I did not see .epub in the .build I guessed it would be a different one

[eksperimental]

What should I do about that .build-markdown file? Should that be merged into the existing .build, since I did not see .epub in the .build I guessed it would be a different one

I think having a subfolder per format would be wise.

I think having everything living in the same folder is no longer viable.

[yordis]

I think having a subfolder per format would be wise.

I agree with this!

[yordis]

I think having a subfolder per format would be wise.
@eksperimental implementing the multi-dir I just realized, is that just for the .build files, are you thinking on something like,

tree doc 
doc
├── epub
│   └── ExDoc.epub
├── html
│   ├── 404.html
│   ├── admonition.html
│   ├── api-reference.html
│   ├── changelog.html
│   ├── cheatsheet.html
│   ├── dist
│   │   ├── html-elixir-VYWJUHZE.css
│   │   ├── html-HBZYRXZS.js
│   │   ├── lato-latin-400-normal-W7754I4D.woff2
│   │   ├── lato-latin-700-normal-2XVSBPG4.woff2
│   │   ├── lato-latin-ext-400-normal-N27NCBWW.woff2
│   │   ├── lato-latin-ext-700-normal-Q2L5DVMW.woff2
│   │   ├── remixicon-QPNJX265.woff2
│   │   ├── search_data-B202F4E6.js
│   │   └── sidebar_items-ED2F9D7B.js
│   ├── ExDoc.Markdown.Earmark.html
│   ├── ExDoc.Markdown.html
│   ├── index.html
│   ├── Mix.Tasks.Docs.html
│   ├── readme.html
│   └── search.html
└── markdown
    ├── ExDoc.Markdown.Earmark.md
    ├── ExDoc.Markdown.md
    ├── index.md
    ├── llms.txt
    └── Mix.Tasks.Docs.md

[eksperimental]

I think that's fine. Having it as before epub and html in the main folder and only markdown in it's own sub-folder was not consistent. As more formatters are added, having everything in one folder it will become unmanageable.

By having sub-folders give the option to the user to easily create a zip file per formatter for users to download the docs.

Maybe having an option to user sub-folders could be desirable to not to break compatibility of apps using this as hexdocs.pm.

Those are my 2 cents.

[yordis]

not to break compatibility of apps using this as hexdocs.pm.

The reason I asked is because of hexdocs and how people could "deploy" these artifacts, having the subdir for pub or html would introduce breaking change, I am guessing since it will require to add /html /pub if a simply static hosting is in place.

So, should I duplicate the pub and html and keep them as well in the subdirs, or should I just add the markdown?

[eksperimental]

Maybe have an optional flag to use sub-folders for now, and not having it the set by default. Later on this can be set by default in v1.0 or remove it completely and make it the default (and only) behavior.

Also I think we should ask @josevalim's opinion.

[yordis]

@eksperimental I will wait for @josevalim and hopefully after that we can get a clear path forward, eitherway I can get it done.

[josevalim]

Sorry for the delay @yordis. I think this could be much simpler by keeping the original markdown and using this structure for modules:

# Module

# fun/3

*annotations*

Source: ...

Signatures:
...

Specs:
...

<%= original_markdown %>

# t:type/1

*annotations*

Source: ...

<%= original_markdown %>

We need to later have a separate discussion if it should be a single or multiple files, but for now, that's the simplest way to go.

[yordis]

@josevalim are you sure about # fun/3 instead of ## fun/3 (two #)? Just want to confirm it was intended.

[josevalim]

Yes, because users can use h2.

[yordis]

@josevalim what should do with .build and what should I do with the markdown files as in where to put them?

 tree -a doc
doc
├── .build
├── 404.html
├── admonition.html
├── api-reference.html
├── changelog.html
├── cheatsheet.html
├── dist
│   ├── html-elixir-VYWJUHZE.css
│   ├── html-HBZYRXZS.js
│   ├── lato-latin-400-normal-W7754I4D.woff2
│   ├── lato-latin-700-normal-2XVSBPG4.woff2
│   ├── lato-latin-ext-400-normal-N27NCBWW.woff2
│   ├── lato-latin-ext-700-normal-Q2L5DVMW.woff2
│   ├── remixicon-QPNJX265.woff2
│   ├── search_data-B202F4E6.js
│   └── sidebar_items-ED2F9D7B.js
├── ExDoc.epub
├── ExDoc.Markdown.Earmark.html
├── ExDoc.Markdown.html
├── index.html
├── markdown
│   ├── .build
│   ├── ExDoc.Markdown.Earmark.md
│   ├── ExDoc.Markdown.md
│   ├── index.md
│   ├── llms.txt
│   └── Mix.Tasks.Docs.md
├── Mix.Tasks.Docs.html
├── readme.html
└── search.html

3 directories, 28 files

[josevalim]

@josevalim what should do with .build and what should I do with the markdown files as in where to put them?

We should keep the build for markdown. Just the one for epub that was not part of this PR!

[yordis]

We should keep the build for markdown. Just the one for epub that was not part of this PR!

Then, it is ready for CR

[josevalim]

I think we are almost there, we just need to remove the to_markdown function and its tests, and remove the changes to the autolinker, cause i think those are not longer necessary either.

[josevalim]

Note to self: we need to check if the original documentation is also in markdown and skip any module where this is not true, but this can be added after merging.