Generating llms.txt
llms.txt is a special Markdown text file placed in the site root and used by AI systems for improved page navigation. Its main purpose is to help correctly understand the site's structure and meaning, highlight key information, and interpret it correctly when generating responses.
llms-full.txt is a special Markdown text file placed in the site root that contains the full text content of all site pages for quick content retrieval by AI systems.
Diplodoc supports generating llms.txt and llms-full.txt files for documentation: you need to add the llms: enabled: true parameter to .yfm or run the build with the --llms parameter.
Structure of the generated llms.txt file:
# Project title
> Description from llms.description
Additional information from llms.details.
## Documentation
- [Article 1 title](article-1-link)
- [Article 2 title](article-2-link): description from article 2 metadata
- [Article 3 title](article-3-link): description from article 3 metadata
...
---
For more comprehensive documentation, see [llms-full.txt](llms-full.txt)
Features:
- for each toc.yaml, a separate pair of llms.txt / llms-full.txt files is generated;
llms.descriptionfromtoc.yamlis added as the project description; if omitted, the value from.yfmis used;llms.detailsadds Markdown paragraphs and lists after the description and before## Documentation; these details are not added to llms-full.txt;- the description of each article's content for llms.txt is taken from the description field in the page metadata;
- in llms-full.txt, to reduce file size,
<style>and<script>tags are stripped along with all their content, and svg images are not inlined.
The link text for an article in llms.txt is selected from the first non-empty value in the following order:
- The
titlefield in the article metadata (meta.title). In a Markdown file, set it at the beginning of the file between---lines. - The first level-one heading (H1) in the article. It is determined after variable substitution and include expansion, respecting content visibility for human readers. For a leading page, the page's own
titlefield is used. - The
namefield of the corresponding item in thetoc.yamltable of contents.
Values containing only whitespace are skipped. If none of these values is set, the article's metadata description is used, followed by the path from the TOC item's href.
Per-TOC settings
Set project-wide defaults for all tables of contents in .yfm:
llms:
enabled: true
description: Product documentation.
details: |
Start with the installation guide.
- Examples assume the latest product version.
url: https://example.com/llms.txt
Override description, details, and url at the root of each toc.yaml. For example, in en/toc.yaml:
title: English documentation
llms:
description: Guides and API reference in English.
details: |
Read the guide before using the API reference.
url: https://example.com/en/llms.txt
items:
- name: Getting started
href: index.md
description is the short summary at the beginning of llms.txt. details is additional Markdown content. These are free-form paragraphs and lists before the link sections, rather than a separate ## Optional section.
Each field is resolved independently:
- An omitted field in
toc.yamluses the corresponding value from.yfm. For example, you can set only the English description and keep the project-widedetailsandurl. - An empty string in
description: ''ordetails: ''removes that text, even when.yfmprovides a default. - An empty string in
url: ''clears the project-wide URL. When generation is enabled, pages link to the llms.txt for their own TOC. When generation is disabled, no link is added.
Use an absolute URL or an empty string for url. All three fields accept strings only. Generation enablement (enabled) and the size limit (llmsFullMaxSize) can only be set in .yfm or build command options.
A separate nested TOC uses its own settings and .yfm defaults without inheriting its parent TOC's settings. A TOC connected through an include contributes entries to the containing TOC without overriding its LLMS settings.
The nearest TOC determines a page's llms.url. The link is added to the article's Markdown companion and to the static HTML build as rel="describedby". The description and details fields in toc.yaml are also extracted for translation as Markdown; the URL is not translated.
When building with --watch, changes to these fields in toc.yaml take effect without restarting. Summary or details changes update llms.txt; URL changes update links on affected pages. Restart the build after changing .yfm.
Configuring UTF-8 when publishing
When publishing llms.txt and llms-full.txt, configure the web server to explicitly declare UTF-8 in the Content-Type HTTP header. Otherwise, a client may misidentify the encoding and display Cyrillic and other non-ASCII characters incorrectly.
If nginx serves your documentation, add charset utf-8; to the existing server or location block that serves the documentation root. For example, for documentation at https://example.com/docs/:
location /docs/ {
alias /var/www/documentation/;
charset utf-8;
}
Here, /var/www/documentation/ is the directory containing the build output. Replace the path and the /docs/ URL prefix with your own values. The setting must apply to both files, including those in nested sections with a separate toc.yaml. If a more specific location block serves them, check its configuration as well.
The expected MIME type for .txt files is text/plain. By default, it is included in charset_types, the list of types to which nginx applies the charset setting. If you override MIME types or charset_types, make sure the setting applies to responses serving these files.
After checking the configuration and reloading nginx, check the headers of both files, replacing the domain and path with your documentation URL:
curl -I https://example.com/docs/llms.txt
curl -I https://example.com/docs/llms-full.txt
Both responses should include this header:
Content-Type: text/plain; charset=utf-8
Open both files in a browser and check that Cyrillic characters are displayed correctly.
Limiting the size of llms-full.txt
For projects with many articles, the llms-full.txt file may become too large for AI systems to read. To set the maximum allowed size of the generated file, you can specify a limit in the llms: llmsFullMaxSize parameter in .yfm.
The parameter is processed as follows: if adding the next article to llms-full.txt causes the final file size to exceed the specified limit, filling of llms-full.txt stops and build code YFM022 is returned (level INFO, can be overridden via yfmlint).
Connecting an external llms.txt
If the documentation is part of a site that already has its own llms.txt file, you can specify a link to it in the documentation articles using the llms: url parameter:
llms:
url: https://example.com/llms.txt
Important: to specify an external file, it is not necessary to enable llms.txt generation in the documentation.
You can override the link per TOC with llms.url in toc.yaml. This override also works when generation is disabled.