Chat with your TTRPG books
Table of Content
⚠️ Before you feed books to a chatbot: I do in no way advocate handing LLM providers your data as training material. Not your own, and certainly not somebody else’s copyrighted material. I run all of this on a paid account with model training explicitly disabled. If you want to learn more, Mozilla has a good explainer on why you should opt out and how to do it for the popular chatbots (point 4, but the rest are well worth a read too). If you can’t turn training off, please don’t do this. And if you don’t want to trust a provider at all, a local model via Ollama never sends your data anywhere.
And a second warning, of a different kind, which I realized only after writing it: this is about making a hobby more efficient which honestly might be a questionable goal by itself. Game designer Soren Johnson wrote that “given the opportunity, players will optimize the fun out of a game”, and I guess that warning counts for GMs too.
So, I’m trying to be clear about what I optimized here, since the reading of the books is the fun part, the part where I have all these ideas of connecting lore and player background, imagining twists and turns. I would never hand that off. What I do want to make easier is finding that one paragraph while 4 players wait for me to say what the world lore is because they asked a somewhat specific question and I don’t know if it is already defined or if that is something everyone or no one in this world outside of some powerful beings might know.
I GM multiple, year-spanning D&D campaigns in published settings. The publishers I get my adventures and settings from usually ship beautiful PDFs. One example is about 1900 pages combined.
And these files are not really searchable in the way I need it while prepping. “Which NPC in this chapter knows about the big secret?” is nothing you can find with a Ctrl+F, right? It might be in a small module, but if your adventure has 400 pages…
So I extracted all of those to Markdown, and, well now I can chat with the books. In this post I try to show how, with just a bit of Python (and some RegEx, I’m sorry).
Why I wanted this
The settings I’m using add whole new worlds but they tend to also extend the core 5.5e rules with custom subclasses and sometimes own subsystems.
An LLM that has the actual book content can usually answer most questions in seconds, and it answers it from your version of the book (that’s important further down!).
Uploading the PDFs directly into chat bots usually does not work well as they tend to be multiple hundred MB in size and also full of beautiful hand-drawn artwork. What the model needs is the text (at least in this case), and that is surprisingly small in comparison: all books together came out at under 6 MB of Markdown.
First try: pdftotext
pdftotext from poppler is a classic tool for this task and depending on your PDFs it might already be all you need. Try it first, it’s literally one command. For my PDFs though, everything came out flat: headings are indistinguishable from body text, paragraphs break at every line, and the decorative text (that looks really nice in the PDF) produces garbage like this at the top of chapters:
a
n
o
p
u
s
s
a
l
g
I think that is a fancy full-page chapter illustration, read letter by letter. The LLM can in theory work with it, but navigating 300+ pages of such text is neither fun nor efficient, and the cherry on top will be clear later as to why we want to read that too!
pymupdf4llm
pymupdf4llm is a wrapper around PyMuPDF made exactly for this.
pip install pymupdf4llm
It tries to detect heading levels from font sizes, tries to join paragraphs properly, can do tables and wraps text it found inside images in HTML comments, which makes the broken, fancy text garbage easy to remove.
A version of the script I used looked a bit like this:
import re
import pymupdf4llm
PIC = re.compile(
r"<!-- Start of picture text -->.*?<!-- End of picture text -->", re.S
)
def clean(text):
# trial and error: check the unfiltered output, add a rule per garbage item
text = PIC.sub("", text) # fancy text garbage
text = text.replace("<br>", " ") # HTML newlines
text = re.sub(r"^!\[\]\(.*?\)$", "", text, flags=re.M) # image placeholders
text = re.sub(r"^-----$", "", text, flags=re.M) # page rules
text = re.sub(r"\n{3,}", "\n\n", text)
return text.strip()
chunks = pymupdf4llm.to_markdown(
"file.pdf", page_chunks=True, show_progress=True
)
parts = []
for chunk in chunks:
body = clean(chunk["text"])
if body:
# keep the PDF page number so the LLM can cite it later
page = chunk["metadata"]["page_number"]
parts.append(f"<!-- PDF page {page} -->\n\n{body}")
with open("file.md", "w", encoding="utf-8") as f:
f.write("\n\n".join(parts))
The clean() function is where your mileage will definitely vary: these rules fit my PDFs, but yours will for sure produce different garbage. Extract a single book or even a few pages only without any cleaning first, check the output and then add a rule for every pattern you want to remove. A few iterations of that and the output gets pretty tidy. Repeat until you are happy.
That page marker comment is kinda sweet, too. page_chunks=True returns one chunk per PDF page instead of one large string and keeps its page number in the metadata. The script writes that number in front of every page which makes the LLM able to see the markers and can tell me “that’s on PDF X page 150”.
Don’t expect too much precision though as there is a difference between PDF pages and the printed page numbers in the PDF. Stuff like covers, table of contents can easily shift them by a few pages. And your LLM can still miscite. Treat the numbers as hints at best.
Performance wise it’s good news too, as all 1900 pages together took about 5 minutes on my machine, and that was WSL2 reading from a Dropbox folder, so one of the worst cases you can find.
Some things just don’t work
Monster and NPC stat blocks lose their layout usually. The ability score tables come out as broken Markdown tables with cells like In |itiat |ive+2. The values are all there and the LLM can usually understand them fine and I don’t mind either way. Stat blocks are the one thing I never look up through the LLM anyway because if I have a monster name, finding it in the book is easy. Offline I usually want to open the book in either case to show off the artwork.
At the table
I dropped the Markdown files into a project in my LLM tool of choice and prep sessions with it. Real questions from my prep, almost none of which can be Ctrl+F’d:
- “We are in this location, give me all monster stat blocks I need to prepare miniatures for.”
- “Which locations in this chapter matter for this one specific character background?”
- “According to the adventure, how much time should the PCs spend in this location?”
- “Which locations in this chapter reference the artifact from the first session?”
And this doesn’t account for the fact that you could have PC backgrounds in there too. Or session recaps!
Answers come back based on your actual books, your data, with pages I can double-check in a pinch. For my games this went from nice gimmick in the beginning to the thing I start every prep session with. And by now I do this for every campaign I run.
Sometimes I can even use it during a session. While the players argue amongst themselves about some plan or NPC (you may know those moments), I can quickly check the lore whether the thing I’ve to improvise contradicts anything.
I’m not a great improviser, and my irrational fear of improvising myself into a corner the setting already has an answer for doesn’t help. Asking the book helps me keep my nerves. Though I am aware that this probably keeps me from becoming a better improviser at the same time… Similarly it works for rule clarifications: faster than checking 3 PDFs while everyone waits. (Although my games all have a house rule that we don’t discuss rules during play it does happen sometimes.)
I have one (for some people obvious) pro-tip: a big PDF easily needs 250k+ tokens as Markdown, which might be more than your model’s context window can handle. But the fix is easy, just split it per chapter! The heading structure from pymupdf4llm makes that easy. Smaller books and single adventures usually fit just fine.
Bonus: Your world is editable now
The chatbot can be one consumer of these files but Markdown is the actual win here. Since the extracted text keeps the heading structure, you can start dissecting the book per chapter and drop it into Obsidian (or any note tool that handles Markdown). So even if you don’t want to use a LLM you now have the whole setting at your fingertips. You can cross-link NPCs, locations whatever with your session notes, searchable (CTRL-F or LLM) fast even without a PDF reader.
And because it’s plain text instead of PDF the setting stops being read-only. It really becomes your world. When your games and sessions create new ideas and changes in a location or the world (if you want to edit quite some text), you edit the file (or files). The queen your players accidentally killed stays dead in your copy. Use the edited version with an LLM and it answers from your world instead of the printed one.