# Transforming a corpus of 7,000 pages into living knowledge

## From 7,000 pages to daily insights

How can a non-profit turn 7,000 pages of articles—a vast sea of text—into content that is alive, readable, and meaningful every day?

<div class="not-prose grid grid-cols-3 gap-3 my-6">
  <div class="rounded-xl border border-gray-200 dark:border-slate-800 bg-white dark:bg-slate-900/60 p-4 text-center">
    <div class="text-xl sm:text-2xl font-bold text-gray-900 dark:text-white">1,800</div>
    <div class="text-xs uppercase tracking-wider text-gray-500 dark:text-slate-400 font-semibold mt-1">Oral teachings</div>
  </div>
  <div class="rounded-xl border border-gray-200 dark:border-slate-800 bg-white dark:bg-slate-900/60 p-4 text-center">
    <div class="text-xl sm:text-2xl font-bold text-gray-900 dark:text-white">1.8M</div>
    <div class="text-xs uppercase tracking-wider text-gray-500 dark:text-slate-400 font-semibold mt-1">Words (1999–present)</div>
  </div>
  <div class="rounded-xl border border-gray-200 dark:border-slate-800 bg-white dark:bg-slate-900/60 p-4 text-center">
    <div class="text-xl sm:text-2xl font-bold text-gray-900 dark:text-white">7,000</div>
    <div class="text-xs uppercase tracking-wider text-gray-500 dark:text-slate-400 font-semibold mt-1">Pages in archive</div>
  </div>
</div>

This project extends a broader effort to **leverage a massive corpus into a discovery system** that makes exploration smarter, more meaningful, and deeply human. Instead of letting a monumental archive gather digital dust, the goal is to let it *speak again*: one insight, one quote, one topic at a time. The technical workflow behind building this corpus is described in [converting Word files to SEO-optimized web pages with AI](https://redaction-technique.org/turn-word-files-seo-optimized-web-pages-ai).

The transformation from dormant files to an active editorial system follows a clear pipeline:

## Exploring audience concerns through Q/A sessions

To better understand the interests and hesitations of different audience segments, we ran **exploratory Q/A sessions with ChatGPT**. These sessions helped surface recurring questions from readers, which guide content curation and engagement strategies.

For example, when querying:

> **Example exploratory query:** "What are 25- to 35-year-olds most concerned about when it comes to mindfulness?"

The responses highlighted three primary areas of hesitation:

<div class="not-prose grid gap-4 sm:grid-cols-3 my-6">
  <ConceptCard title="1. Time & consistency" badge="Concern">
    Many feel their schedule is too crowded or erratic to sustain regular practice, creating guilt or abandonment before habits take root.
  </ConceptCard>
  <ConceptCard title="2. Authenticity" badge="Concern">
    Skepticism about commercialization, trendy app-based wellness marketing, and shallow interpretations stripped of depth.
  </ConceptCard>
  <ConceptCard title="3. Effectiveness & safety" badge="Concern">
    Apprehension that practice might be ineffective, or worry that silent introspection could inadvertently unearth difficult emotions.
  </ConceptCard>
</div>

The non-profit operates with a very limited budget and cannot afford formal commercial market research. To make well-informed decisions, it relies on ChatGPT as a **cost-effective research tool** for exploring audience concerns, identifying trends, and guiding content strategy based on publicly available information.

To keep the system balanced and transparent, automation is applied across four distinct roles:

<div class="not-prose grid gap-4 sm:grid-cols-2 my-6">
  <ConceptCard title="Audience exploration" badge="Public web">
    Scan public forums and thematic discussions to surface common questions, hesitations, and recurring concerns without expensive consulting studies.
  </ConceptCard>
  <ConceptCard title="Corpus analysis" badge="Metadata & tags">
    Parse 1,800 Markdown files to assign domain tags, semantic scores, and cross-references across the collection.
  </ConceptCard>
  <ConceptCard title="Contextual retrieval" badge="Search & match">
    Query the scored archive to match identified audience questions with relevant historical articles and specific quotes.
  </ConceptCard>
  <ConceptCard title="Editorial recomposition" badge="Multi-format">
    Assist human editors in assembling daily articles, 200-page thematic digests, and micro-blogging quotes from the same source files.
  </ConceptCard>
</div>

> **Human-in-the-loop principle:** Automation serves as an instrument for attention, not distraction. AI helps triage a vast archive that no single editor could re-read in full; human curators decide which insights are genuine, faithful, and worth publishing.

---

## Curating content and quotes from identified topics

Once the system has identified key topics within the corpus, the next step is to **leverage those insights for discovery and engagement**.

For each topic, the workflow involves:

* **Querying the corpus** to surface the most relevant articles.
* **Curating these articles** to make them more visible on the website and to compile them into **themed digests** for deeper reading.
* **Extracting quotes** from these articles that resonate strongly with each topic, ready to be shared **daily on micro-blogging platforms**.

This process transforms the archive from a static repository into a **living, thematic ecosystem**, where content is both discoverable and shareable. Readers can explore topics in depth, while daily quotes keep the conversation active and ongoing, bridging the gap between long-form articles and real-time engagement.

---

## From files to flow

The foundation is a structured corpus of ~1,800 oral teachings (1.8 million words spanning from 1999 to present) in `.mdx` files with YAML frontmatter: **the single source of truth**.

Before any automated processing can help, the source data must be **clean and consistent**: duplicates removed, formats standardized, and terminology aligned. This ensures a high-quality foundation for metadata generation, indexing, and content enrichment:

From that clean source, three parallel publishing streams operate:

<div class="not-prose grid gap-4 sm:grid-cols-3 my-6">
  <ConceptCard title="Daily web publishing" badge="Website">
    Continuous daily publishing on <code>zen-deshimaru.com</code> (one article per day), sustaining an ongoing rhythm for over five years.
  </ConceptCard>
  <ConceptCard title="Thematic print digests" badge="Booklets">
    Over 20 seasonal A4 two-column booklets (200 pages each) compiling focused selections for in-person retreats.
  </ConceptCard>
  <ConceptCard title="Master index volume" badge="400 pages">
    A comprehensive master reference combining <code>index-terms.json</code> occurrences and AI-generated definitions from <code>index-glossary.json</code>.
  </ConceptCard>
</div>

---

## Scoring, metadata, and thematic discovery

Metadata transforms the corpus into a **discoverable and analyzable knowledge base**. Automated workflows assist in reading each article, identifying **topics, tags, and relevance scores**, and maintaining consistency across the entire collection. For a detailed look at how AI scores and tags individual Markdown articles, see [transforming meditation class transcripts into an AI-powered discovery service](https://redaction-technique.org/transforming-meditation-classes-ai-discovery).

The metadata layer performs three distinct functions:

- **Topic categorization:** automatically labeling content by domain or focus.
- **Scoring and ranking:** assigning *interest scores* to highlight resonance and contextual depth.
- **Content enrichment:** adding summaries, cross-references, and definitions for technique-related terms.

A Python-based workflow evaluates every article, assigning **semantic scores** that reflect each topic’s presence and intensity. The result is a **harmonized dataset**, where every page carries structured metadata, ready for discovery, filtering, or print curation.

> **Triage, not truth:** An `interest_score` of 10 is a model's judgment, produced in the confident register models always use: it isn't a measurement, and it has had no contact with an actual reader's interest. The same holds for the "ten most common questions" scanned from forums and the resonance ranking on quotes: useful heuristics for *where to look first*, not evidence of what the audience actually wants. They're a way to triage 7,000 pages no human could read in full, which is genuinely valuable—but triage, not truth. Let the model sort the pile; let a person who knows the material decide what rises from it.

---

## From AI-scored digests to thematic discovery dimensions

Once scored and organized, the corpus is ready to be **recomposed** into focused digests: each a lens for re-seeing the same field of knowledge from new angles.

> **One source, multiple dimensions:** The same 1,800 articles can be sliced by pedagogical topic, historical chronology, or audience dilemma. Each thematic digest is a different lens on a single coherent archive, creating tailored reading paths without fragmenting the underlying data.

These digests reveal different dimensions of the corpus, guiding readers to explore content thoughtfully and meaningfully.

---

## From quotes to conversations

The best quotes don’t stay confined to pages: they move. Through **micro-blogging**, daily publishing, and social curation, the project opens channels for dialogue: the corpus becomes a conversation.

Each quote, ranked by resonance and context, is shared not as static text but as a **living signal**: bridging the long-form archive and the fast-moving web. Automated systems support this process by identifying patterns, key sentences, and recurring motifs, ensuring the most meaningful insights circulate outward.

---

## Toward a living, AI-powered knowledge archive

This is more than content recycling; it’s **content renewal**. A living editorial process transforms static archives into **intelligent, evolving knowledge**.

Readers can:
- **Discover** articles by topic, relevance, or context.
- **Explore** cross-linked insights through search or guidance.
- **Read** beautifully designed print anthologies: curated 200-page digests generated from the same dataset.

The goal remains human: to **distill clarity**, **preserve depth**, and **foster presence**. Automation serves as an instrument for attention, not distraction: a tool to bring the archive to life again.

What emerges when archives breathe again isn’t noise: it’s continuity. A flow of attention, insight, and care that transforms reading into renewal.

## Summing up

## External sources

- [Working with a large text corpus](https://en.wikipedia.org/wiki/Text_corpus)
- [Curation as the editorial layer](https://en.wikipedia.org/wiki/Content_curation)
- [The LLM doing scoring and Q/A](https://platform.openai.com/docs)

<small>*Hero image: ["Library & Reading Room - Rand Club"](https://www.flickr.com/photos/andryn2006/8272268557) by [Andrew Moore](https://www.flickr.com/photos/andryn2006/), licensed under [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0/).*</small>

---

Source: https://redaction-technique.org/transforming-corpus-ai-living-knowledge