Skip to main content

Markdown Preprocessing for Efficient Embeddings

Introduction​

Markdown files are widely used for documentation and content structuring, especially in AI applications like VAKStudio. When processing markdown files to create embeddings for a Vector Database (VectorDB), the system splits the content at logical points, such as headings, to generate vector representations of the text.

Efficient markdown structuring plays a crucial role in reducing token usage, improving query results, and lowering the cost of embedding generation. This guide will walk you through best practices for creating markdown files that are optimized for preprocessing, focusing on how the VAKStudio system handles text splitting.

How Text Splitting Works in VAKStudio​

When you upload a markdown file to VAKStudio for embedding generation, the system processes it by splitting the text at heading tags (H1-H6). Each section of the text, defined by the headings, is treated as an independent chunk that is converted into embeddings.

Here’s how the system handles different heading levels:

  • H1: The top-level heading, often representing the main topic or chapter of a document.
  • H2-H6: Subheadings that represent increasingly granular sections of content.

By organizing your content using appropriate headings, the system can break the text into smaller, relevant parts, making the embedding generation process more efficient.

Best Practices for Markdown Preprocessing​

1. Use Headings to Split Content at Logical Points​

Headings (H1-H6) act as natural breakpoints for the system to split the text. Use headings strategically to create well-defined sections that focus on specific subtopics.

Example of Good Heading Usage:​

# Main Topic: Introduction to AI

## Section 1: What is AI?
Artificial Intelligence (AI) refers to the simulation of human intelligence in machines.

### Subsection 1.1: Types of AI
There are two main types of AI: Narrow AI and General AI.

## Section 2: How AI Works
AI systems use algorithms and models to process data and make decisions.

In this example, the system will split the text at H1, H2, and H3 tags, creating smaller, manageable chunks for embedding.

2. Keep Paragraphs Between Headings Short​

Long paragraphs can increase token usage. By keeping paragraphs short and focused, you minimize the number of tokens the system needs to process.

Example of Concise Paragraphs:

## Section 1: Introduction to Neural Networks
Neural networks are a subset of machine learning that uses layers of neurons to process data.

### Subsection 1.1: Structure of a Neural Network
A neural network consists of an input layer, hidden layers, and an output layer.

By keeping paragraphs concise, you help the system focus on the most relevant information, reducing the token count.

Conclusion

Creating well-structured markdown files is critical for efficient embedding generation in VAKStudio. By following the best practices outlined above, you can ensure that your markdown files are optimized for text splitting, leading to lower token usage, faster processing, and better results in your VectorDB.

Key Takeaways:

  • Use headings (H1-H6) to split content at logical points.
  • Keep paragraphs short and focused.
  • Utilize bullet points and lists for concise information.
  • Reduce unnecessary content to optimize token usage and improve embedding quality.