Introduction
At Graphon, we believe that bringing structure into your data can go a long way. This is especially true when that structure exploits the natural properties of the problem you are tackling. A key balance exists between giving machine learning and AI systems complete freedom and imposing too much constraints. If you impose too much specific structure, you lose generalizability and the ability to easily scale with data and computation. This is a core insight of Richard Sutton’s Bitter Lesson. An example of this is trying to predefine an entire ontology in a GraphRAG system for every question you might ask it. For a general purpose system, you simply cannot preindex every possible query.
At the same time, if you impose no structure, your system cannot learn effectively. Early researchers trained fully connected neural networks only to find they performed worse than existing non machine learning systems. Later, we saw huge leaps in computer vision when Yann LeCun introduced convolutional neural networks (CNNs). Likewise in natural language processing and sequential data generally, Transformers revolutionized the performance and scalability of sequence-based architectures. Both CNNs and Transformers represent only a subset of what a fully connected neural network can theoretically perform, yet they are far more useful in practice. That is because their built in structure allows learning and processing to happen much more efficiently.
We take these lessons to heart. We design systems with enough structure to process data effectively, but not so much that we sacrifice the core “learning” aspect of machine learning.
Documents already contain useful structure. Chapters, headings, sections, and paragraphs show how ideas relate. A retrieval system can preserve this hierarchy instead of flattening the document into unrelated chunks. Video rarely contains an explicit hierarchy. Our pipeline creates one from the video itself.
Video Processing
Videos present a particularly difficult medium to index and understand. For many memory or RAG systems, developers rely on some combination of naive or intelligent data "chunking", often adding chunk overlaps to avoid cutting off important sequential events. We take a different approach.
For our video processing pipeline, we found that the most effective structure involves splitting low frequency and high frequency data. In a movie, low frequency data answers broad questions like, "What is this movie generally about?" As you move into higher frequencies, the data changes per scene ("What is the setting?", "Who is in this scene?"). At even higher frequencies, you capture individual actions and events. The frequency domain provides a proven framework for analyzing visual content because it forms the basis of most image and video compression algorithms. It also makes intuitive sense. At Graphon, we build a graph over your data, and establishing solid structural components enables us to build higher quality ontologies and relationships. To demonstrate this, we ran over 9 hours of public domain footage from NASA’s US Spacewalk 95 through our video processing pipeline.
Sections and events
Low and high frequency
Our pipeline naturally identifies a hierarchy of distinct "sections" and "events." Sections capture lower frequency components, such as distinguishing command center footage from views inside the space station or outside during the spacewalk. Events within those sections represent higher frequency components, such as an individual step in repairing a mechanical arm. An unsupervised, content driven algorithm determines all sections and events, ensuring the approach generalizes across different video types. We did not specialize our spacewalk results for that specific footage or its genre.
This hierarchy serves as the base layer of the graph. Each section and event becomes a node, linked directly to the levels above and below it. On top of this base, we build higher-level understanding and relationships, both within a single video and across all the files you provide. The graph finds patterns at every level of the hierarchy. In a movie, a theme might span the whole film, a setting might repeat across scenes, and a single action might fit one event. Similar moments that are far apart in time can be linked too. We did not design the hierarchy to create any specific links. It only gives the graph clean blocks to build on. This gives us our target balance between too much structure and too little: we organize the video into sections and events, but we let the graph discover the relationships.
Query
The low-frequency sections and high-frequency events created during ingestion become useful during retrieval. A flat retrieval system treats each video segment as independent. It can return many similar moments from one repetitive scene. These results can fill most of the available evidence slots while adding little new information. The hierarchy lets the query system group results that belong to the same section or event. It can keep the strongest evidence from each group and reserve evidence slots for other parts of the video.