How Chart Library's Screenshot Search Actually Works

The Problem: From Pixels to Patterns

You take a screenshot of a stock chart on TradingView. It's a PNG file — millions of pixels, colors, gridlines, indicators, labels. How do you go from that mess of pixels to finding similar historical patterns?

This is the core technical challenge Chart Library solves. The answer involves computer vision, vector embeddings, and a clever two-model architecture that bridges the gap between images and numerical time series data.

Step 1: The Numerical Embedding Space

Before we can search with screenshots, we need something to search against. Chart Library pre-computes a 384-dimensional embedding vector for every stock, every trading day, across multiple timeframes. These embeddings are computed from actual price and volume data — not images.

The embedding captures the shape of the price action as cumulative percentage change, resampled to a fixed dimension. It also encodes volume patterns, volatility, and VWAP deviation. Think of it as a mathematical fingerprint for each chart pattern.

With 16+ million of these embeddings in a pgvector database with an IVFFlat index, we can find the nearest neighbors to any query vector in about 9 milliseconds.

Step 2: The Vision Model (ViT v4)

The magic is in the bridge: a fine-tuned DINOv2-ViT-B/14 model that converts screenshot images into vectors in the same 384-dimensional space as the numerical embeddings.

DINOv2 is a self-supervised vision transformer from Meta that produces excellent visual features. We fine-tune it with a regression head that maps image features directly to the numerical embedding space. The training data consists of synthetically rendered charts paired with their known numerical embeddings.

The key insight: we don't need the vision model to understand candlesticks or technical analysis. It just needs to output the same vector that the numerical pipeline would produce for the same price action. The similarity search handles the rest.

Step 3: The Search Pipeline

When you upload a screenshot, here's what happens in about 2 seconds:

• The image is processed by the ViT v4 model, producing a 384-dimensional embedding vector.
• The system auto-detects the timeframe from visual cues (number of candles, aspect ratio, etc.).
• The embedding is normalized and searched against the appropriate timeframe partition in the database using L2 (Euclidean) distance.
• The 10 closest matches are retrieved with their full metadata: ticker, date, similarity score.
• Follow-through data (what happened 1, 3, 5, and 10 days later) is computed for each match.
• Claude Haiku generates a plain-English summary of the pattern and its historical implications.

Why Euclidean Distance, Not Cosine?

Most embedding search systems use cosine similarity, which measures the angle between vectors. Chart Library uses Euclidean (L2) distance, which measures the actual distance between points in the embedding space.

The difference matters for chart patterns. Cosine similarity would treat a 2% move and a 20% move as identical if they had the same shape. Euclidean distance distinguishes between them — because magnitude matters when you're trying to predict what happens next. A gradual drift and a sharp spike may have the same "shape" in cosine space, but they produce very different outcomes.

Platform Agnostic by Design

One of the hardest challenges was making the system work with screenshots from any charting platform. TradingView, Thinkorswim, Webull, Interactive Brokers — they all render charts differently. Different colors, different candlestick styles, different background themes.

The DINOv2 backbone helps enormously here because it was pre-trained on massive visual diversity. During fine-tuning, we augment the training data with various chart styles, color schemes, and artifacts. The result: the model focuses on the price pattern structure and ignores cosmetic differences.

Try It Yourself

The best way to understand the system is to use it. Take a screenshot of whatever chart you're looking at right now and upload it to Chart Library. In under 3 seconds, you'll have 10 historical matches with full follow-through data.