Honeycomb

What it is#

Honeycomb is a GIF searcher built with Antfly that currently indexes over 105k GIFs based on their semantic content. You can try it out at honeycomb.rowan.earth.

Here's a semantic search for GIFs about hobbits with the "playful" category:

Browsing Twin Peaks-related GIFs:

How to search GIFs?#

Many of the Tumblr GIFs did come with short captions, but they were generally extremely vague — they didn't even mention things like celebrity names. If someone searches for President Obama they should find all the President Obama GIFs!

To get the semantic content of GIFs into an index that we can search with natural text queries. CLIP could embed individual frames, but would lose the motion between frames. Video embedders are designed for this sort of use case, except that they tend to expect much longer videos and all the GIFs would need to be converted to a video file first.

In the end, I settled on getting a VLM (vision language model, which can see images as well as process text, which is pretty much any frontier model these days). I had it describe what is happening in the GIF when given a number of the frames, and then embedded that. This actually worked better than I expected:

• Even multi-shot GIFs were generally accurately described. I found 5 frames from the GIFs was sufficient to capture
• We get a human-readable structured output, which we can use for other things besides semantic search like filterable metadata (more on this below)

There are some disadvantages, of course:

• If the VLM doesn't notice something, it becomes invisible to searches. So you need to iterate on your prompt, because that determines what will be more likely and less likely to be noticed
• Movement-heavy or fast GIFs will be harder to interpret. Quick cuts, unusual motion (like floating underwater), or where the main action takes place across only a few frames of a longer GIF, will be less likely for the VLM to accurately describe

Ingesting the GIFs#

I built Honeycomb to be extensible and support adding more GIF datasets easily, because the bulk of GIFs comes from one single dataset: Tumblr GIFs, which is a mid-2010s index of 100k GIFs from the microblogging site (created by Li et al). Initially I was just retrieving the GIFs from Tumblr.com itself and using those original ones but decided it would be kinder to their servers to host them ourselves. So we saved them to a Cloudflare bucket for the next stages, and serving the final site.

Once we had the GIFs saved, we needed to have each one described. We extracted 5 evenly-spaced frames from each GIF and fed it into Gemini 2.5 Flash Lite with the following prompt:

You are analyzing multiple frames extracted from an animated GIF. The frames are shown in chronological order.

Analyze the full sequence of action across all frames and return a JSON object with these fields:

Required fields:
- "literal": Factual description of the complete action/sequence (1-3 sentences, describe what happens from start to finish)
- "source": Your best guess at where this is from — movie title, TV show, meme name, video game, news event, YouTube/TikTok trend, etc. Be specific (e.g., "Spy Kids (2001)" not just "movie"). Use "unknown" only if you genuinely cannot identify it.
- "mood": Emotional tone or vibe (e.g., "funny", "wholesome", "chaotic", "satisfying", "dramatic", "sarcastic", "glib"). Pay attention to positive, negative, and neutral affects, and recognize that GIFs can be mixtures of different moods. Don't neglect to describe the human id when it is present: violence, sexuality, joy, defeat; these are all present in GIFs
- "context": When someone might use this GIF in conversation (e.g., "reaction to unexpectedly good news")
- "tags": Array of 5-10 searchable keywords useful for finding these types of GIFs (kebab case, no spaces). All explicit sexual content should receive a tag of adult-content.
- "rating": i.e. G, PG, PG-13, R, X. The rating is ONLY for the content in the GIF itself. All explicit sexual content should receive a rating of X.
- "names": Array of named characters, recognizable people (e.g., ["Michael Scott", "Dwight Schrute"]), meme formats (e.g., ["Surprised Pikachu"]). Empty array if none recognized.
- "visual_style": Brief description of the visual aesthetic (e.g., "live action", "anime", "pixel art", "3D animation", "screen recording")

Respond with only the JSON object, no markdown formatting or extra text.

Some notes about the prompt:

• Separating out into multiple categories helped immensely in capturing a wider variety of what the GIF meant
• Earlier versions of this prompt often failed to capture subtle negative emotions (sarcasm, anger, etc) so I tried asking for it more explicitly

The resulting JSON was used to populate the Antfly table, with each element getting its own field as well as the concatenated text becoming the field that we then embedded with a text embedding model (bge-small-en-v1.5). Loading the data didn't take long, but it took a few hours before Antfly Inference had finished backfilling all 106k text entries into the embedding space.

But which model?#

Before I ran this on the entire dataset, I needed to choose the final model I'd be using. I wanted to use a model in the Gemini family because I'm already familiar with them and know they're generally useful. But which one? The range was anywhere from Gemini 3.1 Pro down to Gemma 4b. It turned out that I didn't need comprehensive evals, I just needed to compare their performance on the

make compare-models N=25
  MODELS="gemini-3.1-pro-preview,gemini-3.1-flash-lite-preview,gemini-2.5-flash-lite,openrouter:google/gemma-3-4b-it"

That outputs a markdown file that lets me compare the text descriptions of four different models at varying price points. Here's what it looks like when I serve the file with grip:

Just browsing through the outputs, I noticed a few things:

• Gemma 4b was significantly worse than any of the Gemini models, although it was still totally usable
• Gemini 3.1 Pro was ~50x more expensive than 2.5 Flash Lite, and while it was maybe a bit better, $1 per hundred GIFs did not seem worth it!

Making a nice front end#

At this point it was easy to get Claude to spin up a simple front end that showed random GIFs on the landing page and let the user do searches.

Because we extracted text as well as an embedding of the GIF content, we can get hybrid search for free. For example, if I wanted to remove the GIFs of Ein the anime corgi from my search results:

I could filter them out by adding -tag:anime:

Adding a "mood" selector#

Another way that having a structured text description of each GIF came in handy was being able to make a new "mood selector" just by categorizing the mood field for each GIF.

Here's a search for "panda" with "wholesome" selected:

And now with "chaotic":

As you can see, the number of GIFs featuring pandas destroying offices jumps significantly.

Content filtering#

Hopefully this doesn't come as a shock to anyone reading this, but there is porn on the Internet and in particular Tumblr, even today many years after Tumblr cracked down on it. Luckily, there were barely a dozen or two GIFs that needed to be filtered out. Less luckily, the VLM did not always succeed in identifying when the GIF was actually porn. It rated a few of them R and one even PG-13! The process here wasn't systematic and involved just combing through the database for any content that could be semantically close enough to porn while being mistaken for something else, and blocking them. Even with good vision models, content moderation is a hard problem.

Interestingly, there were about 500 GIFs left over at the end of the run that Gemini consistently refused to engage with. Almost all of them appeared to be completely harmless, so I used Qwen3-VL-235B on the rest of them.

What next?#

Evals. Thus far, we've treated evals as one-off and scattershot. We could do much more systematic things like looking for contents of GIFs that the description pipeline consistently misses, building a small eval set off of that, and iterating on the prompt with a small subset of GIFs.

More GIFs. There are a lot more GIFs on the Internet.

Video embeddings. Recent models like Gemini Embedding 2 natively support up to 120 seconds of video, it would be interesting to see if it performs much better on a Honeycomb eval set!

Using Antfly Inference for the description phase. Antfly Inference can actually run a number of VLM models locally; I actually ran the Gemma pipeline through Antfly Inference before going with Gemini. And it natively supports CLIP so we could add a second image embedding index and search on both vector indexes at once.