You don't need LLMs for EVERYTHING

"Just ask (insert any LLM)" is crazy to me. I'm sure you've also seen it, if you go around the saas/entrepreneur subreddit space the most obvious tell is how EVERY homepage looks. Recently the trend has been terminal green with slight glowing elements, and slowly revealing features as you scroll down. This is a bigger issue than the scope of this post, but I will still plant the thought in your head: if everything comes from the same source (even more so if it's all being fed back to it), nothing is unique. It becomes predictable and tacky.

Keytoad turns Reddit content into data. "content" is a broad term, and I use it on purpose to demonstrate the breadth of user inputs, which is very unpredictable and a pain to deal with. The logical solution to this language problem is to use a LANGUAGE MODEL. makes infinite sense. Of course we did that. That entire spine that integrates the LLM into our service is still in the repo, and is one API token away from being used. Why aren't we?

The reasons are unglamorous. Cost, latency, and control. Cost is self-explanatory, you need to pay for every batch of anything you send to the model. Latency is a problem because of caching and readiness, we have a premium tier that offers priority on analysis but that doesn't matter if the LLM takes 90 seconds to chew through a batch. Control was the worst, and still is in the current state of this app. We tried to use two models for this, Gemma 4 and Hy3. Gemma we ran locally, and Hy3 through openrouter (it was free).

Gemma 4 is tiny (in comparison) and we ran it locally. It was confused about anything and everything and we spent genuine chunks of time "perfecting" a prompt to get it in the mood. Didn't work: nothing was consistent, and in an app where you want to track trends you NEED the tokenization to be the same every time. Hy3 was better (thanks to being 10 times the size). While it was able to understand the task, it was even more moody than Gemma. It would be scared of removing keywords, example, when given a list of 400 words and told to remove irrelevant ones and group similar ones, it gave back 396. LITERALLY 4 words removed, no matter how destructive we asked it to be. moody.

What do i mean by moody? look:

What runs instead

The job

"Nvidia announces RTX 5090 price cut"

  -> nvidia
  -> rtx 5090

You want nvidia and rtx 5090. You don't want announces and you definitely don't want cut. It's simple to explain until you try it with thousands of actual user-made posts and get every solution DISMANTLED by them.

Three most painful ones:

"[f]riday plans, anyone?"
"Judge dismisses conspiracy case against Proud Boys | Guardian"
"Why Elden Ring Is The Best Game Of The Year"

The rest of this post is mostly about those.

A tokenizer that knows what a word is

NLTK ships word_tokenize, and we don't use it. We use a regex tokenizer with one deliberate rule:

RegexpTokenizer(r"\w+(?:[-'’]\w+)*")

Hyphens and apostrophes inside a word stay part of the word. Without that, micro-saas becomes micro plus saas, e-bike falls apart (pun maybe intended), and so does every a.i name in any tech subreddit. Possessives get stripped separately, so Biden's counts with Biden.

Then there is [f]riday. Reddit users pun gender and age tags onto the next word constantly, and a tokenizer that splits on brackets hands you riday, which is a word in no language. So one regex runs before tokenization and unglues them.

_GLUED_TAG = re.compile(r"[\[(](\w{1,6})[\])](?=\w)")

The Proud Boys | Guardian case needs something different. Reddit titles carry source attribution after a pipe, and any rule that groups adjacent capitalized words will happily fuse a band of protesters to a newspaper, giving you Proud Boys Guardian. We had a laugh at that. The fix is to tokenize with span_tokenize instead of tokenize, because spans give you character offsets. If anything other than whitespace sits between two tokens, we record a break there, and no phrase is ever allowed to cross one. The pipe becomes a wall. So does a comma, a colon and a full stop.

Capital letters do work a tagger can't

Here is the decision I expected to regret but (ultimately) didn't: completely doing away with part of speech tagging as the thing that groups phrases, and simply using capitalization instead. Reddit is not edited prose, and taggers mislabel brands/names/games in it often enough that we stopped bothering, after several weeks of trying to get it to work mind you.

So the phrase builder walks capitalized words, up to three at a time, and keeps digits in. That keeps Xbox Game Pass together, keeps it distinct from xbox one, and attaches a leading model number to the name that follows it, which is how 7900 XT survives as one thing. The philosophy here was to do the 20% that gets you the 80%.

A run alone isn't enough, though, because of the third title.

The headline problem

Why Elden Ring Is The Best Game Of The Year is capitalized end to end. Capitalization carries no information there, so the phrase rule has nothing to grip. We call this type of post a "headline".

What gives it away is small and reliable: headlines capitalize their stopwords, sentences don't. Count the stopwords present, count how many are capitalized, and if more than half are, you're looking at a headline. Then the keep rule changes. In normal text one uncommon word is enough to keep a capitalized run, but not here. In title case, at least half the words in the run have to be uncommon.

We landed on half by trial and error. Requiring every word to be uncommon threw away real items that contain an ordinary word, so Elden Ring died on ring. Requiring only one word let junk through on the back of a single uncommon token, and Huge Nvidia Announcement became a trending keyword, and that isn't a phrase we consider worthy of taking a spot in the "trending" chart. Too non-specific. "huge" doesn't help you.

"Common" is a number, not a list

I used "common" and "uncommon" several times in this post so far, time i explain what i mean.

A dictionary check is useless here. Inflation and bitcoin are in the dictionary, AND they are "rare" enough in the english language to indicate something. What we want is not "is this English" but "is this so ordinary that it tells me nothing", and wordfreq already does both. Slotted in, bingo bango bongo.

thing       5.7   filler
year        6.0   filler
              --- threshold 4.5 ---
bitcoin     4.2   keep
inflation   4.1   keep
nvidia      3.3   keep

The 4.5 threshold sits around the top three thousand words of everyday English. Phrases get their own, higher threshold of 5.0, because a two word phrase is always rarer than either of its halves: student loans scores 4.4 and gets through, last year scores 5.6 and does not.

That is a threshold I can move. When a keyword shows up that shouldn't, I just run a script that compares its current threshold with the desired one and get a before and after. Trial and error here too, and the fact that i know what i can tune exactly instead of being blackboxed by an LLM is sorta stress free. Comparatively.

One spaCy pass, three answers

spaCy does NER, Named Entity Recognition, and word tagging: nouns, verbs, adjectives, etc.

Named entities give us people, organizations, products, places, events and works of art. All of that comes with spaCy and is really convenient. It's placed at the end of the chain to catch names that the capitalization rule misses.

The tagging: comes free with the parse. We build a verdict per word: a lone word is dropped only if every occurrence of it in that post was tagged as a verb, auxiliary, adverb, adjective or interjection. Seen as a noun even once, it stays. That asymmetry is deliberate, since it stumbles on informal text (which is common on social media in general), and it is what finally killed announces from the earlier Nvidia example, which scores 3.9 (low enough to count as "interesting") on Zipf and had been sailing through every step unchallenged.

When the spaCy results get merged back in, anything already covered by an existing keyword is dropped, and where a phrase/word is already contained in a previously normalized phrase (even in part), it gets added: elden ring covers elden ring dlc. meta does not cover metallica.

A subreddit's own name is never a trend

This is pretty self explanatory. If you're on r/startups, it's already obvious that startup is a trending topic. We kick that out of the list (and its plural/close variants) so that it isn't in the top trends, stealing a spot away from something that's useful to users.

Multi word keywords get compacted, singularized and checked as a prefix. Two guards keep it from overreaching: it only applies to multi word keywords, so steamworks survives on r/steam, and only to sub names of five characters or more, so r/ai doesn't swallow every keyword starting with "ai".

Anyone can count, but that isn't necessarily useful

Extraction is only the counting layer. Afterwards, keywords roll up into posts containing and total uses, variant spellings merge, and singletons get absorbed into bigrams that already account for most of their appearances.

One detail there is worth stealing. Our singularizer is naive, and it turns texas into texa. Rather than fix it or replace it with something that "works", plural merging simply refuses to fire unless both surface forms actually occur in the data. If you have "thing" and "things" they get joined into one keyword, but if you have only one or the other, nothing gets stripped.

Then comes ranking, which is where all of these layers and filters and rules actually show their power. Raw frequency alone is NOT a trend signal. The word feedback is enormous on r/gamedev every single week, forever, and for users, it's useless. After going through the whole pipeline, you still are left with a much more relevant list, and on that list you can run some frequency analysis to pull trends.

What works is comparing a word against the community's own history. We look for topics/terms that spike relative to this day's/week's usual rate, and then score each keyword by the log ratio of its share now against its share then. A word running at or below its usual rate scores exactly zero, no matter how frequent it is.

score = posts x burst x global_idf x rarity

The rarity term is a capped multiplier taken from the rarest word in the keyword, so among things running at a normal rate the specific still beats the generic. steam playtest gets a spot and best engine gets nothing.

Where did the LLM go?

It is sitting in keyword_cleanup.py, behind an empty config value, and it is functional code. Everything before the actual LLM step is mint.

The design is alright. It's solid. It takes the top 80 keywords post tokenization with their counts plus the subreddit name and asks for verdicts: keep, merge or drop. Merges sum their counts and renames carry counts unchanged, so the numbers stay true no matter what the model says. If more than 80% of the list comes back dropped, we distrust the whole response and use the uncleaned list. Provider down, timeout, malformed JSON, fenced JSON, nonsense: every path falls back to the uncleaned list.

We built it, ran it, and then turned it off. It, ironically, did a worse job than good ol algorithms.

The other reason is debugging. When our pipeline surfaces a bad keyword I can trace it: which rule kept it, which threshold let it through, whether the tagger got confused. We have a whole test suite of painful titles just for this and we run it through every time we tweak anything, knowing the culprit of every failure. The LLM on the other hand is a black box, and we could only change the prompt and pray. Unreliable and wishful.

What I'd actually tell you

I'm not here to rant about AI bad. LLMs are really good at a continuously surprising range of tasks. However, what they are, are two things:

1. Overqualified. Models are made to overthink and overanalyze, and we observed that for this specific job it was tripping itself over all the time.
2. Overqualified. again! Overqualified workers are expensive workers, and the "best" solution isn't always YOUR solution.

The question I now ask before deeming a solution as OURS: can I write this rule down? If a competent person could explain the decision in a sentence, that sentence is cheaper as code. Keyword extraction from short titles turned out to be dozens of such sentences, and the frequency table we use (the zipf one) encodes what "common" means far better than we could describe it, from a corpus larger than anything we could realistically ever send over an API.

Save the model for the parts where it works the best, and don't trust it blindly. Scrutinize like you're its estranged mother. idk. For the scale of our operation using an LLM is the wrong move, maybe right now, maybe forever? i'll probably post about that when.

Thank you for reading.