Engineer to Engineer: Three tries to get the LinkedIn course matcher right

Alireza Ebrahimkhani

Alireza Ebrahimkhani

— 7 min read
Share
Engineer to Engineer: Three tries to get the LinkedIn course matcher right

I recently developed the LinkedIn matcher widget on DataChef Academy. It reads a visitor's job title, headline, and seniority, and recommends the courses most relevant to them, instantly.

No meetings to loop in a human curator. No stale spreadsheet of "who gets recommended what." Just paste a profile, and get an answer.

For the engineer reading, this will sound like a straightforward feature, yet it took me three tries to get the right solution. This blog post is the story of how we ended up discarding two versions of it and finally landing on the solution we have in production now.

The Problem: One Decision, Made Thousands of Times, By Nobody in Particular

The Academy catalog currently features nine courses. Every visitor who lands on the site fits one, or two, of them better than the rest. A "Head of Procurement" wants the procurement course. A "CTO with an engineering background" wants the modernization course, not the generic leadership one. A data analyst who's really doing product work sits somewhere in between.

There's no human in the loop to make this call. The decision has to happen automatically, in well under a second, for every single visitor, correctly enough that the recommendation feels personal rather than generic.

💡 This is not a "which AI model should we use" problem. It's a small, well bounded ranking problem wearing an AI shaped costume. And the shape of the fix depends entirely on noticing that.

Why the Obvious Fixes Don't Work

You might think: write a rules table that maps job titles to courses, or skip the rules entirely and let an LLM read the profile and decide. We tried both. Each one looked right at first, and each one fell short in a different way.

Attempt one: if/else with extra steps. The first version was a rules table. Lowercase the headline and title, run the string through an ordered list of regexes, first match wins.

{
  match: (t) => /\b(cpo|chief\s+procurement|procurement|sourcing(?:\s+lead)?)\b/i.test(t),
  primary: COURSES.aiForProcurement,
  primaryReason: "Tailored to procurement leaders applying AI to sourcing...",
  primaryScore: 98,
},
{
  match: (t) => /\b(cio|head\s+of\s+(?:engineering|platform)|software\s+architect)\b/i.test(t),
  primary: COURSES.legacyDisplacement,
  secondary: COURSES.eventstorming,
  primaryScore: 95,
},
// ...9 courses, ~9 rules, ordered from most specific to catch-all

It's free, instant, deterministic, and debuggable in about ten seconds: "why did this profile get X" always has a one line answer, this regex matched. It shipped in an afternoon, and it's still in the codebase today, just not as the main path anymore.

The ceiling shows up fast, though. A "Head of Data & AI Product" or a "VP Engineering, ex CTO" doesn't fit one bucket cleanly. The moment two rules can plausibly match the same headline, you're playing whack a mole with ordering to get the right one to fire first. I had to deliberately sort industry specific rules (procurement, marketing, deskless) ahead of generic seniority rules (VP, director, chief), purely so a CMO didn't fall into the generic "C-suite" bucket before reaching the marketing specific one. The rule list becomes a priority queue you maintain by hand, forever, and it gets more fragile with every course you add, not less.

Attempt two: throw an LLM at it. The obvious next move: stop hand writing rules, ship the profile and the full course catalog to an LLM, and ask it to pick. Profile summary and course descriptions in a prompt, anthropic/claude-haiku-4-5 via OpenRouter, JSON mode, asking for a primary and secondary pick with a one sentence personalized reason each.

const res = await fetch("https://openrouter.ai/api/v1/chat/completions", {
  method: "POST",
  headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "anthropic/claude-haiku-4-5",
    response_format: { type: "json_object" },
    messages: [
      { role: "system", content: "You are a course advisor... Return ONLY valid JSON..." },
      { role: "user", content: `Profile:\n${profileText}\n\nCourse catalog:\n${courseCatalog}` },
    ],
  }),
  signal: AbortSignal.timeout(6000),
});

The output quality was genuinely good. The generated reasons read more naturally than my regex authored copy. But it was the wrong shape of solution for what is, underneath, a small classification problem: 9 courses, not 9,000. And the wrongness showed up as concrete engineering cost, not a philosophical objection.

A live network hop landed on the critical path of a conversion widget: every visitor now blocks on a round trip to a hosted model before they see anything, on a page whose whole job is to convert a stranger into a lead in the first few seconds. There was non-determinism exactly where you don't want it: same profile, two different sessions, occasionally two different recommendations, which is fine for a chatbot and not fine for "why did this person get recommended this course" when someone on the sales team looks at the CRM record a week later. And there was a whole new failure mode class stacked on top of the old one: timeouts, rate limits, the model wrapping its JSON in a markdown fence, the model inventing a course slug that isn't in the catalog. I ended up writing a "fall back to the rules engine" path for every one of these, which means the system that actually shipped paid for an LLM call, kept the regex rules running underneath it as a safety net, and handled a new category of runtime failure the rules only version never had.

The common thread across both attempts: neither one matched the actual shape of the problem. Ranking 9 known options against a profile isn't a lookup table to hand maintain, and it isn't a generation task either. It's a similarity search.

What We Built Instead

The version running now: embed the visitor's profile text, embed a short description of each course once, offline, at build time, rank by cosine similarity, take the top matches.

Think of it as the rules table's ordering problem and the LLM's cost and non-determinism problem, solved by the same move: stop asking a model to decide, and let the numbers rank themselves.

export async function rankCoursesForProfile(profileText: string, embed: EmbedFn) {
  const profileVector = await embed(profileText);
  return courseVectors
    .map((c) => ({ slug: c.slug, title: c.title, score: dot(profileVector, c.vector) }))
    .sort((a, b) => b.score - a.score);
}

That's most of the actual matching logic. No prompt, no JSON parsing, no "which model do we trust to output valid slugs today," no network call sitting in the hot path. The profile embedding is the only thing computed at request time, and it's a small local operation, not a chat completion.

How It Works

1. The vector, not the model, makes the decision. Every course gets a description and a vector, computed once. Every visitor's profile gets a vector at request time. Cosine similarity ranks the courses. Nothing about this step depends on an LLM being available, fast, or in a good mood.

2. The embedder had to be boring everywhere it runs. "Run a model" means something different in each deployment target, and this is the part that actually took the time.

async function getEmbedder(): Promise<EmbedFn | null> {
  // Cloudflare Workers AI in prod, fastest, no external call.
  const ai = getCloudflareContext().env.AI;
  if (ai) return makeWorkersAIEmbedder(ai);

  // AWS Lambda can't load the native ONNX runtime, use HF's hosted inference API there.
  if (process.env.AWS_LAMBDA_FUNCTION_NAME) {
    return hfKey ? makeHuggingFaceEmbedder(hfKey) : null;
  }

  // Local dev, the bundled ONNX model, no network dependency at all.
  return makeLocalEmbedder();
}

Three different runtimes, three different ways of getting a vector out of bge-small-en-v1.5, one EmbedFn interface so the ranking code upstream never has to know or care which one it's talking to.

3. A course a visitor is already viewing gets its own quick check. The rules engine didn't disappear. It's the last resort fallback if no embedder is available, and it also runs a small, explicit computeCourseFit check that decides whether the course someone is already looking at is a good match for them, or whether we should point them somewhere else. Simple margin comparison against the top ranked score, no model call needed.

4. It gets checked against reality, not vibes. "Simple and pragmatic" is only a good story if it's measurably good enough. The product team hand built a golden set, 20 real CRM profiles, each with the courses they'd expect a person like that to be recommended, ranked. A script runs all 20 through the real production path and scores the result. Baseline: 19 of 20 profiles got a recommendation at all, mean recall@3 of 75%, top-1 accuracy of 47%. Recall is the number that matters most here, is the right course somewhere in the top 3, and it's solidly good. Top-1 is the honest weak spot, and it's not a mystery why: a couple of course descriptions are worded broadly enough that they win comparisons they shouldn't. That's a known, named, fixable problem, not a mystery black box.

What We Got

Instant recommendations: no network round trip in the hot path, no waiting on a hosted model to answer before a visitor sees anything.

Same input, same output: a recommendation made today matches the one made next week for the same profile, which matters the moment someone on the sales team asks "why did this person get this course."

A maintenance bill that shrinks instead of growing: adding a tenth course means writing one short description, not re-auditing nine ordered regex rules to see where the new one has to slot in.

A fallback that fails closed: if the embedder isn't available, the rules engine answers instead. Nothing breaks, nothing hangs waiting on a timeout.

A measurable baseline instead of a feeling: a 20 profile golden set and a script that scores the live path against it, so "is this good enough" has an actual number attached, and a regression check to catch it if the number drops.

Conclusion

Nine courses never needed a chat completion to get ranked correctly. The lesson that kept repeating across all three builds: match the tool to the actual shape of the problem, not to how capable the tool sounds on its own. A rules table answered fast but couldn't scale gracefully. An LLM answered well but added a second system to maintain underneath it. Embeddings did what the problem actually asked for, and nothing more.

That's a small, deliberate model doing exactly one job well. And it's a decent way to decide what any team reaches for next.

Curious how the matcher performs in practice? Try it yourself on any course page at datachef.academy, or reach out to us at datachef.co/contact.

#engineering #machinelearning #embeddings #buildinpublic #pragmaticengineering

Read more