AI labs need annotated data, and the current market for it is a mess. Freelance annotators are hard to vet — anyone can claim to be a domain expert. Skill claims are unverifiable without an assessment layer. Quality control is manual, inconsistent, and usually happens after bad data has already entered a training set.
Payments across borders introduce their own delays and disputes, especially when annotators are in emerging markets and clients are in the US or Europe. Existing generic freelance platforms — Upwork, Fiverr — weren't built for the specific trust primitives an annotation workflow demands: proof of identity, proof of domain skill, and proof of data quality before it lands in a training set.
Lingtec's bet was that a purpose-built marketplace could compress that entire trust stack into one platform — and that AI labs would pay a premium for data they could actually trust without running their own second-pass QA.
Marketplaces don't fail on their matching algorithms. They fail on trust. Lingocore is a trust stack that happens to have a marketplace on top.
Build order is strategy
Building a marketplace means building trust primitives — you don't ship the two-sided flow first, you ship the trust stack that makes the two-sided flow safe. The order I built the systems in mattered, because each one unlocked the next.
KYC came first
Nothing else works without verified identity. The platform can't pay someone it can't verify, and clients can't accept data from someone anonymous. I built the KYC flow to be strict on identity but lightweight on friction — annotators can start browsing tasks immediately but can't accept a paid task until they're verified.
That balance took several iterations to get right. Too strict early and annotators bounced during signup. Too loose and we ended up with duplicate accounts and identity ambiguity that made payouts a compliance headache. The final flow verifies at the moment of first paid engagement, which is when the friction is tolerable because the value is clear.
Assessments came second
Letting unqualified annotators near client tasks poisons the marketplace's reputation on the first bad delivery. But building a bespoke assessment engine per task category doesn't scale. I designed the assessment infrastructure to be domain-agnostic — the same underlying engine runs a linguistics test, a medical annotation test, or a code review test, with the specific questions and scoring rubrics defined per task category.
Annotators are tiered by demonstrated capability, and tiers gate which tasks they can bid on. A Tier 3 medical annotator can bid on Tier 1, 2, and 3 medical tasks. They can't bid on legal tasks unless they've passed the legal assessment.
Wallet came third
Annotators quit fast if payments feel unpredictable. Cross-border payment infrastructure has more edge cases than any other part of the build: currency conversion timing, provider settlement windows, payout thresholds, dispute flows, and reversal handling each introduce their own failure modes.
I designed the wallet layer to abstract over multiple payment providers so the platform isn't locked in. If a provider gets slow or expensive, we can route to another without annotators noticing anything except that their payout arrived on time. That abstraction has already saved us once when a provider had a two-day settlement delay during a regional banking issue.
QA came fourth — and it's the hardest problem
Catching annotator drift is easy when you have ground truth. But in a marketplace, the ground truth is itself annotated by humans. Which means the QA system can't be "compare against the answer key" — there is no answer key.
The system I ended up with is probabilistic. Every annotation is scored across two axes: correctness (agreement with expected outputs where they exist) and reviewer agreement (agreement across multiple independent reviewers). Low-agreement annotations get escalated for a senior reviewer pass before they enter the client dataset. The QA layer is transparent to annotators: they can see which of their annotations got escalated and why, so a low score is never a black box.
Dataset delivery is where the platform earns its premium
The client isn't buying annotations. They're buying annotations with a quality guarantee attached. The delivery pipeline packages verified annotations into a format the client can train on directly, with quality metadata attached at row level. Clients can filter by confidence threshold at ingest — 'give me only annotations where all three reviewers agreed' — without doing their own second-pass QA.
That single feature is why AI labs pay a premium for data from a marketplace instead of hiring their own annotation team: they skip the internal QA loop entirely.
What I learned
Marketplace platforms are less about the matching algorithm and more about the trust systems underneath. KYC, assessments, and QA are what actually determine whether the marketplace works. The matching UI is the easy part. Every hour I spent on the QA layer paid back tenfold; every hour I spent optimizing the annotator dashboard paid back marginally.
Cross-border payment infrastructure has more edge cases than any other part of the build. If I built the wallet system again from scratch I'd start with a multi-provider abstraction on day one, not layer it in after the first provider got flaky.
Human ground truth is a moving target. Quality control at marketplace scale has to be probabilistic, not deterministic.
And QA has to be transparent to the annotator. If they get penalized on a score they don't understand, they leave the platform — and the marketplace loses supply-side liquidity long before it loses demand.
What I owned
- Built the KYC and onboarding pipeline — identity verification and qualification flows that gate every annotator before they can bid on paid work, balanced to minimize signup friction while enforcing strict identity checks at the moment of first paid engagement
- Designed and shipped the skills assessment engine — a domain-agnostic infrastructure that runs different rubrics per task category and tiers annotators by demonstrated capability, with tier gates controlling which tasks each annotator can bid on
- Built the wallet and cross-border payments infrastructure — abstracted over multiple payment providers so the platform isn't locked in, with currency conversion timing, payout thresholds, and dispute handling all built into the wallet layer
- Implemented the QA and data quality layer — reviewer workflows, weighted multi-reviewer quality scoring, and a probabilistic model that catches low-agreement annotations and escalates them for a senior reviewer pass before they enter a dataset
- Shipped the dataset delivery pipeline — packaging verified annotations into client-ready datasets with row-level quality metadata, so clients can filter by confidence threshold at ingest time and skip their own second-pass QA
- Owned the backend architecture end-to-end — data models, API design, cloud infra, and the trust-flow that connects onboarding → assessment → task → QA → payout as one integrated system