AI Agents

Building a Recruiting Agent That Screens Resumes Fairly

Pramod Dutta · May 6, 2026 · 15 min read

The Resume Pile Nobody Wants to Touch

Every recruiter has felt it: a job posting goes live on a Friday, and by Monday there are 800 resumes sitting in the applicant tracking system. A human being is supposed to read all of them, extract the relevant signal, and rank candidates fairly. In practice, nobody reads all 800. Recruiters skim. They pattern-match on school names, previous employers, and keyword density. They get tired around resume 200 and start rejecting faster than they did at resume 20. This is not a moral failing — it is what happens when a cognitively demanding task is scaled past human capacity.

The instinct to fix this with AI is correct. The instinct to fix it by throwing resumes at a language model and asking "does this person seem qualified" is where things go wrong. An unstructured LLM call on a resume will happily reproduce every bias baked into its training data, plus new biases introduced by how you phrased the prompt. It will penalize employment gaps without asking why. It will rate a resume with a "prestigious" university higher even when the actual work experience is identical to a candidate from a state school. It will infer gender from a name and let that infer influence a "culture fit" judgment nobody asked it to make.

This article is about building a recruiting agent that does the tedious extraction and ranking work at scale, while treating fairness as an engineering requirement with tests and audit logs, not a hope. We will walk through the architecture, the specific bias-mitigation techniques that actually hold up, and working code for a screening pipeline you could extend into production. This is also exactly the kind of multi-step, tool-using, auditable agent system we build hands-on in 30 Days of Hermes Agent, so if this pattern resonates, that course goes much deeper into the agent design shown here.

Why Naive LLM Screening Fails

Before designing the fix, it's worth being precise about the failure modes, because each one implies a different mitigation.

Proxy discrimination. A model asked to predict "job success" from a resume will latch onto correlates of protected characteristics even if you never mention race, gender, or age. Zip codes correlate with race. Graduation years correlate with age. Names correlate with ethnicity and gender. Career gaps correlate with caregiving responsibilities, which correlates with gender. If your agent's prompt is "rate this candidate's fit," the model has no scaffolding to avoid these proxies — it just optimizes for whatever pattern looks like "success" in its training distribution.

Prompt-order and framing sensitivity. LLMs are surprisingly sensitive to how information is presented. Put the candidate's name and photo-adjacent details (club memberships, personal interests) at the top of the context, and they color everything read afterward. Two functionally identical resumes, reordered, can get different scores.

Halo effects from brand names. "Google," "Goldman Sachs," or "Stanford" on a resume triggers a positive halo that has nothing to do with the actual bullet points describing what the person did. This is a real bias in human recruiters too, but an LLM will do it more consistently and at higher volume, which means it does more damage per unit time if left unchecked.

No decision trail. Perhaps the biggest practical problem: if your screening step is "ask GPT/Claude to rank these 800 resumes," you get a ranked list and no reasoning artifact you can audit later. When a rejected candidate asks why, or a regulator asks for adverse-impact statistics, you have nothing to show except a black box.

The fix for all four is the same general strategy: decompose the judgment into structured, independently checkable sub-decisions, strip identity signals before scoring, and log every step.

Architecture: Extract, Redact, Score, Aggregate, Audit

The agent is not one LLM call. It is a pipeline with five distinct stages, each with a narrow job:

Extraction — pull structured facts out of unstructured resume text (skills, years of experience, education, employment history) without judgment.
Redaction — strip or mask fields that are legally protected or bias-prone (name, photo references, graduation dates beyond a threshold, address, gendered pronouns in self-description).
Structured scoring — score the redacted, structured data against an explicit, pre-published rubric tied to the job requirements — not a vague "fit" prompt.
Aggregation and calibration — combine sub-scores, and run a statistical check for disparate impact across the batch before anyone sees a ranked list.
Audit logging — persist every extraction, every redaction decision, every rubric score, and every calibration adjustment so a human can trace exactly why a candidate ranked where they did.

Here is the skeleton in Python, using a simple orchestration pattern (the same pattern generalizes to LangGraph, a custom agent loop, or the Hermes-style agent loop we teach in the course):

from dataclasses import dataclass, field
from typing import Optional
import json
import re

@dataclass
class ExtractedProfile:
    candidate_id: str
    skills: list[str]
    years_experience: float
    education_level: str  # "bachelors", "masters", "phd", "none", "other"
    employment_history: list[dict]
    certifications: list[str]
    raw_text_hash: str  # for audit trail, never the raw text itself downstream

@dataclass
class RedactedProfile:
    candidate_id: str
    skills: list[str]
    years_experience_bucket: str  # bucketed, not exact, to reduce age proxy signal
    education_level: str
    relevant_employment_summaries: list[str]  # employer names stripped
    certifications: list[str]

@dataclass
class ScoreResult:
    candidate_id: str
    rubric_scores: dict[str, float]
    total_score: float
    rationale: dict[str, str]


def extract_profile(resume_text: str, candidate_id: str) -> ExtractedProfile:
    """
    Stage 1: pure extraction, no judgment calls.
    In production this is an LLM call constrained to a JSON schema,
    with the prompt explicitly forbidding evaluative language.
    """
    # Pseudocode for the LLM call; the real implementation enforces
    # a strict JSON schema via function calling / structured outputs.
    extraction_prompt = f"""
    Extract ONLY the following fields from this resume as JSON.
    Do not evaluate, rank, or comment on quality. Do not infer
    demographic attributes. If a field is not present, use null.

    Fields: skills (list), years_experience (number),
    education_level (string), employment_history (list of
    {{title, employer, start_year, end_year, responsibilities}}),
    certifications (list)

    Resume:
    {resume_text}
    """
    # result = llm_call(extraction_prompt, response_format="json")
    result = mock_llm_extract(resume_text)
    return ExtractedProfile(
        candidate_id=candidate_id,
        skills=result["skills"],
        years_experience=result["years_experience"],
        education_level=result["education_level"],
        employment_history=result["employment_history"],
        certifications=result["certifications"],
        raw_text_hash=str(hash(resume_text)),
    )

Notice what stage 1 deliberately does *not* do: it does not ask the model "is this a strong candidate." It only extracts facts. That separation is the single highest-leverage design decision in this whole pipeline, because it means the model never gets to blend "what is true" with "what I think of it" in one pass.

Redaction: The Bias-Mitigation Core

The redaction stage is where most of the actual fairness engineering happens. Names go first — obviously. But the subtler wins are in bucketing continuous variables and stripping employer names before scoring, then re-attaching them only after the score is locked.

import re

BLIND_EMPLOYER_MAP: dict[str, str] = {}
_employer_counter = 0

def anonymize_employer(name: str) -> str:
    """Replace employer names with stable pseudonyms so halo effects
    from 'prestigious' companies can't leak into scoring, while still
    letting the model reason about role progression and tenure."""
    global _employer_counter
    if name not in BLIND_EMPLOYER_MAP:
        _employer_counter += 1
        BLIND_EMPLOYER_MAP[name] = f"Company_{_employer_counter}"
    return BLIND_EMPLOYER_MAP[name]


def bucket_experience(years: float) -> str:
    """Bucket exact years into ranges. Exact years correlate too
    tightly with age; buckets keep the signal recruiters actually
    need (junior/mid/senior) without the fine-grained proxy."""
    if years < 2:
        return "0-2 years"
    if years < 5:
        return "2-5 years"
    if years < 10:
        return "5-10 years"
    return "10+ years"


def strip_identity_signals(text: str) -> str:
    """Remove common identity-adjacent tokens: names already removed
    upstream, but also strip pronouns, university names beyond a
    generic 'degree granted' fact, and personal interest sections
    that often carry cultural or demographic signal."""
    text = re.sub(r"\b(he|she|him|her|his|hers)\b", "", text, flags=re.I)
    text = re.sub(r"(?i)hobbies:.*", "", text)
    text = re.sub(r"(?i)interests:.*", "", text)
    return text.strip()


def redact_profile(profile: ExtractedProfile) -> RedactedProfile:
    summaries = []
    for job in profile.employment_history:
        anon_employer = anonymize_employer(job["employer"])
        summary = strip_identity_signals(
            f"{job['title']} at {anon_employer}: {job['responsibilities']}"
        )
        summaries.append(summary)

    return RedactedProfile(
        candidate_id=profile.candidate_id,
        skills=profile.skills,
        years_experience_bucket=bucket_experience(profile.years_experience),
        education_level=profile.education_level,  # degree level kept, institution dropped
        relevant_employment_summaries=summaries,
        certifications=profile.certifications,
    )

A few deliberate choices worth calling out. Employer names are pseudonymized, not deleted — the model can still reason about "did responsibilities grow across three roles," which is legitimate signal, without knowing whether "Company_3" is a Fortune 500 brand or a ten-person startup. Education level is kept as a coarse category (bachelor's, master's) but the institution name is dropped entirely, because institution name is doing almost pure prestige-signaling work and very little job-relevant work for most roles. Years of experience is bucketed rather than exact, which blunts age-proxy inference while preserving the seniority band recruiters actually care about.

None of this is a silver bullet. A sufficiently sophisticated model can sometimes infer prestige from writing style or the sophistication of project descriptions. Redaction reduces bias, it does not eliminate it — which is exactly why the pipeline doesn't stop here.

Structured Scoring Against a Published Rubric

The scoring stage should never ask an open-ended "how good is this candidate" question. It should score against a rubric you wrote and published *before* looking at any resumes, tied directly to the job requirements. This matters legally as well as technically — a documented, pre-registered rubric is your best defense if a hiring decision is ever challenged.

JOB_RUBRIC = {
    "required_skills_match": {
        "weight": 0.35,
        "criteria": "Percentage overlap between candidate skills and "
                     "the job's required_skills list.",
    },
    "relevant_experience_depth": {
        "weight": 0.30,
        "criteria": "Does employment history show increasing scope "
                     "of responsibility in directly relevant work?",
    },
    "certification_alignment": {
        "weight": 0.15,
        "criteria": "Do certifications match role requirements "
                     "(e.g. AWS cert for a cloud role)?",
    },
    "experience_band_fit": {
        "weight": 0.20,
        "criteria": "Does years_experience_bucket match the role's "
                     "target band (junior/mid/senior)?",
    },
}


def score_required_skills(profile: RedactedProfile, required_skills: list[str]) -> tuple[float, str]:
    matched = set(s.lower() for s in profile.skills) & set(s.lower() for s in required_skills)
    ratio = len(matched) / max(len(required_skills), 1)
    rationale = f"Matched {len(matched)}/{len(required_skills)} required skills: {sorted(matched)}"
    return round(ratio * 100, 1), rationale


def score_experience_band(profile: RedactedProfile, target_band: str) -> tuple[float, str]:
    bands_order = ["0-2 years", "2-5 years", "5-10 years", "10+ years"]
    candidate_idx = bands_order.index(profile.years_experience_bucket)
    target_idx = bands_order.index(target_band)
    distance = abs(candidate_idx - target_idx)
    score = max(0.0, 100 - distance * 30)
    rationale = f"Candidate band '{profile.years_experience_bucket}' vs target '{target_band}', distance={distance}"
    return score, rationale


def score_candidate(profile: RedactedProfile, job_requirements: dict) -> ScoreResult:
    skills_score, skills_rationale = score_required_skills(
        profile, job_requirements["required_skills"]
    )
    band_score, band_rationale = score_experience_band(
        profile, job_requirements["target_experience_band"]
    )

    # These two would similarly be broken into narrow, auditable
    # functions rather than one big LLM "judge" call.
    cert_score, cert_rationale = 70.0, "Manual placeholder for cert-matching logic"
    depth_score, depth_rationale = 65.0, "Manual placeholder for progression analysis"

    rubric_scores = {
        "required_skills_match": skills_score,
        "experience_band_fit": band_score,
        "certification_alignment": cert_score,
        "relevant_experience_depth": depth_score,
    }

    total = sum(
        rubric_scores[key] * JOB_RUBRIC[key]["weight"]
        for key in rubric_scores
    )

    return ScoreResult(
        candidate_id=profile.candidate_id,
        rubric_scores=rubric_scores,
        total_score=round(total, 2),
        rationale={
            "required_skills_match": skills_rationale,
            "experience_band_fit": band_rationale,
            "certification_alignment": cert_rationale,
            "relevant_experience_depth": depth_rationale,
        },
    )

Where an LLM is genuinely useful here is scoring the qualitative sub-criteria — "relevant_experience_depth" is a judgment call that benefits from language understanding. But even there, the call should be scoped tightly: give the model the redacted employment summaries and the specific rubric criterion, ask it to return a score and a one-sentence rationale citing specific text, and nothing else. Never let it see the whole resume and produce a single holistic number.

Aggregation and Disparate-Impact Calibration

This is the stage most screening tools skip entirely, and it's the one that catches bias you didn't design out earlier. Before anyone sees a ranked list, run a statistical check across the batch. You will not always have demographic data to check against directly — and you often shouldn't collect it for this purpose in most jurisdictions — but you can and should check for suspicious correlations with proxies you do have, and you can run the "four-fifths rule" style check when self-reported EEO data is available through a separate, access-controlled channel.

from statistics import mean, stdev

def flag_score_anomalies(scores: list[ScoreResult], threshold_stdevs: float = 2.5) -> list[str]:
    """Flag candidates whose total score is a statistical outlier,
    for human review rather than automatic rejection or advancement.
    Outliers in either direction get a second look — this catches
    both scoring bugs and edge cases the rubric didn't anticipate."""
    totals = [s.total_score for s in scores]
    if len(totals) < 5:
        return []  # not enough data for meaningful stats
    avg, sd = mean(totals), stdev(totals)
    flagged = []
    for s in scores:
        if sd > 0 and abs(s.total_score - avg) > threshold_stdevs * sd:
            flagged.append(s.candidate_id)
    return flagged


def four_fifths_check(
    pass_rates_by_group: dict[str, float],
) -> dict[str, bool]:
    """Standard adverse-impact heuristic used in US EEO compliance:
    the pass rate for any group should be at least 80% of the
    pass rate for the group with the highest rate. This requires
    demographic data pulled from a separate, voluntary, access-
    controlled source -- never inferred from the resume itself."""
    if not pass_rates_by_group:
        return {}
    max_rate = max(pass_rates_by_group.values())
    if max_rate == 0:
        return {group: True for group in pass_rates_by_group}
    return {
        group: (rate / max_rate) >= 0.8
        for group, rate in pass_rates_by_group.items()
    }

The four_fifths_check function deliberately takes pre-aggregated pass rates as input rather than raw candidate records with demographic labels attached. That separation is intentional: the scoring pipeline never sees protected-class data, but a compliance process running in parallel, with access controlled separately from the recruiting agent, can still audit outcomes against it. This is the same "separation of duties" pattern you'd use for any sensitive-data pipeline — the system that makes the decision and the system that audits the decision should not share unrestricted access to the same data.

If the four-fifths check fails for any rubric-driven cutoff, that is a signal to a human that the rubric itself, or a redaction step, needs revisiting — not a signal to quietly adjust individual scores until the numbers look better.

Human-in-the-Loop by Design, Not by Afterthought

A genuinely fair recruiting agent is not one that removes humans from the loop — it's one that hands humans a much better starting point and never lets the automation make the final call alone. Concretely, this means:

The agent produces a ranked shortlist with rubric-level rationale, not just a single score.
Anomaly-flagged candidates (from flag_score_anomalies) get manual review before any rejection.
Rejections are never sent automatically; a recruiter reviews the bottom of the list too, not just the top, since rubric blind spots often show up as good candidates scored low.
Every candidate, not just the shortlisted ones, gets their extracted profile and score stored, so a later audit or a candidate inquiry can be answered with specifics.

def build_review_package(scores: list[ScoreResult], profiles: dict[str, RedactedProfile]) -> dict:
    ranked = sorted(scores, key=lambda s: s.total_score, reverse=True)
    anomalies = flag_score_anomalies(scores)

    return {
        "ranked_candidates": [
            {
                "candidate_id": s.candidate_id,
                "total_score": s.total_score,
                "rubric_scores": s.rubric_scores,
                "rationale": s.rationale,
                "flagged_for_review": s.candidate_id in anomalies,
            }
            for s in ranked
        ],
        "requires_human_decision": True,
        "rubric_version": "job_req_2026_v1",
    }

That rubric_version field matters more than it looks. Rubrics should be versioned artifacts, checked into source control like code, so that if a rubric changes between hiring rounds, you can explain exactly what changed and why, and re-score old candidates consistently if they're reconsidered.

Building and Testing the Bias Checks

Because fairness claims are easy to assert and hard to verify, treat the redaction and scoring logic like any other code with a correctness requirement: write tests that assert the bias-mitigation properties directly, not just that the pipeline runs.

def test_identical_resumes_different_names_score_equal():
    resume_a = make_resume(name="Michael Chen", employer="Acme Corp", years=4)
    resume_b = make_resume(name="Fatima Al-Rashid", employer="Acme Corp", years=4)

    profile_a = redact_profile(extract_profile(resume_a, "a"))
    profile_b = redact_profile(extract_profile(resume_b, "b"))

    score_a = score_candidate(profile_a, JOB_REQUIREMENTS)
    score_b = score_candidate(profile_b, JOB_REQUIREMENTS)

    assert score_a.total_score == score_b.total_score


def test_employer_prestige_does_not_change_score():
    resume_a = make_resume(name="Person One", employer="Tiny Startup LLC", years=5)
    resume_b = make_resume(name="Person Two", employer="Famous Big Tech Co", years=5)
    # same responsibilities text on both

    profile_a = redact_profile(extract_profile(resume_a, "a"))
    profile_b = redact_profile(extract_profile(resume_b, "b"))

    score_a = score_candidate(profile_a, JOB_REQUIREMENTS)
    score_b = score_candidate(profile_b, JOB_REQUIREMENTS)

    assert score_a.total_score == score_b.total_score


def test_exact_years_bucketed_before_scoring():
    profile = RedactedProfile(
        candidate_id="c1",
        skills=["python"],
        years_experience_bucket="5-10 years",
        education_level="bachelors",
        relevant_employment_summaries=[],
        certifications=[],
    )
    assert profile.years_experience_bucket in {"0-2 years", "2-5 years", "5-10 years", "10+ years"}

These tests should run in CI on every rubric or prompt change. If a prompt tweak makes test_employer_prestige_does_not_change_score fail, that's a regression, exactly the same as a failing unit test on a payment calculation, and it should block the deploy the same way.

Where This Fits in a Real Hiring Pipeline

It's worth being honest about scope. This agent is a screening assistant that produces a ranked shortlist with rationale — it is not, and should not be, the sole decision-maker for who gets hired. Interviews, reference checks, and human judgment on things resumes can't capture (communication in conversation, collaborative problem-solving) remain essential. The value this pipeline adds is turning an unmanageable pile of 800 resumes into a defensible, rationale-backed shortlist of 30, in a way that's consistent across candidates and auditable after the fact — which is a meaningfully higher bar than what most manual screening achieves today, without pretending the model is doing something it isn't.

It's also worth noting the legal landscape is moving fast: several jurisdictions now require bias audits for automated hiring tools before they're used in production, and disclosure to candidates that AI was involved in screening. Build the audit trail described above from day one, and you're already most of the way to compliance instead of scrambling to reconstruct one later.

Building This Yourself

The pattern here — decompose a judgment task into narrow extraction, redaction, scoring, and calibration stages, each independently testable, each producing an audit artifact — generalizes far beyond recruiting. It's the same shape you'd use for loan-application triage, content moderation, or insurance claim review: anywhere an agent is making a consequential decision about a person and needs to show its work.

If you want to build agents like this end-to-end — tool use, structured outputs, multi-stage orchestration, evaluation harnesses, and the guardrails that keep an agent from quietly doing something unfair or unsafe — that's the core of what we teach in 30 Days of Hermes Agent. The course walks through building production-grade agents from first principles, including the exact kind of decompose-and-audit pattern used here, so you leave with working systems, not just a mental model of one.