AI and Machine Learning · Prediction contest with a public and a hidden test set

🤖 How to play Overfit

Your team enters Wadi Bank's loan-prediction contest. The public leaderboard is visible; the hidden test set decides.

Join a game Run it as facilitator Print this guide

Your goal

Win by earning the most hidden-test points over three rounds. Each round your model is really trained on the same seeded loan data as everyone else. Points come from accuracy on 1,500 hidden recent loans, not from the public score. Keep the public–hidden gap small and treat men and women applicants fairly.

You run one data science team. Every team trains on the same 240 historical loans, is shown accuracy on the same 150-loan public set, and is scored on the same hidden set.

Join in 30 seconds

Scan the QR code on the big screen with your phone camera, or open the game page and type the 4-letter code.
Type your first name. No app, no account.
Join a team or create one and name it. Your teammates get a role each. In individual mode you run your own data science team.
If your phone locks or the page reloads, open it again: you return to the same seat.

Team roles

Lead: Chairs the team and submits the final decision. If the Lead's phone dies, any teammate can take over. In this game: Decide what you are optimising: hidden accuracy, not the public number on the screen.
Analyst: Owns the numbers and the team's prediction for the round. In this game: Compare training, public and hidden accuracy: a big train–test gap means overfitting.
Treasurer: Watches the budget and costs before the team commits. In this game: Buy extra data only when the model is complex enough to need it.
Risk and ethics officer: Flags harms and pushes back before the team commits. In this game: Ask of every feature: is it known at decision time, and is it a proxy for gender?
Voice: Talks to other teams and presents the team's story at the debrief. In this game: Be ready to explain to the ethics board why each feature is fair to use.

The three rounds

Each round you predict your own result, make your decisions within about 8 minutes, and then the model reveals what happened. An event card lands between rounds.

1
Round 1 · First model
Choose features, complexity, regularisation, data, threshold and whether to use gender. You see training, public and hidden accuracy in your results.
Watch for: Compare training and hidden accuracy across teams. The teams with the most features rarely lead on the hidden set.
2
Round 2 · Tuning
Tune with last round's public leaderboard on your phone. Probing the public set many times is allowed and tempting.
Watch for: The bar chart of public minus hidden accuracy. Teams with 20 probes usually show the biggest positive gap.
3
Round 3 · Final and fairness audit
Final model on the hidden set. The ethics board fines 8 points for an approval-rate gap above 10 points between men and women who really repaid, and 5 more for using gender.
Watch for: The shake-up between the public rank and the final points. Which teams paid audit fines?

Your decisions

What each control on your phone does, and how to think about it.

Features to use	Every round	Pick any of eight features (nine after the leak event). Revenue, debt ratio and late payments carry real signal; years trading is weaker; handset price echoes revenue; night-time use and followers are pure noise; district code is a proxy for gender. Start with the strong signals. Every noise feature adds weights the model can use to memorise training noise.
Model complexity	Every round	Linear uses one weight per feature. Quadratic adds every square and every pair (4 features → 14 weights). Cubic also adds cubes. The true pattern has one interaction and one curve, so a little flexibility helps. Quadratic on a few good features is often best. Cubic on many features memorises noise unless you add data or regularisation.
Regularisation (L2)	Every round	None (λ = 0), light (0.003), medium (0.03) or strong (0.3). The penalty λ/2·‖w‖² pulls weights towards zero. The more weights you have, the more regularisation you need. Watch the training–hidden gap in your results.
Extra training data	Every round	None, +240 or +720 loans for 2 or 5 points this round (half price after the cheap-data event, but that data is biased against women). More data matters most for complex models and after the mislabelled-batch event. For a small linear model it rarely pays.
Decision threshold	Every round	0.2 to 0.8. A loan is approved when the predicted chance of repayment is at least the threshold. Accuracy counts approvals and refusals alike. 0.5 is a sensible start. Move it only with a reason, such as drift that lowers repayment in recent loans.
Sensitive feature: gender	Every round	Use or remove gender. Old labels recorded some loans repaid by women as defaults, so gender learns that bias. It costs Trust and, in Round 3, a 5-point fine. Removing gender is not enough if you keep the district code, which carries much of the same information.
Public leaderboard probes	Every round	1, 5 or 20. Each extra probe tries a variant (often dropping one feature, and moving the threshold by up to 0.1) and keeps the one with the best public score. With only 150 public loans, the best of 20 probes is often just lucky. It raises the public score more than the hidden one.

What you see on your phone

The data: Training loans (240), extra loans for sale (720), the public test set (150), the hidden test set (1,500) and the share of training loans that were repaid.
Feature catalogue: Every feature you may use with a note: real signal, weak signal, pure noise, a proxy for gender, or, after the leak event, a column only known after a default.
What happens when you submit: The training loop in short Python: expand the features, run 600 Adam steps on batches of 20 with L2, then approve when the predicted chance of repayment reaches your threshold.
Your model and the public leaderboard: From Round 2: your training, public and hidden accuracy, fairness gap and model size, plus last round's public leaderboard.

How you are scored

Every game in the library uses the same scoreboard. Your round result is your Hidden-test points (pts); the leaderboard adds resilience and trust.

Score = 100 × (0.5 × performance + 0.25 × resilience + 0.25 × trust) + prediction points

Performance: Hidden-test pointsPoints = 200 × (hidden accuracy − 0.5), minus data cost (2 or 5), minus 2 per personal feature after the privacy rule, minus Round 3 audit fines (8 for a fairness gap above 10 points, 5 for using gender). 78% hidden accuracy with no costs gives 56 points.

Resilience (0–100)Generalisation. Each round R moves halfway to 100 minus 4 per percentage point of gap between public and hidden accuracy. A 5-point gap gives a target of 80; a 25-point gap gives 0.

Trust (0–100)Fairness and privacy. Each round T moves halfway to 100 minus 3 per point of approval-rate gap between men and women who repaid, minus privacy penalties: gender 25, district code 8, night-time use 6, followers 6 (all ×1.5 after the privacy rule).

Performance is your total Hidden-test points over the rounds, scaled from the lowest to the highest in the room. Before each round you also predict “Your points this round (hidden set)”: land close and earn up to 10 bonus points; miss by the whole amount and earn none. The facilitator may change the weights before the game starts.

Tips

Hidden accuracy pays, public accuracy only informs. A small public set is noisy.
Read three numbers together: training, public and hidden. Training far above hidden means overfitting.
Add weights only when you add data or regularisation to control them.
Ask whether each feature is known at the moment the loan is decided.
Check for proxies. A feature that tracks gender brings the historical bias back.

Common mistakes

Using every feature with a cubic model and no regularisation. Training accuracy soars and hidden accuracy falls.
Running 20 probes and trusting the public score that comes back.
Using the collections flag after the leak event. It looks perfect on old data and is useless on new loans.
Removing gender but keeping the district code, then failing the fairness audit.

A worked example

In one session a team uses revenue, debt ratio, late payments and years trading, quadratic, light regularisation, no extra data, threshold 0.5 and no gender. The model has 14 weights trained on 240 loans: training 72.9%, public 77.3%, hidden 78.1%, so points = 200 × (0.781 − 0.5) = 56.1. A rival uses all eight features, cubic, no regularisation: training 82.9%, public 78.0%, but hidden only 72.1%, about 44 points.

Shocks that can hit

The facilitator draws one event card before rounds 2 and 3. It changes the market from that round on.

Data drift. An economic slowdown hits small businesses. In the hidden set, revenue falls and repayment gets harder. The training data does not change.
Mislabelled batch. A data-entry error flipped the outcome of about 15% of the 240 base training loans. Purchased extra data is clean.
Leaked feature. A new column appears in the training and public data: the collections flag. It looks very predictive. It is only set after a loan defaults.
New privacy rule. A new data-protection rule requires consent for personal data. Gender, district code, night-time app use and social followers now cost 2 points each to use, and their trust penalty rises by half.
Cheap but biased dataset. A broker offers extra loan data at half price. In it, 30% of loans repaid by women were recorded as defaults.

The model behind the game

Logistic regression with feature expansion, trained by mini-batch Adam with L2. Every team's model is actually trained on the same seeded synthetic loans. More terms let the model bend to the training data, including its noise. Regularisation and more data pull it back. The hidden set comes from recent loans, so shortcuts that only worked on old data stop working.

p = σ(w·x + b) ; loss = −mean[y·log p + (1−y)·log(1−p)] + λ/2·‖w‖² ; approve if p ≥ threshold ; points = 200·(hidden accuracy − 0.5) − data cost − fines

Overfitting: A model that fits noise in its training data, so it scores well there and worse on new data.
Regularisation (L2): A penalty on large weights, λ/2·‖w‖², that keeps a flexible model simpler.
Logistic regression: A model that turns a weighted sum of features into a probability with the sigmoid function.
Public vs hidden test set: The public set gives feedback during the contest; the hidden set gives the final, unbiased score.
Data leakage: Using information that would not be available when the prediction is made.
Proxy variable: A feature that stands in for a sensitive attribute, such as district code for gender.
Equal opportunity: A fairness check: people who would really repay should be approved at similar rates in every group.
Data drift: When new data differs from the data the model was trained on.

For facilitators 3 rounds · about 50 minutes

Set-up

Team mode with four to eight teams works best; bots fill the room to four.
Allow 8 minutes per round. Round 1 needs time to read the feature catalogue.
Before Round 1, explain the three numbers: training, public and hidden accuracy, and that only hidden accuracy scores.
Mention that every model is really trained on the same data, so teams can compare choices fairly.

Modes

Team: One model per team. The Analyst picks features and complexity, Risk sets regularisation and threshold, the Treasurer buys data, the Voice decides on gender and the Lead sets probes.
Individual: Every player trains their own model on the same data. The public leaderboard gets crowded, so the shake-up in Round 3 is larger. Bots join only if fewer than four players.
Hybrid: Each player submits alone inside a team, and the team board averages members' scores. Teammates can compare choices because the data is identical.

Run sheet

Before class	Open the facilitator console on the projector laptop, choose the mode in Setup, and check the QR shows.
0–5 min	Students scan and join. Read the situation aloud while teams form. Point out the roles on their phones.
5–15 min	Round 1: 8 minutes to decide and predict, then Reveal. Read the headline and one surprising line from a team.
Between rounds	Draw or choose the event card. Give teams 30 seconds to read it before opening the next round.
15–25 min	Round 2, then Reveal.
27–37 min	Round 3, then Final results: podium, awards and the score chart.
Last 12 min	Debrief with the questions below, then open reflections (4 minutes). Download the CSV.

What usually happens

Teams that tick every feature and pick cubic. Ask them to compare training and hidden accuracy.
Heavy probing in Round 2. Show the public-minus-hidden bars and ask who chased noise.
Teams that remove gender but keep the district code. Point to their fairness gap.
After the leak event, someone will use the collections flag. Ask when that flag is recorded.

Debrief questions

Whose public score was much higher than their hidden score, and what did they do in Round 2 to get there?
Did removing gender make your model fair? What did the district code do?
When did extra data or stronger regularisation help, and when was it wasted?

Follow-up probes

Why can the public score rise while the hidden score falls?
Which choice did more for your hidden accuracy: fewer features, regularisation or more data?
Is a model that never uses gender automatically fair? What evidence do you have?
When would you buy cheap data that you know is biased, if ever?
What would you put in a model card for the bank's ethics board?

Reflection and evidence

Write a short model card for your final model: features used and why, complexity and regularisation, training vs public vs hidden accuracy, the fairness gap, and one thing you would change before this model decides real loans.

Students write 80–120 words on their phones. The Data tab exports every decision, result, prediction and reflection as one CSV for grading.

Console controls

Pause, extend or set the timer; step back a round
Draw, choose or write the event card
Move players, make a Lead, add computer players
Award or fine with a reason that appears on phones
Message every screen; hide names; spotlight a team
Keys: P projector mode, Q join QR