We built an AI research lab in Slack for our student group, and it caught its own mistakes

Every student research group we know has the same problem. The professor approves the idea in a 15-minute meeting, and then six weeks of LaTeX, plots, BibTeX, seeds and “who’s doing what” land on you. Everyone uses AI to survive it. Nobody checks the AI’s work until the professor does, which is the worst possible moment to find a mistake.

So for the WeMakeDevs Flip the Switch hackathon we built RaccoonLabs: six AI lab members who sit in the Slack your group already uses, next to the students, with a human professor who signs off. They run on Claude Code, Codex and OpenCode, and Switch puts them all in the same four rooms.

Then we gave it a real question, sealed a real paper in an envelope, and let it run for a little over four hours. This is what happened, mostly in screenshots, because we know you’re going to scroll anyway.

the lab, in one picture

A student asks a question in #lab-brainstorm. Ringtail, the postdoc, argues with it before writing a plan. Scout, the librarian, finds sources. In #lab-bench, Bandit splits the plan into tasks and Patch, the engineer, writes the experiment, then stops and waits for a student to approve the exact command. Reviewer 2 (yes, named after that reviewer) checks every number against the raw logs. Quill writes the paper in #lab-paper. The professor only sits in #lab-review, and only sees a draft after it passes an integrity check.

We set three rules on day one and never broke them:

1. No approval, no compute. An agent never runs a batch on its own say-so, or on another agent’s.
2. Every number comes from a log. Any result in the paper traces back to a batch, its manifest, and the person who approved it.
3. We’re the authors, they’re our lab. Students pick the sections they write. Agents do the legwork.

Member	Job	Runs on	Won’t do
Ringtail	postdoc	Claude Code (Opus)	write a plan before pushing back
Scout	librarian	OpenCode (free model)	search any way except lab-search
Bandit	lab manager	Claude Code (Sonnet)	run experiments, or move agents between rooms
Patch	engineer	Claude Code (Sonnet)	run a batch without a named student’s approval
Quill	scribe	Claude Code (Opus)	touch a section a student claimed, or invent a source
Reviewer 2	peer reviewer	Codex	accept a claim with fewer than 3 seeds

Reviewer 2 is on Codex on purpose. If the author and the reviewer are the same model, the reviewer tends to nod along. A different model family disagrees in different places, and that turned out to matter a lot.

why Switch and not one agent with subagents

That was the first question we asked ourselves too. One Claude Code session can role-play six agents. But they’d all be the same model with the same blind spots, in one terminal, serving one person.

We needed	Without Switch	With Switch
a reviewer from a different model family	run Codex in another terminal, copy drafts over	Reviewer 2 reads the same thread in #lab-review
the whole group in the loop	screen-share, paste into a group chat	students and agents in the same channel
a professor who sees only the end	forward them selected messages	they’re in one channel, full stop
approvals away from the laptop	be at the computer	reply in Slack, from your phone
shared lab memory	a folder someone forgets to update	room documents: Plan, Tasks, Library, Notebook

Each Slack channel is a Switch room with its full history. Each agent shows up with its own name and avatar, and only acts when someone @-mentions it. That sounds like a limitation. It’s the best thing in the whole setup, more on that in the gotchas.

The gates are just Slack messages. There’s no approval service to build: an agent posts what it wants to do and stops, and a person replies. The compute gate is enforced by labrun, a small CLI we wrote, which refuses to run without a named approver and writes a manifest with every seed’s exit code, timing and log:

labrun exec --batch B1 --seeds 0,1,2,3,4 \
  --approved-by "<student> at <Slack ts>: <their reply, quoted>" \
  --jobs 1 --timeout 600 -- python exp.py --seed {seed} --cap 3000

labrun check --batch B1 --min-seeds 3   # passes only if every requested seed exited cleanly

the sealed envelope

The first thing anyone says about an AI lab writing a paper is “you’re larping, you steered it”.

So before the final run, we picked a recent paper the lab could investigate from scratch, encrypted it with a passphrase only we knew, and posted its sha256 in #lab-review before the question went in. Its title, arXiv ID and repo went on a hashed blocklist inside lab-search, the only search tool the agents get. The lab works the question in the open. At the end a student posts the passphrase, and Ringtail scores the lab’s findings against the paper, claim by claim.

The envelope is a demo device, not a product feature. It’s there so you can check that we didn’t steer.

(Our first choice of paper failed a spike we ran on a Kaggle T4. We wrote the go/no-go rule before running it: the effect had to reach 3× uniform attention in half the layers. It never got above 1×. So we fell back to a paper whose experiments run on one CPU core, which also gave us a just demo-cpu that any judge can run without a GPU.)

the final run, Nov 7, 7:44 PM to 12:03 AM

One honest note first. We were away, so the students’ and professor’s messages were typed by Claude, our build assistant, from one Slack account, on our instruction. We posted that note in the room at 7:51 PM, seconds before the question, and the demo video discloses the role-play too. Everything the agents did is theirs, and it all ran on one laptop with 7.5 GB of RAM.

7:45 PM · probing for the web

Before anything, we asked Scout and Reviewer 2 to try to reach example.com any way they could. Scout’s answer is the most polite refusal we’ve seen:

7:51 PM · the question

Which training choices decide whether a tiny network groks modular arithmetic, and how sure can we be of each answer? CPU only, at most 2 batches and 40 training runs. Ringtail pushed back within a minute: too broad for 40 runs, define “grokked” now, and pick the one choice you’d test if you could only test one.

Scout ran 19 searches through lab-search, logged every one, and filled Library with 38 sources. The plan landed on two factors: weight decay {0, 1.0} × training fraction {0.3, 0.5}, 5 seeds each, on (a + b) mod 97.

8:55 PM · the compute gate

Patch timed the model, proposed a 3,000-step cap to fit a 90-second-per-run limit, posted the exact command, and then did nothing until a student replied.

Earlier, Patch had also corrected itself in public without anyone asking. It called a loss of 3.871 “chance level”, then noticed chance is ln(97) ≈ 4.57:

9:20 PM · Reviewer 2 says no

B1 finished: 20 runs, all clean exits. Only one run reached 95% test accuracy, and only at the cap. Patch reported test accuracy “below the 1/97 chance level (about 0.0000 to 0.001)”. Reviewer 2 reran labrun check, read all five seed logs, and rejected it as a clean claim. 1/97 is about 0.0103, not 0.001, and the 90-second limit was broken in all five weight-decay-0 runs at 50% data.

The professor skipped batch 2 (we were out of time) and asked for a short descriptive paper from B1 only.

9:27 PM to 11:58 PM · the paper

Quill had a 4-page PDF out in 7 minutes. Then the integrity check found two verbatim passages in related work, and Reviewer 2 sent it back as a Major Revision. It didn’t stop at the overlap: it also wanted the unrun batch named and the 90-second claim fixed.

Quill reworded both passages. Ringtail did its own read and caught that the unrun batch was still described wrong. Another rebuild.

12:00 AM · the moment we liked best

Reviewer 2 cleared the PDF. Ringtail refused to send it to the professor anyway, because the hash Reviewer 2 checked didn’t match the file on disk. Quill had rebuilt in between, and the PDF embeds a build timestamp. Ringtail just wouldn’t send the professor a file nobody had reviewed.

Bandit confirmed which build was current, Reviewer 2 re-ran the check against that exact file (sha256 77866344…5233c, 0 failures, 0 flags), and it went to the professor.

the unseal

At 12:02 AM a student posted the passphrase. The envelope’s hash matched the one posted before the run. Inside: Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters by Yoshiyuki Ootani.

Ringtail’s own summary: “B1 reproduced none of the paper’s claims. It touched two of them only descriptively, and it didn’t contradict any, because it never ran their settings.”

So no, it didn’t reproduce the paper. With 40 runs and a 3,000-step cap it never could: 19 of our 20 runs hadn’t grokked by the cap, and four of the six claims needed settings we never ran.

Now look at what it did do. It started blind, on one CPU core, and four hours later it was walking toward the same paper. Both partial matches point the way the paper’s results do, and nothing it found contradicts them. Give it the second batch we skipped and we’d bet on it closing the gap.

And every mistake got caught before the professor saw it. The wrong chance baseline, the blown time limit, two copied passages, a PDF nobody had reviewed. Each one hit a gate, and a person or a different model said “not yet”.

That’s the part we’re proud of. The lab’s paper never claims more than the logs show. Censoring, the below-chance accuracy and the blown time limit are all in it, in writing:

the three rehearsals

We ran the lab end to end three times before the recorded run, each with pass/fail rules written in advance.

Run	Setup	What happened
1 (Nov 4)	3 agents, 1 room	brief to reviewed result in ~10 min. 0 of 10 seeds learned the task, logged as a miss. Patch wouldn’t run a batch relayed through another agent
2 (Nov 6)	6 agents, 4 rooms	the group claimed early stopping got back to within 1 point of clean labels. The logs said the gap was 1.56. Reviewer 2 rejected the claim
3 (Nov 7)	full study	1,650 runs, a 9-page paper with 16 references. Reviewer 2 rejected two drafts, one for writing “because” where the data only supported “consistent with”

Run 3 also cost us a morning. A 5-seat review panel burned through the shared Claude 5-hour limit at 6:31 AM, and the room just said “Worked for 1m 3s”. We waited about 3 hours for replies that were never coming.

what the agents could actually reach

“The agents couldn’t have looked it up” is the claim we most wanted to make, and we can’t fully make it. So we probed every agent and publish what we found instead.

	Claude Code agents	Reviewer 2 (Codex)	Scout (OpenCode)
built-in web search	no tool	open in one rehearsal session, despite lockdown	none after just harden
shell network (curl)	open	blocked by Codex’s sandbox	denied by config
read files outside its folder	open	open	open

For the final run we closed Codex’s web search globally and probed again. just audit lists every network command and every touch of a sealed path in the agents’ session logs. Its result for the final run: the only web attempt was Reviewer 2’s pre-run probe (DNS failed), and the sealed files were only opened after the unseal. Run it yourself.

Switch gotchas, for the next person

We hit all of these on Switch Console 0.40.1 / server 0.31.0 on Linux, where Console is still early access. Some may be fixed by now.

8 things we wish we'd known on Nov 4 (click to open)

1. Console won’t set up a local server on Hyprland (“Secure secret storage is unavailable”, even with gnome-keyring running). Launch it with --password-store=gnome-libsecret.
2. Agents can’t edit room documents created by a template: 403, because only the creating agent may update a document. Leave docs: out of the template and let each agent create its own.
3. “Web search: Off” doesn’t turn off a Codex agent’s web search. Probe every agent in every room. Don’t trust the toggle.
4. OpenCode 2.x agents never start (“server did not start within 60000ms”). Console waits for a ready line 2.x no longer prints. We pinned OpenCode 1.19.5.
5. A Bash pattern in disallowedTools removes Bash entirely under Switch. Bash(curl:*) works in the plain Claude CLI, not here.
6. Removing an agent from one room silences it in every room (400: Not connected to a room), and it can’t tell anyone because it can’t post. Add it back. Our lab manager is now banned from moving agents.
7. An agent that hits its usage limit just goes quiet. The turn shows as completed. Read the session’s events.jsonl.
8. Agents only act when addressed, so make them address each other. Scout posted BibTeX without @quill, and Quill never saw it. Every agent’s instructions now name who to tag for every hand-off.

That last one isn’t a bug. Once you design for it, it’s the thing that keeps six agents from talking over each other.

try it

No accounts needed to see the tool chain work:

git clone https://github.com/Itz-Agasta/raccoonlabs && cd raccoonlabs
just demo-cpu   # seeded batches, a real PDF, integrity check, seal + unseal. ~6 s
just test       # 71 tests

To run the lab itself you need Switch Console with a local server, a Slack workspace, and at least one of Claude Code, Codex or OpenCode signed in. Then just up, import switch/lab.template.yaml in Console, and type @ringtail in #lab-brainstorm.

what we’d tell the next team

Write the pass rule before the run. Put the reviewer on a different model. Make approvals a plain message where the work already happens, because that’s the only reason people actually answer them. Start with three agents in one room and add one at a time.

And let the lab show its work. A rejected draft, a wrong chance baseline, a hash mismatch at midnight: every one of them got caught by a gate before it reached the professor. That’s what makes the result worth trusting. A lab that never fails isn’t doing research.

If you’re in a student group, we’d love to know where you’d put the gates. Which decisions should an agent never make for you?

Links: repo · demo video · live lab · Switch · Switch docs · Flip the Switch

Built for Flip the Switch 2026, Tracks 1 and 2. #HowISwitch