Faculty Pedagogy Resources

The intuitive moves toward AI-resistant assessment — opening the question up, making it reflective, pushing it toward theory — are close to where the machine is strongest. An earlier version of our FAQ advised that AI struggles with multi-layer reasoning; the Spring 2026 data did not bear that out.

If you are redesigning an exam to be harder for AI, run the redesign through a model before you rely on it. The durable principles are not a snapshot of this year's weaknesses: assess what you intrinsically value, assess the human-plus-AI system rather than the unaided human, and favor process over one-shot product.

Offline models run locally with no internet connection at all — LM Studio makes dozens of open-weight models free and easy to install. Through 2025 and 2026 these narrowed the gap with frontier cloud models substantially, and capable versions now run on the laptops our students already carry.

Examplify's “no internet” setting was always the softer of the two options, and it is softer now. If your goal is genuinely to prevent AI use during an in-class exam, “secure” mode is the setting that does that work.

Secure mode blocks other files and applications on the student's laptop. Students cannot reach outlines or class notes, cannot copy and paste from anything, and cannot open e-book casebooks they have purchased.

That last one catches people. Publishers push students hard toward e-books, which are often cheaper. Unless you say otherwise, many students will buy the e-book for your course and find out later that they cannot use it on even an open-book exam. If you plan an open-book in-class exam in secure mode, tell them at the start of the semester.

Students are watching this too. Student Affairs has seen an uptick in chatter about impermissible AI use during exams, including suggestions that some students have used locally run models. None of it has been substantiated, but the concern is real and worth weighing when you choose an exam format.

Do not treat an AI-detector score as proof of misconduct. Performance varies across tools and testing conditions, including authorship type, and studies have documented false-positive disparities for non-native English writing in specific datasets. At most, a score may prompt ordinary inquiry under established academic-integrity procedures. See Research and limitations for the evidence.

Faculty graders in our study flagged machine-written answers in eleven of nineteen blind cells — but most of the early detections rode on formatting artifacts rather than the writing: single spacing where the proctoring software double-spaces, a different font, reserved ID numbers sitting at the end of a sorted list. Those are fixable bugs. What survived was subtler: two independently generated answers to the same exam often read as though one hand wrote both, and the prose carried a polish that time-pressured student work does not.

You can require that any AI use be disclosed, and set parameters on what is permitted — no AI for creating blocks of text, but AI allowed for refining or editing student-written text. That distinction is useful for non-native English speakers in particular.

You can also run your own questions through a leading model and hand those sample answers to students during the exam. It defuses anxiety: everyone sees what the machine produces and proceeds accordingly. This works whether or not you allow AI use.

Starting points to adapt. For syllabus-level policy language and Professor Struve's attribution-and-disclosure model, use the policy templates and the AI Syllabus Guide, which are kept current.

• “You may not use any generative AI tools, such as ChatGPT, Claude, Gemini, Copilot and the like in any way on this exam.”
• “You may use generative AI tools to assist you in brainstorming about how to answer the exam questions. You may not use it to craft the language with which you write your answer.”
• “For your reference, I have run each of the questions on the exam through [AI model]. The resulting output is provided in unedited form in the Generative AI Supplement to this exam. I make no representations about the accuracy or quality of the generative AI output.”
• “Any use of generative AI tools must be disclosed at the end of your answer, immediately after the word count disclosure. Words used in an AI disclosure statement do not count towards the word limit.”

The trajectory. GPT-3.5 passed real Minnesota Law exams at a C+ level in 2023. GPT-4 was reported at roughly the top decile of the Uniform Bar Exam that year, though later work showed that figure was overstated by the choice of comparison sample. Three years later a model tops real class curves.

A possible plateau. A Maryland group that has run their own exams against successive models found in June 2026 that the current frontier model did not clearly outperform the prior year's, despite more inference-time compute. Small sample, so I would not plan around it — but I would stop assuming a step change every spring.

Beyond exams. In a Stanford-led study, sixteen Contracts professors across fourteen schools made 2,918 blind forced-choice comparisons between human and AI answers to student questions. They preferred the AI answer about 75% of the time, and flagged AI answers as harmful 3.5% of the time against 12% for the human ones.

The reliability counterpoint. Commercial legal research tools still hallucinate, and general-purpose models hallucinate legal material at high rates. Strong exam performance is not general reliability.

Any timed assessment of less than 24 hours, whether in-class or takeaway. Even ungraded assessments formally require them — a student may request accommodations on a timed practice midterm, though in practice many do not.

Students have also begun requesting and receiving accommodations on in-class work: timed quizzes, short in-class writing. Expect more of this as the overall numbers grow.

These vary considerably and are set by Disability Services, not the Law School. The common ones:

• 50 percent additional time (1.5x the listed limit);
• 100 percent additional time (2x the listed limit);
• Additional time for “stop-the-clock” breaks, administered as 30 minutes of extra time;
• A “distraction reduced setting”;
• Text-to-speech (reading) software;
• Speech-to-text (dictation) software.

They are frequently combined — 100 percent extra time, plus 30 minutes for breaks, plus a distraction reduced setting is a realistic package.

From the actual limit in your instructions. This is the most common place faculty are caught out. If you tell students the exam should take three hours but set a five-hour limit so everyone has room, accommodated students get extra time on the five.

Separately timed parts are each treated as their own exam. A one-hour multiple choice section plus a two-hour essay section, with a 50 percent accommodation, becomes 90 minutes and three hours respectively.

If the calculated limit exceeds eight hours, the student may request to take the exam over two consecutive days — eight hours the first, the balance the next. For in-class exams, Academic Affairs holds the student's materials and laptop overnight.

Most students receiving it sit with a small number of other accommodated students, all offered cardboard privacy screens. All students, accommodated or not, are offered earplugs during in-class exams. Some students receive a private room when Disability Services specifically determines it — a small number to date, though it may rise.

The practical consequence for exam design: in some cases a student takes the exam entirely or nearly alone, without peers or proctors. The Honor Code applies and there is no particular reason to think those students violate it, but it is worth keeping that setting in mind when you decide on closed book or no electronics.

Implementing this system takes an enormous amount of staff time and puts real strain on proctoring and facilities. Two things make a large difference:

• Be deliberate about time limits. Some students receive 1.5x your listed limit, others 2x. Anything whose accommodated total runs past eight hours may spill across two days.
• Give plenty of lead time. Try not to change limits at the last moment, and flag timed practice exams early so accommodations can be arranged if students request them.

Questions about a specific exam go to Claire Wallace (cwallac2@law.upenn.edu).