This study explored the effectiveness of a training workshop for animal-based measures (ABM) of welfare in dairy cows, in which 14 people were trained to evaluate 6 ABM: hock injuries (HOCK), lameness (LAME) in freestalls and tie-stalls, body condition score (BCS), and udder, flank, and leg cleanliness (CLEAN). All scoring systems were modified to a dichotomous outcome: acceptable (AC) or unacceptable (UN). AC hocks: no swelling and/or hair loss. UN hocks: swelling and/or scab. AC BCS: < 2, on a 5 point scale. UN BCS: ≥ 2. AC cleanliness: up to minor splashing. UN cleanliness: distinct plaques to a solid manure plaque. LAME was evaluated using locomotion score (LS) or in-stall lameness score (SLS), depending on cow housing. UN lameness: ≥ 3 LS, on a 1 to 5 scale, where 3 = mild lameness or ≥ 2 SLS, where 2 out of 4 behavioral indicators of lameness were detected. Classroom instruction took place on d 1 of training. Day 2: live group assessment of LAME (n = 25 cows), live group assessments of HOCK/CLEAN/BCS (n = 30 cows), followed by live individual assessment of HOCK/CLEAN/BCS (n = 20 cows). Day 3 included live individual assessment of HOCK/CLEAN/BCS (n = 33 cows), and individual video assessment of LAME (n = 27 cows). An additional training video for LAME was sent to trainees 3 weeks after the workshop, and another follow-up assessment of LAME took place via video (n = 37 cows). Repeatability and accuracy of the trainees was assessed using Fleiss’s Kappa (FK) and Byrt’s Kappa (BK) to examine group inter-rater agreement and expert-trainee agreement, respectively. Both Kappa systems use a scale of poor (<0), slight (0.01–0.20), fair (0.21 −0.40), moderate (0.41–0.61) substantial (0.61–0.80), or almost perfect (0.81–1.00). At the conclusion of the workshop, FK = 0.66 and 0.41 for HOCK and LAME. BK mean = 0.81 (from 0.63 to 1.00) and 0.62 (from 0.56 to 0.85) for HOCK and LAME. Each trainee achieved a BK of “almost perfect agreement” for BCS and CLEAN. After the follow-up video, trainees achieved a FK = 0.73 and a BK mean = 0.72 (from 0.63 to 0.78). Multiple assessors can achieve substantial agreement for ABM with adequate training.