Helping Give Away
Psychological Science to
Improve Teaching Assessment
and Statistics:
What You Can Use,
What You Can Add
Eric Youngstrom, PhD & Mian-Li Ong, MA
University of North Carolina at Chapel Hill
Goals
 Mashing up
ROC Party = Better way of building evidence
Wikipedia = Making the evidence available to the
widest audience
Focus on Wikiversity
Goal:
Getting people involved,
organized to work effectively
(Joe’s question… who will write it?
Next action:
Visit site
Sign up!
ROC Underlying data
(same as t-test!)
Ready to ROC!
ROC Party
ROC as more clinically relevant technique
ROC Party as a collaborative teaching model
flipped classroom
 distance learning
Email [email protected] &
ask for “ROC Primer care package”
Google: Wikiversity + ROC
Resources attached to Primer
Article
(key deliverable for academic! Often the only…)
Goal = teaching, not new data findings
Redundant analyses (to be able to compare & contrast)
Syntax in SPSS & R
Open access to data used in examples
Excel sheet with formulae not in SPSS
We can do better!
ROC Party Rationale
Dissemination:
Not just for bipolar disorder
If I did anxiety, ADHD… so what?
If Tom Ollendick, Gerry Gioia did same…
people would notice!
If their team learned technique, that would be a multiplier
Teaching:
Experiential learning
Problem based learning
Could do small group, high structure, report out (flipped class)
Core ingredients
 Sample code, data, annotated figures and output
 In primer
 Online (DropBox OSF.IO Wikiversity)
 Agenda/syllabus
 From twinkle in eye to aims, methods, results, tables and figures
done in 6 meetings
 Vet the data ahead of time (planning canvas)
 Want to ensure success experience
 Increases chances of conference presentation, publication
 Provides incentive for participation
ROC Party I – Summer 2014
 Invitation only
 6 teams started projects (3 at UNC, in room, 3 long distance)
 6 meetings
 “Classes” -- GoogleHangouts/Skype/Phone only/WebEx
 Asynchronous shairing: Email, DropBox, OSF.IO
 Learnings:
 Seeing each other’s analyses provided a lot of exposure to different
scenarios
 Code sharing (how did you make that?)
 Fun!
 5 are published, one cancelled due to conceptual issues
ROC Party II
 Extended network of collaborators (Children’s National, OSU, NUS in Singapore,
University of Lancaster in UK)
 5 meetings instead of 6
 Continued hiccups with technology
 Again, lot of fun, lot learned
 Started to save process materials
(WebEx recording; annotated output; responses to peer reviews (Venkatraman test)
 Team structure:
 ROC Star: Experienced ROCer, can run or troubleshoot analyses, knows how to write up
Results
 Wingman: Understudy, may duplicate analyses – if external data, this often is a grad
student or postdoc familiar with the data
 Big Dawg: Content expert (will make sure Intro, Discussion are current and accurate)
 6 manuscripts published, 3 more still in process
Wikipedia:
“Best of the Free” Assessments
Write pages for free use tools that have good
score psychometrics across samples
 Link to copies of measures
 Solves Awareness and Access issues
 Supported by grants from SCCAP, APS, SSCP, APA CODAPAR &
D12
 Project Site: http://www.tinyurl.com/HGAPS2017
#1: Google
Reaching People
#5: Yahoo
#6: Baidu
#7: Wikipedia
GYMR
#150,000+: ABCT.org, AACAP.org, SCCAP.org
#150,000ish: EffectiveChildTherapy
(#1,000,000+: Youngstrom.anything)
Wikiversity:
EBA, Teaching and Clinical Application
Evidence-Based Assessment (EBA)
overview and concepts
Disorder-specific assessment toolkits
Teaching Vignettes
Research: Toolkit for EBA statistics
Ready to ROC
Helping Give Away
Psychological Science (HGAPS)
 New student organization to
 Teach people editing skills
 Critical thinking and evaluating evidence
 Helping increase the information about Psych Science available
online
 Like HGAPS on Facebook at
https://www.facebook.com/HGAPS/
 Follow us on Twitter at @_HGAPS
(audience participation!)
Wikiversity Resources to Teach ROC
(Google: Wikiversity ROC)
Organizing Small Groups
ROC Planning Canvas
(from Business School –
get for triage & consulting)
Wikipedia home page
#1 -- Talk page
Talk page
 Functions similar to the “comments” function in MS Word
 Adds ability to talk backchannel (within Wikipedia) by clicking user name
 Keeps dialog organized on threads (better than a subject line in email)
 Keeps discussion tightly linked to the content
#2 – View history
View history
 Functions similar to the “track changes” function in MS Word
 EVERYTHING done on Wikipedia is permanently archived & easily accessible
 Don’t need to worry about losing good information
 Disagreements? They get hashed out on Talk page (and there is an arbitration
process if needed)
 Better transparency than DSM, RDoC
 Keeps discussion tightly linked to the content
View history
 This is similar to “Compare Documents” in MS Word
 Better transparency than DSM, RDoC
 Keeps discussion tightly linked to the content
#3 – Account information
#3 – Account information
• Similar to setting up a home page on Facebook or
ResearchGate
• Add links to your other pages
(practice, university, whatever you want)
• When people “backchannel” you on Wikipedia, it
will be on your Talk page (but you can get alerts)
#4 – Watchlist
#4 – Watchlist
• Get notifications when edits
are made to page
• (You don’t need to keep
checking pages)
• Can watch people’s pages
as well as content pages
Those four tools let us make
effective changes to Wikipedia
Talk and View History to advocate and negotiate
change
Account as a way of building presence, credibility
(and linking back to real world work & life)
Watchlist as way of monitoring the things that you
care about (without it taking lots of extra steps)
Keeping it organized and
building small teams
Using WikiEducation “classroom” to build list
Use their teaching materials to orient people
who want to learn about Wikipedia
Build small teams assigned to high priority pages
Keeps automatic track of our progress
Our “Class” for the coming year

Dashboard provides real-time
feedback about progress!
Dashboard provides real-time
feedback about progress! 10-30-2016
EBA Assessment Template –
Wikiversity
What you can use
Any of the content on Wikiversity
Any of the teaching techniques or principles
Working on a Meta-Analysis party!
IRT Party?
MLM Party?
Could plug module into class
ROC in 5-6 meetings
What you can add
Email me suggestions ([email protected])
Add comments
Make edits
Add examples
Write new pages (HGAPS will consult)
What you can do now
Like HGAPS on Facebook at
https://www.facebook.com/HGAPS/
Follow us on Twitter at @_HGAPS
Thank you!