Badges to Acknowledge Open Practices
•Download as PPTX, PDF•
2 likes•843 views
Presented at IASSIST 2014 meeting, Toronto, Canada.
Recommended
Recommended
More Related Content
What's hot
What's hot (13)
Viewers also liked
Viewers also liked (8)
Similar to Badges to Acknowledge Open Practices
Similar to Badges to Acknowledge Open Practices (20)
More from Andrew Sallans
More from Andrew Sallans (7)
Recently uploaded
Recently uploaded (20)
Badges to Acknowledge Open Practices
- 1. Badges to Acknowledge Open Practices Andrew Sallans Center for Open Science http://cos.io IASSIST June 2014
- 2. Openness is a core value of scientific practice
- 3. Challenges: Perceived norms Norms Communality Open Sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Counternorms Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity
- 4. Anderson, Martinson & DeVries, 2007
- 5. Challenges ● Perceived norms (Anderson, Martinson & DeVries, 2007) ● Motivated reasoning (Kunda, 1990) ● Minimal accountability (Lerner & Tetlock, 1999) ● I am busy (Everyone)
- 6. How might this change?
- 7. Top-down ● Committees ● Journal Editors ● Federal Agencies ● University administrators Bottom-up ● Exposing users to good practices ● Training sessions ● Workshops ● Spreading the message through active publishing ● Going to where the user is now ● Consultants serving as “Shock troops” Focus on simple interventions that researchers can actually adopt.
- 9. How to get a badge... 1. Disclosure --- author provides public statement that they’ve met badge criteria 1. Peer Reviewed --- independent review of author’s public statement and meeting of criteria
- 10. What are the criteria? ● Persistent path to the data, materials, pre registration, etc. ● Sufficient information for an independent person to reproduce the results ● Other more specialized items...
- 11. Who endorses these badges?
- 12. Which journals are adopting?
- 19. What comes next? ● Improve the badge issuing workflow ● Refine integration into peer review process ● Gain adoption of more journals/organizations ● Measure effect
- 20. Lastly, this is a community effort... Ben B. Blohowiak | Johanna Cohoon | Lee de-Wit | Eric Eich | Frank J. Farach | Roger Giner-Sorolla | Fred Hasselman | Alex O. Holcombe | Macartan Humphreys | Melissa Lewis | Brian A. Nosek | Jonathan Peirce | Andrew Sallans | Jeffrey R. Spies | Chris Seto | Sara Bowman Note: bold are non-COS
- 21. Thanks! Find out more and get involved here: https://osf.io/tvyxz/ Email me: andrew@cos.io Twitter: @asallans
Editor's Notes
- Merton’s norms of science (1942) - http://iie.fing.edu.uy/ense/asign/hciencia/trabs2001/victor/docs/merton.html We can understand the nature of the challenge with existing psychological theory. For example: 1. The goals and rewards of publishing are immediate and concrete; the rewards of getting it right are distal and abstract (Trope & Liberman) 2. I have beliefs, ideologies, and achievement motivations that influence how I interpret and report my research (motivated reasoning; Kunda, 1990). And, even if I am trying to resist this motivated reasoning. I may simply be unable to detect it in myself, even when I can see those biases in others. 3. And, what biases might influence me. Well, pick your favorite. My favorite in this context is the hindsight bias. 4. What’s more is we face these potential biases in a context of minimal accountability. What you know of my laboratory work is only what you get in the published report. … 5. Finally, even if I am prepared to accept that I have these biases and am motivated to address them so that I can get it right. I am busy. So are you. If I introduce a whole bunch of new things that I must now do to check and correct for my biases, I will kill my productivity and that of my collaborators. So, the incentives lead me to think that my best course of action is to just to the best I can and hope that I’m doing it okay. Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume
- 3,247 mid- and early-career scientists who had research funding from NIH. ideal to which most scientists subscribe scientists perceptions of their own behavior scientists perceptions of their peer’s behaviors self-regulation, substantiall autonomy, the complexity of scientific projects, professional expertise, innovative work on cutting-edge problems, and a system of largely voluntary compliance with regulation and codes of ethics all point to the futility and inadvisability of direct administrative control over scientists’ behavior Anderson MS, Martinson BC, De Vries R. Normative dissonance in science: Results from a national survey of U.S. scientists. Journal of Empirical Research on Human Research Ethics.2007;2(4):3–14.
- We can understand the nature of the challenge with existing psychological theory. For example: 1. The goals and rewards of publishing are immediate and concrete; the rewards of getting it right are distal and abstract (Trope & Liberman) 2. I have beliefs, ideologies, and achievement motivations that influence how I interpret and report my research (motivated reasoning; Kunda, 1990). And, even if I am trying to resist this motivated reasoning. I may simply be unable to detect it in myself, even when I can see those biases in others. 3. And, what biases might influence me. Well, pick your favorite. My favorite in this context is the hindsight bias. 4. What’s more is we face these potential biases in a context of minimal accountability. What you know of my laboratory work is only what you get in the published report. … 5. Finally, even if I am prepared to accept that I have these biases and am motivated to address them so that I can get it right. I am busy. So are you. If I introduce a whole bunch of new things that I must now do to check and correct for my biases, I will kill my productivity and that of my collaborators. So, the incentives lead me to think that my best course of action is to just to the best I can and hope that I’m doing it okay.
- think about very simple interventions that people can adopt top down versus bottom up community bottom up exposing users to good practices training, workshops, active in community with publishing, going to where the user is now, stat consultant “shock troops”, help reduce barriers and connect people to OSF tools that help them get there community top down committees, journals, feds, etc.