In the field of adult psychology, online testing has sped up and democratized data collection, expanded the scope of questions we can answer, and provided a new form of “convenience” sampling that’s far more representative than relying on undergraduates or even the volunteers who respond to local ads. Research with children stands to benefit even more, given the disproportionate difficulty of bringing them into the lab, yet developmental psychology has been slower to embrace online testing, largely because of technical hurdles and the particular demands of our discipline.
Online research could help alleviate many of the practical barriers that limit the scope and quality of developmental research. If we could run studies with kids in their homes anywhere in the world, we could reduce the degree of sampling bias inherent in lab testing, in addition to recruiting families with rare diagnoses and being able to observe more natural interaction. If we could bypass the bottleneck of recruiting and testing participants, we could more easily measure graded effects and look at change over time or individual differences. And—most importantly but least flashily—we could adequately power our studies so that other researchers could productively build on our work.
In what started as a small side project, I’ve been working on a platform called Lookit that allows developmental researchers to conduct online studies for babies and children. Families participate from home, on their own computers and within their own schedules. Parents can select studies from a variety of labs. To participate, they complete a short activity in the web browser with their child, and webcam video of the child’s responses is sent back to Lookit. We plan to open the platform to other researchers in May, and I welcome researchers to consider joining us.
The Lookit Platform: Current Status and Capabilities
Back in 2015, as a PhD student studying cognitive development at MIT, I completed several test studies on a simple Lookit prototype. These studies established that it was possible to collect developmental data online and code dependent measures, such as preferential looking from the video collected, as reliably as in the lab (Scott & Schulz, 2017). Since then, with a second grant from NSF’s Developmental and Learning Sciences program to expand access to Lookit, I’ve been focusing on scaling the platform to support multiple researchers running their own studies independently, initially in collaboration with the Center for Open Science, and now with our own in-house software engineer.
Lookit is built using Django, a widely used Python-based web framework that provides solid general-purpose functionality, such as a login system. From the researcher interface, researchers can create, edit, and try out studies; start and stop data collection; manage collaborators’ access; confirm consent; download child, demographic, and session data and video; and contact participants.
The Lookit platform fits into a growing ecosystem of tools in related but complementary spaces: Yale’s TheChildLab.com project, for instance, has developed techniques to run scheduled live-interaction studies with children via webcam. Platforms such as JSPsych, LabJS, PsyToolkit, Testable, PsychStudio, and Gorilla offer functionality for building and/or hosting (generally adult) online behavioral experiments to run in the web browser. Prolific, Testable Minds, and Amazon Mechanical Turk offer (adult) participant recruitment and management. Our own custom experiment builder within Lookit has allowed us to support a variety of developmental tasks, which are often more complex than adult test trials (e.g., including more audio and/or video instructions, custom animation, interactive elements); however, as other systems mature, we look forward to being able to integrate other options for experiment creation.
With Lookit under active development, we’ve accepted a limited number of collaborative beta-test studies to make sure we’re focusing on what matters to our eventual users (both researchers and participants). We now have 10 studies from eight institutions in various stages of completion—four actively testing, three completed, and three in preparation at the time of writing. Over the past year alone, more than 800 families have registered, and children from 41 US states speaking 20 different languages have participated. Data quality is excellent; the platform’s video quality and general reliability alike have improved substantially since initial studies using the prototype. Additional comparisons with in-lab data, and the first novel results, are coming soon from our beta testers.
Launch and Vision
We’re targeting May 2020 to officially “launch” Lookit, making it available free of charge to anyone to use for their own research. (Stay up to date on our progress toward launch via Github Projects: github.com/orgs/lookit/projects/.) Researchers will need to sign a general institutional agreement with MIT, developed by MIT’s lawyers and modeled after the Databrary agreement, that allows researchers to access and contribute to a shared repository of video from developmental studies. Researchers will also be responsible for obtaining approval from the institutional review board (IRB) at their own institutions.
Our aim is to provide infrastructure that enables a wide range of researchers to creatively and productively address their own questions. As more researchers run studies on the platform, we hope they will give back by participating actively in the community: trying out one another’s studies, offering advice, adding to the documentation, and filing bug reports and feature requests to provide input on development.
While all of Lookit’s code is open source and publicly available, there’s substantial benefit to researchers in banding together to use and support a particular instantiation of such a tool. We all benefit from economies of scale in hosting, feature development and software maintenance, and IRB and legal coordination. And we can make Lookit a place with constantly refreshing interesting content, which will help with recruiting and engaging participants.
Unique Challenges of Collecting Developmental Data Online
While the basic idea behind Lookit is straightforward, it’s taken the better part of a decade to get from an idea to a platform. There have certainly been some project-management and administrative hurdles—we are, after all, proposing to let other people collect video of children on the internet using MIT infrastructure. But for the most part, we’ve come up with ways to handle the challenges of creating and administering a platform for the field at large. Here are some of the solutions we’ve been working on:
Collecting video in the home
Collecting video online and in participants’ homes introduces a few ethical and privacy considerations beyond those of typical in-lab videotaping. As we plan to scale up, a lot of our effort has been centered on building safeguards for responsible handling of participant data.
First, we need to be extremely certain that all families agree to webcam videotaping and that we appropriately handle those cases in which we cannot be certain. Unlike in the lab, it’s possible for someone to end up on Lookit and try to “click through” the study out of curiosity without understanding what’s going on. Families on Lookit consent to participate by making a recorded verbal or signed statement, which ensures they can understand written English and know they’re being recorded. To minimize the potential for human error in the consent process, a “consent manager” view allows researchers to confirm consent videos in the web browser. That confirmation is stored in the database, and only responses with confirmed consent are available to view or download.
Second, video recording in the home is more likely to incidentally capture other family members, personal information, and uncensored interaction. At the end of all Lookit studies, a standard “exit survey” component lets participants select acceptable uses for their video, including whether it can be shared with Databrary. Parents also have the (very rarely used) opportunity to withdraw video from the study altogether. Withdrawn video is automatically made unavailable and deleted from our servers. Providing privacy options at the end rather than the beginning of the study allows parents to make an informed decision about sharing video on the basis of what happened throughout the study.
Supporting data analysis and sharing workflows while minimizing potential for unintended disclosure
One of our central engineering goals is to make it as easy as possible to download, analyze, and share data replicably and responsibly. Researchers can download CSV files with study data as well as automatically generated data dictionaries. Potentially identifying information, such as a child’s birthdate or exact age at time of participation, is technically accessible, but by default it is omitted from response data downloads. By setting this default behavior, we “nudge” researchers toward planning for eventual data sharing from the start by keeping sensitive data separate. To avoid each lab having to reinvent the wheel for common workflows, Lookit also provides processed data that can be published without redaction. For instance, researchers can download rounded ages as well as child IDs unique to their own studies to avoid unanticipated combination of data across studies.
A shared reputation
One of the biggest remaining hurdles is scaling online recruitment. Developmental labs have built up substantial expertise around recruiting in-person participants, but they generally have very little experience recruiting online participants. We have had a few undergraduate students experiment with organic and paid social-media advertising; our primary conclusion is that we should not expect the platform to magically attract thousands more parents without concerted—and skilled—efforts toward advertising and media outreach. On the bright side, recruitment is empirically not a zero-sum game for the multiple researchers using Lookit: Advertising one study has substantial benefits for the other studies on the site.
Tips for Translating Your Study to an Online Environment
Focus on communication
In my experience, the biggest challenge developmental researchers face in moving studies onto Lookit is nontechnical: communicating clearly with parents despite the noninteractive setting. Although we’re great at making studies intuitive and clear to young children, we’re used to explaining the goals of and instructions for a study to parents in a social context where we can adapt on the basis of their cues and questions.
Trying to cover these same bases without an interactive setting is hard. It’s worth the time, however, to really polish parent-facing study overviews, debriefings, and instruction text and/or verbal instructions. Even the study design itself may need a bit more signposting. Transitions that might be fine in the lab—for instance, proceeding from the last experimental test trial to the exit survey and debriefing—can feel more abrupt online.
Make it fun!
Online, families will be implicitly comparing a study to a kids’ app or video game (by comparison, the closest equivalent for a lab visit is a trip to the doctor!). Although it’s not realistic to expect a developmental experiment to be quite as engaging as a top-selling app, sometimes delivery or stimuli are drier than they have any scientific need to be, just out of habit.
When researchers record audio or video, they should ask themselves these questions: Is there actually any scientific need for a flat delivery? Will adding silly pages or a fun-to-say chorus to your storybook actually invalidate the interpretation of kids’ answers? Is there a reason parents can’t have an active role in asking questions or giving feedback?
Think creatively about the value for families
For families deciding whether to participate, the intangible rewards can be just as important as the compensation provided. We’re all just getting started figuring out what the possibilities are online, but here are some nonmonetary “rewards” to consider:
- A cute video still from the study. For example, our study that involved dense sampling produced a video “collage” of clips from the 12 sessions.
- Insight into a child’s particular behavior or unique strengths
- Some social connection via study “feedback” on the platform, even if it’s just your opinion that a particular child’s mind is fascinating
- Personal advice or encouragement about common parenting questions related to your research, or customized references to educational or parenting resources
Interested in Joining Us?
Again, we’re planning to fully open up the platform in May. But there are a few steps researchers can take ahead of time to hit the ground running.
Do I have to be a “techie”?
The short answer is no. Researchers can create an experiment without writing a single line of code; they do not need to know any programming languages or to have run studies online before. Study specification isn’t drag-and-drop (yet), but what needs to be edited to build a study is a simple, human-readable text document that says what frames to use in what order.
That said, it’s important for researchers to read the documentation and keep an open mind about their own abilities, such as not assuming that an error message is beyond them to figure out. Running experiments with babies on the internet might by definition create a techie person.
Will it work for my study? What dependent measures can I collect?
In terms of study protocol, researchers can do most of what one can imagine happening in a web browser. It’s actually easier to focus on what isn’t likely to work yet: live coding of looking measures; precise (frame-level) timing of videotaped responses, such as in some predictive-looking studies; strict control over the environment (e.g., precisely controlling lighting in the room or distance from the screen); and making live experimenter interaction or particular bespoke objects key to the experimental design.
Some studies may not be a good fit for the platform because of its shared reputation. For instance, we don’t recommend exposing preschoolers to “the debate” about climate change or making a study that is deliberately frustrating for parents.
Currently, our beta testers are collecting or planning to collect data from newborns through teenagers. They’re using looking measures, verbal or pointing responses, laughter, surveys, and performance on custom games. If researchers are not sure if something’s possible, they can just ask us!
A Platform for Everyone
At this point, we’ve only scratched the surface of the possibilities for online developmental research: Initial beta-testers have largely focused on direct adaptations of in-lab studies to verify that their basic protocols will work. I’m especially eager to start supporting the applications I haven’t yet imagined and the (potential) researchers who wouldn’t otherwise be running developmental studies at all because they lack resources.
I hope you’ll join us this spring!
Here’s How to Get Started
Step 1: Get your legal ducks in a row
The first step if you’re interested in using Lookit is to ask your institution to sign the Lookit Access Agreement. You can find the agreement and more information at github.com/lookit/researchresources/wiki/IRB-and-legal-information.
We recommend getting started on this piece right away. It should only take a few minutes, but your institution may take a few months to return it. You just need to email the form to someone at your institution who can act as an “authorized institutional representative.”
Step 2: Learn to set up your study
The best way to get up to speed on Lookit is to complete the tutorial included in the documentation at lookit.readthedocs.io/en/develop/tutorial.html. You’ll learn how to access the Lookit platform and community; get comfortable editing and troubleshooting an example study; walk through setting up a realistic study from scratch; and practice coding consent and interpreting downloaded data. You’ll even learn to add to the documentation yourself (I know, thrilling!), in case you find something missing.
References and Resources
Scott, K., & Schulz, L. (2017). Lookit (part 1): A new online platform for developmental research. Open Mind, 1(1), 4-14.
Lookit wiki: github.com/lookit/research-resources/wiki
Lookit documentation and tutorial: lookit.readthedocs.io/en/develop/
Lookit codebase and development plans: github.com/lookit/