Designing a survey instrument for a national study of direct-pathway and returning engineering graduate students

Though a majority of engineering PhD students begin their doctoral career shortly aftercompleting an undergraduate degree (and perhaps a Master’s), a significant minority of studentsare “returners,” students who pursue a PhD after working outside of academia for five or moreyears. In the first phase of a three year NSF-funded study which aims to characterize thepopulation of returning engineering PhD students, explore the interactions of their previous workexperiences and their academic work, and investigate stakeholder views and institutional policiesrelated to returning PhD students, we developed a nationally-distributed survey to compareexperiences and perspectives of returners and traditional students.The survey development was grounded in Eccles’ Expectancy Value Theory (EVT), a theoreticalframework that explains how and why people make choices, based on the expected results ofthose choices, the costs required to make the choice, and their own interests and values. It wasalso informed by literature on returning students and by a pilot study that involved interviewswith ten returning students about their motivations for returning and their experiences duringgraduate school. The survey included questions about students’ motivation for returning, theirprevious work and school experience, their future career plans, the challenges of graduate school,and their strategies for adapting to these challenges.The development of our questionnaire included numerous levels of review and iteration. Ourresearch team created, discussed, and revised a list of questions based on our framework, pastresearch, and the personal experiences of current and former graduate students. After revisingseveral versions of the instrument, evaluating it for clarity, comprehensiveness, and relevance toour objectives, we asked our advisory board to review the survey. After again revising thesurvey based on our advisors’ feedback, we then piloted the survey with 7 current PhD studentsin STEM fields. We utilized a think-aloud cognitive interviewing technique to ensure thequestionnaire captured these students’ experiences as well as to check for the four components ofmeasurement error identified by Collins (2003), which include: comprehension problems,validity problems, processing difficulties, and pronunciation and communication difficulties.Participant feedback was carefully recorded and compiled in order to identify commondifficulties or suggestions. We then created the final draft of the survey based on this feedback.The focus of this paper and presentation will be on the development of the survey, in which wewill highlight best practices from the literature that informed our creation and refinement process.We will show iterations of the survey and data from the advisory board and our cognitiveinterviews that informed the final version of the instrument. We will also discuss the distributionof the instrument and some preliminary results from our national data collection effort.