Achieving rigor and relevance in online multimedia scholarly publishing
First Monday

Achieving rigor and relevance in online multimedia scholarly publishing by Mark Anderson-Wilk and Jeff Hino

This paper discusses the importance of relevance and rigor in scholarly publishing in a new media–rich world. We defend that scholarship should be useful and engaging to audiences through the use of new media, and at the same time scholarly publishers must develop and maintain methods of ensuring content accuracy and providing quality controls in the production of scholarly multimedia products. We review examples and a case study of existing scholarly publishing venues that attempt to maintain quality control standards while embracing innovative multimedia formats. We also present lessons learned from the case experience and challenges that face us in the scholarly publication of multimedia.


The problem of rigor versus relevance and accessibility
Attempts at achieving rigor and relevance
Oregon State University case study
Lessons learned




Scholarly publishing has long served two interests — the interest of the publication’s audience, whether that be colleagues in the discipline or a broader audience, and the interest of the authors in obtaining the peer–review publication necessary for their promotion and tenure. In recent years, as the Web has matured with greater interactive, multimedia content, the needs and preferences of audiences have rapidly evolved. Audiences now seek to engage with information and other users in multimodal, multisensory ways.

Meanwhile, scholarly publishing, as a broad generalization, has taken only baby steps in this direction — moving from printed publications to online publications, from physical delivery to online delivery — without benefitting from the many engaging opportunities available through new media. As tenure–track scholars seek publication outlets, they naturally trend toward the venues (peer–reviewed journals) that provide them the greatest reward in their quest for promotion and tenure (LaBelle, et al., in press). The unintended consequence is that the other objective of scholarly publishing — to serve the audience’s needs and interests — has become even more neglected. As a result, scholarly publishing’s slowness to embrace new media may contribute to a proliferation of higher education’s reputation of being institutions of irrelevance. Ironically, many University faculty — especially those with Extension responsibilities — are under increasing pressure to move away from focusing solely on print publications, and begin to use new media in their education and outreach efforts.

This paper discusses the importance of both relevance — embracing new media formats — and rigor in scholarly publishing, and the tension between them. We review examples of existing scholarly publishing venues that attempt to maintain quality control standards while embracing innovative multimedia formats. Scholarly works must expand to live and be relevant in a world of viral YouTube videos, immersive game environments, and the massive “many–to–many” exchange of knowledge through social media.

We will present a case study of one scholarly publisher, Oregon State University Extension and Experiment Station Communications (EESC) as a model for bringing engaging multimedia into scholarly publishing while upholding the credibility and content accuracy ideals of higher education.



The problem of rigor versus relevance and accessibility

Whitworth and Friedman (2009) describe scholarly publishing as a “feudal academic knowledge exchange system, with trends like exclusivity, slowness, narrowness, conservatism, self–involvement and inaccessibility.” In their estimation, most methods of providing rigor and quality control in scholarship do so at the expense of relevance.

Establishing credibility is critical to the integrity of digital scholarship (Maron, et al., 2009). Peer review is the “primary avenue of quality assessment and control in the academic world” (Harley, et al., 2010), even though peer review, as commonly conducted, has been identified as a contributing factor to scholarly publishing’s limited social relevance (Scott, 2007). Recent surveys have shown that most academics still deeply value peer review and perceive it as indispensible to the rigors necessary of scholarship (Ware, 2008).

Ideally we want to achieve relevance without sacrificing rigor. Scholarship should be useful and engaging to audiences, and at the same time scholarly publishers must maintain methods of ensuring content accuracy and providing quality controls in the production of scholarly products.

An important first step would redefine online scholarship to embrace the audience–engagement potential of new media (Morris, 2009). Scholarship should not be limited to reproducing documents on the Web — “text under glass” — but rather should “take advantage of the capacities for rich multimedia and networked communication that the Internet offers” (McPherson, 2009). Adams, et al. (2005) suggest that a “diversified portfolio of scholarship” is necessary. This portfolio could include interactive Web sites, educational games, videos, and audio programs.

It also means defining and gaining acceptance of new criteria and metrics to evaluate these new media–based scholarly products when “citation–based metrics are no longer indicative of the relative importance of a given piece of scholarly work ... and new forms of peer review and approval ... are not yet recognized as significant” (Johnson, et al., 2010).

While these new media formats have potential to be more relevant and useful to audiences, as suggested by Whitworth and Friedman (2009), how can they be carried out in ways that maintain excellence in content, design, and execution? How can we design new systems of evaluation within a culture steeped in the tradition of peer–reviewed print–based publications? The following sections take a look at some existing examples for producing relevant and rigorous scholarship in new online multimedia formats, and a case study of an evolving effort at Oregon State University.



Attempts at achieving rigor and relevance

Current outlets for publishing online multimedia scholarship that would qualify for the rigors necessary for promotion and tenure are limited. However, a number of notable pilots and a few full–scale efforts have been made to deliver scholarship in innovative, engaging formats:

  • Wardrip–Fruin (2009) conducted an experiment to compare the quality of scholarly peer review conducted through a blog with that conducted traditionally. Though the work was primarily textual, the alternative media of the blog showed some advantages over traditional methods. Wardrip–Fruin found that though the blog approach had its limitations, it was successful in engaging a wider audience of peer reviewers and thus had potential to improve the application and relevance of the work.

  • Land–grant universities across the country have partnered on a large–scale Web site for public education scholarship, eXtension is an interactive online learning environment with content vetted by communities of practice (Lambur, 2007).

  • The University of Maine has made a concerted attempt to expand the definition of scholarship to include online multimedia efforts through a combination of pilot projects, policy development, and supportive scholarship that outlines the rationale for raising the quality and academic weight of new media (Ippolito, et al., 2009). A notable part of this effort is “the Pool,” a Web site ( the University’s Department of New Media developed for sharing and peer review of new media projects (Foster, 2008).

  • The Multimedia Educational Resource for Learning and Online Teaching (MERLOT, is a large–scale venue for online educational multimedia to be shared and peer reviewed (MERLOT, 2008). With over 20,000 peer–reviewed resources and significant user traffic, MERLOT is perhaps the most successful existing example of relevance and rigor in online scholarly multimedia publishing.

  • Vectors ( is an innovative and varied format scholarly journal (McPherson, 2009). The journal combines multimedia pieces with companion articles that discuss and provide context to the multimedia applications.

  • A number of venues such as SciVee ( and the Journal of Visualized Experiments ( also exist specifically for video scholarship. Online video, with its relatively long history and ubiquitousness as a consumer format, is perhaps the strongest candidate as multimedia format for scholarship.



Oregon State University case study

Extension and Experiment Station Communications (EESC, at Oregon State University recognized and responded to the creative tension that exists in the new media world and that of traditional scholarly publications, and developed standards and protocols for faculty to publish new media with the same degree of rigor and peer review as scholarly print publications. EESC is the scholarly publisher for the University’s Extension Service, the outreach arm of the University. Faculty wishing to publish content in the Oregon State University Extension Publications and Multimedia Catalog enter into a partnership with specialists at EESC who have multimedia design and production expertise.

The goal of EESC as publisher is to partner with faculty authors (called “co–creators’ in the context of multimedia creation) to complete a successful multimedia product that engages audiences and represents a rigorous scholarly effort. The key components of the EESC multimedia publishing system are as follows:

  • Faculty co–creators are responsible for providing content. Peer review is used to ensure the credibility of the content.
  • EESC specialists are responsible for ensuring that multimedia selection, design, and production meet standards of excellence.
  • When projects are accepted, faculty co–creators and EESC specialists enter into an agreement to follow the established publishing process and to each carry out their responsibilities as defined in the agreement.

This system addresses quality control in both content accuracy and multimedia production. The faculty co–creators are responsible for the quality of the content. The content is peer reviewed for accuracy and appropriateness for the audience and objectives.

To ensure that multimedia products have the same scholarly rigor and oversight, EESC requires that faculty co–creators develop scripts of the content to be included in multimedia formats and have it peer reviewed in the same way a manuscript would be reviewed before multimedia production begins.

The EESC specialist managing the multimedia project is responsible for quality control in media selection, instructional design, and production. Depending on the media type and the faculty co–creator’s proficiency with the multimedia, the faculty may share some of the design and production responsibilities, but the EESC specialist serves as the party accountable for meeting the standards for the specific medium.

A team of EESC multimedia specialists developed standards for each media type that EESC publishes. The standards consist of a very basic checklist for each media type. The checklists contain simple questions that can be answered yes or no by the qualified EESC specialist who is managing the project; if the answer is no, the EESC specialist works with the faculty co–creator and production team to improve the product such that the standards are met. Examples of the kinds of requirements addressed in the checklist standards are provided below.

For all multimedia:

  • Is this the best medium or delivery mechanism for this content?
  • Are the learning objectives clear?
  • It is compliant with ADA (Americans with Disabilities Act) Section 508 accessibility standards (photos with alt tags, videos with captioning, etc.)?
  • Is it compatible with commonly used browsers and mobile devices?
  • Are additional learning resources provided where appropriate?

For video:

  • Is the lighting sufficient?
  • Is the picture focused?
  • Is the camera work steady?
  • Is the audio clear and easy to hear, especially the voices?
  • Does the video tell a clear and engaging story?

For learning modules:

  • Is the size appropriate and manageable?
  • Is the navigation easy to follow?
  • Is the opening screen compelling?
  • Is the font type, size, and color appropriate for the audience and format?
  • Does it take good advantage of embedded video and sound?
  • Are images of sufficient quality and adequately captioned?
  • Is there cohesion between the different parts?

&lrquo;Grapevine nutrition” (Skinkis and Schreiner, 2011) ( is an example of a learning module which underwent our peer–review process for both content and multimedia approach. This interactive, nonlinear multimedia module targets a wide array of audiences, from undergraduates, to viticulture scientists, to wine industry professionals. The peer–reviewed content provides information on nutrient needs of grapevines, symptoms of deficiency and/or toxicity, and vineyard nutrient management. Meanwhile, the multimedia content was reviewed for interactive navigation design, media quality, and overall instructional design. The program has been highly praised by academic and industry user alike. It is our hope that, given the rigor of both content and multimedia peer review, that the module will be considered scholarly work in the promotion and tenure process for the authors.

The cooperative partnership between faculty co–creators and EESC specialists begins with a formal proposal to publish. Faculty co–creators complete an online proposal form that asks key questions about the potential project’s audience, purpose, and scope. A team of EESC specialists with a variety of expertise (editors, videographers, instructional designers, multimedia producers) meets to review proposals and discuss media selection and approach.

During this process, a qualified EESC specialist becomes project manager and reports back to the faculty co–creator with the team’s recommendations, including which media format would be most effective for engaging the intended audience. The project manager provides the co–creator a project plan that outlines the production process, the standards that must be met, and instructions that define which tasks are the responsibility of the faculty co–creator and which tasks are the responsibility of EESC.

We’ve found many faculty authors are confident about their content knowledge and their ability to write an effective publication manuscript, but may be hesitant or uncomfortable with new media types. So the EESC project manager empowers co–creators with what they do best — developing content pieces — and guides them with step–by–step instructions on the rest of the process.

EESC is careful not to guarantee the multimedia work it publishes will “count” for any level of credit in promotion and tenure. It is the administration and policies of an institution (not the publishers) that govern “what counts” and for how much in promotion and tenure reviews. However, the University administration has requested EESC to provide a detailed description of the processes, especially peer–review protocols that are followed. Clearly, to be “in the game” of scholarly publishing, it is in the interest of publishers to set and maintain consistent peer–review and production processes and quality control standards. If a publisher cannot clearly report or confirm the process required for works it publishes, the institution will not give any weight to the works it publishes, nor will faculty authors/co–creators have an incentive to publish with them.

The authors also freely admit that this process for applying rigor and relevance using a multimedia approach may not yet be a good fit in those instances where scholarship is strictly defined by the institution as by scholars for scholars. In many traditional peer–review models, relevance to non–scholarly audiences is not of concern: only that it be relevant to scholarly peers. Our model offers a best practice approach where the scholarship of outreach and engagement is encouraged, for example, in extension programs, or other service–oriented academic assignments.



Lessons learned

The following lessons have been drawn from EESC’s efforts to create a system for publishing online multimedia products that engage audiences and constitute scholarship at Oregon State University:

  • The publisher of multimedia scholarship must have a consistent, documentable process.
  • Multimedia content should be fully peer reviewed for accuracy with same process and rigor as for publications, and also reviewed with agreed–upon rubrics for multimedia presentation quality.
  • Media specialists must be involved in helping plan, guide, and review multimedia products to ensure that quality in multimedia selection, design, and production is achieved.
  • Faculty co–creators must be valued for the expertise they bring and assisted with the media–development pieces for which they may not have experience or expertise.

More efforts such as the one described here are necessary if scholarly publishing — and higher education in general — are going to escape the fate of irrelevance highlighted by Whitworth and Friedman (2009).




The evolving arena of online new media publication in a scholarly environment continues to challenge content creators, media specialists, and university administrators. The authors recognize the following questions:

  • How can the development of new media be encouraged if those products are not represented in the evaluation of scholarly success?
  • How can the process of peer review in new media progress if those who do the peer review are unfamiliar (or biased against) the use of new media in scholarly work?
  • How do promotion and tenure (P&T) committees apply metrics that don’t exist in typical P&T guidelines?

Through the examination, acceptance of new media guidelines and systems as outlined above, perhaps these hurdles to a more dynamic and engaging scholarly canon will be overcome. End of article


About the authors

Mark Anderson–Wilk is the Publishing Leader and Assistant Professor at Extension and Experiment Station Communications (EESC), Oregon State University.
E–mail: Mark.Anderson–Wilk [at] oregonstate [dot] edu

Jeff Hino is the Learning Technology Leader at EESC.
Direct comments to Jeff [dot] hino [at] oregonstate [dot] edu



R.G. Adams, Jr., R.M. Harrell, D.J. Maddy, and D. Weigel, 2005. “A diversified portfolio of scholarship: The making of a successful extension educator,” Journal of Extension, volume 43, number 4, 4COM2, at, accessed 3 June 2011.

Andrea L. Foster, 2008. “New–media scholars’ place in &lslquo;the Pool’ could lead to tenure,” Chronicle of Higher Education (30 May), volume 54, number 38, at, accessed 16 June 2011.

D. Harley, S.K. Acord, S. Earl–Novell, S. Lawrence, and C.J. King, 2010. Assessing the future landscape of scholarly communication: An exploration of faculty values and needs in seven disciplines. Berkeley: Center for Studies in Higher Education, University of California, Berkeley, at, accessed 1 December 2011.

Jon Ippolito, Joline Blais, Owen F. Smith, Steve Evans, and Nathan Stormer, 2009. “New criteria for new media,” Leonardo, volume 42, number 1, pp. 71–75, at, accessed 16 June 2011.

L. Johnson, A. Levine, R. Smith, and S. Stone, 2010. 2010 Horizon Report. Austin, Tex.: New Media Consortium, at, accessed 27 July 2011.

Chris LaBelle, Mark Anderson–Wilk, and Robert Emanuel, in press. “Leveraging new media in the scholarship of engagement: Opportunities and incentives,” Journal of Extension.

Michael Lambur, 2007. “The scholarship of eXtension,” at, accessed 3 June 2011.

Nancy L. Maron and K. Kirby Smith, 2009. “Current models of digital scholarly communication: Results of an investigation conducted by Ithaka Strategic Services for the Association of Research Libraries,” Journal of Electronic Publishing, volume 12, number 1, at, accessed 16 June 2011.

Tara McPherson, 2010. “Scaling Vectors: Thoughts on the future of scholarly communication,” Journal of Electronic Publishing, volume 13, number 2, at, accessed 16 June 2011.

MERLOT, 2008. “About MERLOT peer reviews of learning materials,” at, access 16 June 2011.

Sally Morris, 2009. “Is the journal article fit for purpose, or stuck in the past?” Learned Publishing, volume 22, number 1, pp. 3–4.

Alister Scott, 2007. “Peer review and the relevance of science,” Futures, volume 39, number 7, pp. 827–845.

Patricia Skinkis and R. Paul Schreiner, 2011. &lrquo;Grapevine nutrition,” at, accessed 13 October 2011.

Noah Wardrip–Fruin, 2009. “Blog–based peer review: Four surprises,” Grand Text Auto (12 May), at, accessed 16 June 2011.

Mark Ware, 2008. Peer review: Benefits, perceptions and alternatives. London: Publishing Research Consortium, at, accessed 1 December 2011.

Brian Whitworth and Rob Friedman, 2009. “Reinventing academic publishing online. Part I: Rigor, relevance and practice,” First Monday, volume 14, number 8, at, accessed 16 June 2011.


Editorial history

Received 8 September 2011; revised 20 October 2011; accepted 1 November 2011.

Creative Commons License
This work is in the Public Domain.

Achieving rigor and relevance in online multimedia scholarly publishing
by Mark Anderson–Wilk and Jeff Hino.
First Monday, Volume 16, Number 12 - 5 December 2011

A Great Cities Initiative of the University of Illinois at Chicago University Library.

© First Monday, 1995-2017. ISSN 1396-0466.