A Qualitative Analysis of Open Textbook Reviews

Open textbooks are a type of Open Educational Resource (OER). They present educators with an alternative to commercial textbooks, afford students and educators permissions granted by open licenses, and reduce student costs. The purpose of this qualitative study is to examine how educators evaluate the quality of open textbooks. We analyzed 954 educator reviews of 235 unique open textbooks. American postsecondary educators authored the reviews between April 2014 and March 2017 and the Open Education Network (OEN; formerly the Open Textbook Network, https://open.umn.edu/otn/collected) and openly published the reviews in the Open Textbook Library (OTL, https://open.umn.edu/opentextbooks/), unedited and with Creative Commons Attribution 4.0 International licenses (CC BY 4.0). Overall, reviewers found the open textbooks to be of sufficient quality for use. The reviews provide insight into educator concerns and interests regarding the quality and characteristics of open textbooks and may support peer educators’ consideration, and authors’ and publishers’ creation and revision, of open textbooks.


INTRODUCTION
While for many students pursuing postsecondary education is a priority, the cost of higher education is often a barrier. The 2015-16 National Postsecondary Student Aid Study (2018) found that in the 2015-2016 school year, 72% of all undergraduates received some form of financial aid. The costs of textbooks and course materials increase students' higher education expenses and U.S. institutions suggested that students budget $1,002-$1,504 for books and supplies in 2017-2018 (The Institute for College Access & Success, 2019). While course material costs are only part of the significant expense students face pursuing higher education, these costs may negatively influence student behaviors and choices related to academic success.
The 2018 Student Textbook and Course Materials Survey (Office of Distance Learning & Student Services, 2019) found by surveying postsecondary students across Florida that "[t]he top 5 highest percentage answers reported by students when asked about the impact of textbook costs" were "not purchasing the required textbook; taking fewer courses; not registering for a specific course; earning a poor grade; and dropping a course" (p. 13). Postsecondary administrators and faculty appreciate that course materials costs are problematic and concerning (Seaman & Seaman, 2020), and in response to the Ithaka S+R US Faculty Survey 2018 (2019), seven in 10 faculty "indicated that reducing the cost that students pay for textbooks and other course materials is highly important" (p. 47).
Faculty members appear to be increasingly aware of Open Educational Resources (OER) as alternatives to commercial materials. The William and Flora Hewlett Foundation (2020) defines OER as: teaching, learning and research materials in any medium -digital or otherwise -that reside in the public domain or have been released under an open license that permits nocost access, use, adaptation and redistribution by others with no or limited restrictions. Seaman and Seaman (2020) found that 53% of faculty surveyed reported awareness of OER and In this report, we present the encouraging results from this analysis.

REVIEW OF RELEVANT LITERATURE
Research studies that examine or include postsecondary educator perceptions of OER are increasing in number. Hilton (2019) counted 29 studies concerning faculty or student perceptions of OER, published between 2002 and 2018 that met a set of criteria for inclusion. He commented that "[e]very study that has asked those who have used both OER and CT [commercial textbooks] as primary learning resources to directly compare the two has shown that a strong majority of participants report that OER are as good or better" (p. 17), with the important note that educators' positive perceptions of OER may be influenced by their sensitivity to student appreciation for no-cost OER as course materials. We set aside perceptions reports of educators who may or may not have used OER (Blankstein & Wolff-Eisenberg, 2019;Cardoso et al., 2019;Jaschik & Lederman, 2018;Seaman & Seaman, 2020) and review studies reporting postsecondary educators' perceptions after using OER or open textbooks were excluded from this study. The studies we review here reveal educators' motivations for exploring and adopting OER, their experiences using OER, and how educators perceive OER quality and efficacy. While there is a body of literature concerning textbook evaluation, it is expansive and addresses too wide a variety of values to examine in the current article. However, Bliss (2013) reviewed textbook evaluation research in his in-depth literature analysis and found that primary evaluation criteria included cost, sensitivity to diversity, content comprehensiveness and accuracy, readability, educational impact, inclusion of effective pedagogical aids, interaction, human interest, learnability, and usability.
A number of studies have examined the perceptions of educators after they have used OER in their teaching.  and Hilton, Robinson et al. (2013) each surveyed small numbers of faculty members (20 or less) who used OER in community college courses. In each study, a majority of faculty evaluated OER quality as equivalent to, or better than, the quality of commercial materials they had used. Delimont et al. (2016) interviewed 13 university instructors who received a grant to transition to open and alternative resources in their courses. All but one instructor preferred these resources to a commercial textbook. Jhangiani et al. (2016) surveyed 78 postsecondary educators, 77% of whom reported having used OER. Of those, 59% rated OER as comparable to or better than commercial materials. Abramovich and McBride (2018) surveyed 35 educators who used OER and 97% found the OER to be equally useful, or more useful, than commercial materials. Overall, these perception studies evidence positive perceptions of OER by educators, particularly those who have used OER in their teaching (Venegas Muggli & Westermann 2019).

Researchers have also investigated the perceptions of educators after educators have
taught with open textbooks and found that the majority had positive perceptions of these resources. Petrides et al. (2011) reported the perspectives of educators who used open textbooks found that adoption of open textbooks were influenced by "cost, content quality, and ease of use" (p. 43), but that reducing student costs was most influential; multiple sources contributed to the educators' perception of content quality; and perceived ease of use related to the digital format of the open textbooks and related possibilities for educators to edit and integrate the content with other resources. Bliss, Robinson et al. (2013) surveyed 58 community college educators who taught with open textbooks. The majority found the quality to be comparable to or better than that of commercial textbooks. Pitt (2015) reported findings from two surveys of 127 total educators who had used OpenStax textbooks and 70-80% reported that they would also use other OER. The California OER Council (2016) surveyed 16 postsecondary educators who had adopted open textbooks for their courses. Most rated the open textbooks as comparable to, or better than, the commercial textbooks used for the courses. Ozdemir and Hendricks (2017) reviewed 51 educator e-portfolios in which the educators described their use of open textbooks in postsecondary courses. The majority reported that "the quality of the textbooks was as good or better, than that of traditional textbooks" (p. 98) and that they had "overwhelmingly positive experiences with using open textbooks" (p. 110). Jung et al. (2017)  To our knowledge, only one published study has analyzed educator reviews included in the OTL. Whereas our study concerns the free responses included in these reviews, Fischer et al. It is encouraging that postsecondary educators who have used OER in their teaching have generally positive perceptions of OER quality. Continued research investigating educator perceptions of and experiences using OER, along with a growing body of studies investigating OER efficacy, may encourage more faculty to explore OER.

METHOD
Our initial data set consisted of 963 reviews of open textbooks that the OEN collected from educators at American universities and colleges between April 2014 and March 2017.
When we compiled the set of reviews, it comprised all of the reviews that the OEN collected during that time period and 69% of the 1,375 reviews were collected from OEN and then also included in the OTL. The OTL has since grown to include just over 2,000 reviews. We The reviews follow a standard format that the OEN provided to reviewers in the form of an online questionnaire. Reviewers wrote free responses to prompt questions for 10 areas of concern (see Table 1) and were invited to provide additional comments. While reviewers were also asked to provide a 5-point Likert scale rating for each area of concern, and the open textbook overall, our analysis considered only reviewers' free responses.

Reviews Questionnaire: Areas of Concern with Prompt Questions for Free Responses
Area of concern Prompt question Comprehensiveness Accuracy Please comment on the book's comprehensiveness. The text covers all areas and ideas of the subject appropriately and provides an effective index and/or glossary.
Please comment on the book's accuracy. Content is accurate, error-free and unbiased.

Relevance and longevity
Please comment on the book's relevance/longevity. Content is up-to-date, but not in a way that will quickly make the text obsolete within a short period of time. The text is written and/or arranged in such a way that necessary updates will be relatively easy and straightforward to implement.

Clarity
Please comment on the book's clarity. The text is written in lucid, accessible prose, and provides adequate context for any jargon/technical terminology used.

Consistency
Please comment on the book's consistency. The text is internally consistent in terms of terminology and framework.

Modularity
Please comment on the book's modularity. The text is easily and readily divisible into smaller reading sections that can be assigned at different points within the course (i.e., enormous blocks of text without subheadings should be avoided). The text should not be overly self-referential, and should be easily reorganized and realigned with various subunits of a course without presenting much disruption to the reader.
Organization, structure, and flow Please comment on the book's organization/structure/flow. The topics in the text are presented in a logical, clear fashion.

Interface
Please comment on the book's interface. The text is free of significant interface issues, including navigation problems, distortion of images/charts, and any other display features that may distract or confuse the reader.

Data Analysis
We used template analysis to code the reviewers' open responses in the qualitative data analysis software programs Dedoose (www.dedoose.com) and NVivo (www.qsrinternational.com/nvivo/home). Template analysis is a style and technique (King, 2014) where researchers extract themes, main ideas, and contextual information as latent content (Cassell & Symon, 2004). Researchers develop a "coding template, usually on the basis of a subset of the data, which is then applied to further data, revised and reapplied" (King, 2014, p. 2).
We developed and refined a thematic coding template by coding a subset of 400 of the 954 reviews in several, progressive stages. First, we independently coded a common subset of 20 reviews in order to independently generate draft templates. We then discussed our coding and draft templates and collapsed them into one revised template. Second, the revised template was used to independently code a common subset of 200 reviews that included the initial 20 reviews.
We discussed our coding and revised the template for use in all subsequent work. Third, we used the revised template to revise our coding of the 200 reviews and to independently code a new, common subset of 200 reviews. Fourth, we randomly selected 40 of the 400 reviews we had coded and assessed the consistency of our coding for these 40 reviews. We then identified and resolved coding disagreements across the larger subset of 400 reviews and divided between us, and independently coded, the remaining 554 reviews.

Our coding template included primary codes such as (+) Comprehensive and (-) Not
Comprehensive that were specific to each of the 10 areas of concern and the additional comments section in the reviews. We coded every reviewer response, in each area of concern, with one primary code for that area of concern to reflect reviewers' evaluation of the open textbook in that area of concern. In this paper, we address findings from our application of primary codes. Our template also included secondary codes we used to code comments addressing ideas and perspectives. Our application of secondary codes supports a separate analysis that is beyond the scope of this paper.

Rigor
We each analyzed the reviews. Through several successive stages, we independently coded a common set of 400 of the 954 open textbook reviews (a significant ratio) in order to attempt to account for individual researcher interpretation (Larsson, 1993;Scandura & Williams, 2000) of the reviewer comments. We came to consensus on independently coded subsets by comparing inconsistencies and resolving disagreements through discussion. This consensus coding (Larsson, 1993) informed our coding of the remaining reviews, which we divided between us and coded independently.

RESULTS
We organize the results of our coding by the reviews' 10 areas of concern. We summarize our results in tables. Where counts do not total 100%, it is because we excluded comments that did not respond to the area of concern.

Comprehensiveness
In Table 2, we summarize our primary coding of reviewer evaluations of content comprehensiveness. The overwhelming majority of reviews described the open textbook content as comprehensive (73%) or somewhat comprehensive (22.1%). Only 2.1% of reviewers described the content as not comprehensive and thus insufficiently comprehensive for use.

Coded Reviewer Comments on the Topic of Comprehensiveness
Code n % Example (+) Comprehensive 703 73.0 "This textbook is amazingly comprehensive" (Sylwester, 2016) (-) Somewhat comprehensive 213 22.1 "The book is quite comprehensive, and covers similar materials to other public speaking texts" (Crawford Barniskis, 2016) (-) Not comprehensive 21 2.1 "The textbook does not cover all the material one would need to address in college algebra" (Frankl, 2013) *Note. Counts do not total 100% because we excluded comments that did not respond to this area of concern.
Reviewers who found the textbook's comprehensiveness to be commendable often noted alignment with both course curricula and their own teaching, detailed topic coverage, and noted for example, "[t]his is a very comprehensive textbook that provides an appropriate balance between the different fields of biology" (Allen, 2015). Some reviewers commented that the textbook was exceptionally comprehensive as compared to commercial textbooks. "All the standard topics are there as well as additional material not found in most introductory physics books" (Papavasiliou, 2015). At times, reviewers related comprehensiveness to other content elements, including glossaries, exercises, assessments, assignments, discussion questions, and appended content: "The amount of material that is reviewed is awesome and useful. The index and other main components a textbook covers was [sic] accurate and meaningful" (Danielson, 2014).
In comments we coded as (-) Somewhat Comprehensive, reviewers' comparisons to commercial textbooks were still largely positive and the reviewers commented on minor flaws.
"It covers all the appropriate areas, but the coverage is a bit thin when it comes to examples" (Perry, 2016). Others noted content omissions and room for improvement. "This provides an excellent level of detail for a non-majors biology course. Only a couple of areas were lacking" (Ansley, 2016). Some reviewers noted content portions they found too basic or superficial. "The text is fairly comprehensive for an introductory level course, but it often lacks detail--even for an Intro text" (Addae, 2015). Reviewers often noted if a table of contents, index, or glossary was missing and that this detracted from overall comprehensiveness Reviewers often acknowledged that a single textbook-commercial or open-can rarely present an entirely comprehensive treatment of a topic and still be an effective learning resource.
Overall, reviewers overwhelmingly found the open textbooks they reviewed to be sufficiently comprehensive for use. Many made comparisons to specific commercial texts.
Reviewers who described a textbook as not comprehensive indicated that it could not stand alone as the primary course text, topic coverage was insufficient for their teaching or too superficial, coverage compared poorly to commercial textbooks, or missing topics were too significant an omission.

Accuracy
Overall, reviewers found the content of the open textbooks to be accurate. Table 3 summarizes our findings that 80.8% of reviews described the content as accurate, with only 15.2% describing the accuracy as somewhat flawed and 1.0% indicating that the content was not accurate.

Coded Reviewer Comments on the Topic of Content Accuracy
Code n % Example (+) Accurate 778 80.8 "As far as the science and economics are presented in an elementary fashion, there is little to be disputed in its accuracy" (Fithian, 2015) (-) Somewhat accurate 146 15.2 "The text is mostly accurate, especially the sections on probability and statistical distributions, but there are some puzzling gaffes" (Murtaugh, 2014) (-) Not accurate 10 1.0 "There are many overstated generalities, inaccuracies and incomplete descriptions of function through the text" (Wilson, 2017) *Note. Counts do not total 100% because we excluded comments that did not respond to this area of concern.
Reviewer comments evidenced careful scrutiny, as reviewers commented on the presence or absence of current, referenced sources, and noted errors while assessing the content as accurate overall. "The book is accurate and unbiased. The book is up to date and very well researched. There are virtually no errors" (Johnson, 2015). When commending content accuracy, reviewers still offered suggestions for additions or improvements. "The [book] provides a reliable guide to the musical periods and movements, personalities and forms it covers. It may be served with a chapter that details the beginning of western music and it's [sic] history prior to the Middle Ages" (Mulcahy, 2017). Comments specific to bias tended to be limited, with reviewers indicating that they did not detect bias, or identifying bias but without describing it as detrimental to the text.
Comments coded as (-) Somewhat Accurate generally described the content as accurate but were often more nuanced. "The content of the text is generally accurate but not uniformly.
There are many errors and the use of terminology that has since been changed due to new findings" (Sam-Yellowe, 2015). When highlighting minor errors, reviewers usually also noted the overall accuracy of the text's content, using qualifiers such as "mostly," "reasonably," "largely," "generally," and "average." Reviewers who found the textbook content inaccurate were frank: "The book contains serious errors and oversimplifications. For example, the assertion about jazz eighth notes on p.

Relevance & Longevity
The prompt for this section was complex compared to others, as it asked about both relevance and longevity (see Table 1). As Table 4 documents, the majority of reviews (81.2%) described the content as relevant, while a minority of reviews described the content as somewhat (sufficiently) relevant (14 %) or not relevant (2.5%).  (Lee, 2014) (-) Not relevant 24 2.5 "The text used an old version of the Project Management Body of Knowledge which limits its usefulness. The essential project management concepts are still valid but the latest research and trends aren't evident here" (Griep, 2016) *Note. Counts do not total 100% because we excluded comments that did not respond to this area of concern.
In line with the multifaceted prompt question, reviewers often commented on the currency of the content, the relevance of the content for a contemporary student audience, whether or why the content might grow dated in time, and how difficult it might be to update the content over time to ensure the textbook's continuing relevance and use. Reviewers indicated appreciation that many textbooks must strike a balance between enduring but also current content reflecting changes in the field of study, society, and culture: The climate section is so well written that I believe that this has at least 5 years of life in it, before it might become painfully obvious that it needs updating. The policy sections are written in a general enough way that I also think they will stay current. (Lajtha, 2014) Reviewers were generally optimistic that open textbooks would have reasonable longevity, updates would be feasible, and instructors could supplement dated content with current materials.
Comments we coded as (-) Somewhat Relevant often addressed issues such as the lack or currency of graphics, photographs, or references to current culture or information sources (e.g., blogs/videos); dated sources, statistics, cases, or examples; and the need to address new developments in a rapidly changing area of study. "The content mostly references experts and other texts from the 1990s and 2000s with most major references ending by 2007. It seems the text could use some updating on recent developments" (Carroll, 2016). Comments also indicated reviewer appreciation for the desirability but challenge of encompassing enduring content and current content vulnerable to rapid obsolescence: It deals relatively well with controversial topics such as deforestation and climate change.
Its treatment of such issues is current and up-to-date, but broad enough that the book will remain relevant in the short term to medium. It is less successful at presenting new themes, currents, and debates within cultural geography. The book's depiction of this subfield is somewhat outdated already (Williams, 2016) When describing the content as not relevant, reviewers related relevance to the coverage of important content but also to engaging students, sometimes speaking from their experience teaching with the open textbook. "There were not many current or recent developments included.
This made it particularly hard to engage the students" (K. . Some reviewers indicated that a lack of relevance rendered the textbook unusable. At the same time, reviewers often commented on how easy or difficult it would be to make the content relevant: "In short, this textbook is due for a major revision…. This revision would be a major undertaking and a challenge for the authors" (Mitra, 2017).
Overall, in the Relevance section, educators provided rich insights and observations that could inform the work of open textbook authors and publishers.

Clarity
The prompt for this segment asked educators to consider if the textbook content is "written in lucid, accessible prose, and provides adequate context for any jargon/technical terminology used" (see Table 1). Reviewers addressed the presence or absence of these characteristics, but also numerous qualities they perceived to be related, including the flow of ideas, level of language, and consistency of author voice. Table 5 summarizes our finding that 80.4% of reviews that described the text as clear. A minority (15.6%) of reviews found clarity somewhat lacking and only 3.6% of reviews indicated that the text was not clear.

Coded Reviewer Comments on the Topic of Clarity
Code n % Example (+) Clear 774 80.4 "This text is extremely and unusually well-written and clear. This is one of the text's greatest selling points" (Goren, 2014) (-) Somewhat clear 150 15.6 "For the majority of the content, the clarity is excellent. However, at times, I needed to read through the entire section, then revisit [sic] early paragraphs [sic] to get the entire message" (Colvin, 2014) (-) Not clear 25 3.6 "The organization of the text is laborious--both for student and instructor" (Richars, 2017) *Note. Counts do not total 100% because we excluded comments that did not respond to this area of concern.
Positive evaluations illustrated reviewer attention to elements perceived as contributing to clarity, including the level of language; humor and tone; the inclusion of context and definitions for terminology and clear explanations of key concepts; concision; supplementary media; text formatting; and the ability of the text to engage learners and communicate the content to specific or varied audiences: The book is written in a clear and easy-to-understand style that is adequate for those who are novice to educational psychology…. Although the book is written by two authors, it's hard to detect the difference between the authors' writing. (Koç, 2015) Even positive reviews often included suggestions for further improving the clarity of the text, elements such as formatting or the accessibility or tone of the writing.
When reviewers evaluated clarity as sufficient but flawed, they often indicated how clarity was lacking in specific instances or varied across the text, and how clarity could be improved: I like the way the text is written to be approachable for a wide variety of students. I think balancing chemical reactions could be done in a clearer way, as it is hard to tell which numbers are the co-efficients [sic]. (Smith, 2017) Reviewers also commented on relationships between the clarity of the text and a wide variety of elements such as long quotations, jargon, definitions, detail, inadequate chapter transitions, and non-ideal examples, which might detract from students' comprehension of what evaluators perceived to be generally clear content: This textbook is easy to follow and the inherent technical jargon of GIS is explained well.
Repetitive sentences and unnecessary phrasing, however, abound, and a few of the context examples the authors provide are too in-depth for an introductory textbook. (Widener, 2015) In comments we coded as (-) Not Clear, reviewers described similar issues detracting from clarity, but as so significant or prolific that these would negatively affect readers' comprehension. "Text was heavy, with lengthy meandering discussions on different approaches to a topic that were too in depth before even discussing the actual topic" (Matoush, 2017). These reviewers indicated frustration reading the text and concern that their students would experience even greater difficulty. Overall, however, reviewers found the text to be clear enough for classroom use.

Consistency
The prompt for this section pointed to the internal consistency of the text in terms of terminology and framework. Table 6 describes our finding that a large majority (83.2%) of reviewers described the open textbooks as consistent, while 12.7% described some weaknesses, and just 3.0% described the content as not consistent.

Coded Reviewer Comments on the Topic of Consistency
Code n % Example (+) Consistent 801 83.2 "Consistent terminology is used throughout. Even better, the terminology the author uses is consistent with the language of the cases and the rules. This will help to limit unnecessary student confusion" (Sherowski, 2015) (-) Somewhat consistent 122 12.7 "The framework of the book is internally consistent, though I think it takes on too much to have true consistency. Compared to similar books on the market for introductory textbooks I think it may cover a bit too much to provide real consistency" (Bell, 2017) (-) Not consistent 29 3.0 "Each chapter in this book was written by nine individual writers in what seems like an environment where collaboration was not emphasized. This means that each chapter is tonally very different from the others" (Weedman, 2017) *Note. Counts do not total 100% because we excluded comments that did not respond to this area of concern.
Reviewers who described the content as consistent highlighted organizational frameworks, consistent terminology, and relationships between these characteristics and reader comprehension. "The framework of the book is perhaps its greatest strength. The author has framed research concepts within the proper epistemological and ontological frameworks, which allows her even-handed treatment of qualitative and quantitative methods to cohere well within each section" (DeCarlo, 2017). They described consistency in terms of theoretical and pedagogical approach; language, writing style, and author voice; and formatting and layout.
When consistency was acceptable but lacking, reviewers related this to potential use or comprehension issues for educators or students: "The content in each chapter does match the content in other chapters, but the poor placement of the chapters only makes that consistency accessible if one were to really dig" (Harker, 2016).
Comments we coded as (-) Not Consistent indicated that effective use of the text could be significantly impeded by the absence of consistency. "The book felt like it was almost two separate books put together -which is part of why it can be considered so comprehensive" (Brown, 2015). These comments also suggested that the intellectual framework of a book may be communicated through structural and formatting elements, such as section divisions. "The text needs some work in terms of the consistency of its structure/framework. A less minimal approach to section/subsection headings would help" (Shapiro, 2017).

Modularity
Reviewer comments suggested high educator interest in modularity.   (Aspelund, 2015) (-) Not modular 21 2.2 "Modularity is not the best in the book. It takes time to explore and navigate through chapters. Once you are in a chapter then it's pretty well organised" (Bhargava, 2016) *Note. Counts do not total 100% because we excluded comments that did not respond to this area of concern.
Reviewers who found the open textbooks to be modular described this modularity. They indicated how chapters, sections, sidebars, exercises, and "key takeaways" sections organized and divided the content and made it possible to use the content in part or whole, or in a different order than presented by the text. They commented on how commendable modularity could be improved further; whether or not the text self-referenced; how the sequencing of the content compared to commercial textbooks with the same focus; and how the text's modularity related to the varied or consistent structure of associated courses: "This text is designed with modularity in mind…. In the instructor's information, the authors are even kind enough to include several sample syllabi with a variety of lengths and subject emphasis" (Rittenbach, 2017). Evaluators highlighted modularity as directly related to instructor integration. "The text materials can easily be divided into subunits suited to the instructor's purposes, who may easily pick and choose which materials to use or not to use" (Eubanks, 2017).
When reviewers indicated that a textbook was modular but not to the extent that it could be, they often indicated how and where modularity was lacking, and provided constructive suggestions for improving modularity: Modularity is important to me because I often like to cover topics in a little different order than is traditional. For example, I prefer to discuss conservation of momentum before Newton's laws. As with most texts, this text makes it difficult to do that as the chapter on conservation of momentum makes extensive reference to force. I feel that the modularity of this text is typical for the genre. (Rees, 2016) Reviewers noted it might not be possible to make modular use of a text that was not structured to be divided or reordered. "The sections and exercises have some modularity for utilization as stand-alone elements. However, as a whole the text builds from a foundation in theory and proceeds through increasingly complex methodological approaches making a reorganization challenging" (Raley, 2017).
When evaluating open textbooks as not modular, reviewers described teaching considerations, such as dividing the text into reasonable reading assignments for students and to align the text with lessons or class sessions. "Since the modularity is based on relatively topiccentered arrangement, reorganization and realignment of subunits does not seem easy to do" (Zuganelli, 2015).

Organization, Structure and Flow
Reviewers commented on organization in several sections of the reviews (e.g., Clarity and Modularity) but the prompt for this section was simple. "Please comment on the book's organization/structure/flow. The topics in the text are presented in a logical, clear fashion." Table   8 documents our findings that 78% of reviews described the open textbook as organized, 15% indicated the organization was flawed, and 3% described the content as not organized.

Coded Reviewer Comments on the Topic of Organization
Code n % Example (+) Organized 751 78.0 "The organization is fine. The book presents all the topics in an appropriate sequence" (Gorecki, 2016) (-) Somewhat organized 174 15.0 "I struggled with the flow of the content. I would like to see definitions and Cultural Intelligence model presented early in the text. I would also encourage chapters to be logically and clearly connected to your model" (Friedman, 2016) (-) Not organized 29 3.0 "The main weakness of the textbook is in the ordering of topics within chapters. It tends to jump from one topic to the next without a proper transition, e.g. Middle America chapter discusses the colonial era and then, afterwards, addresses Native American cultures as they existed before the colonial era" (Timms, 2014) *Note. Counts do not total 100% because we excluded comments that did not respond to this area of concern.
Reviewers who complemented the textbook's organization often described it as being a key attribute. "The organization of this book is one of its greatest strengths" (Pihlaja, 2017).
They compared the textbook's organization to that of related, commercial textbooks, made recommendations for further improving strong organization, and specified how the organization was logical and would likely serve learners, even if the content sequencing did not align with the reviewer's course curriculum. "I noted no issues with organization or structure, and the ordering of topics appears reasonable. The author's sequence is not identical to what I am used to teaching, but it appears logical, workable and perhaps superior" (Moore, 2016).
When reviewers found the organization to be sufficient but flawed or atypical, they indicated how it might be improved: Content in chapters 1 and 2 are thorough, but uneven in their treatment of topics and would benefit from reorganization…. The whole book would have benefited from having resources listed at the end of each chapter, in addition to being individual links on which the reader must click to examine. (White, 2017) Criticisms often related to the inconsistent quality of the logic, content organization, sequencing, and flow, and suggested that inconsistency affected the unity of the text overall.
Reviewers who described the organization as poor or lacking indicated how this could impede learning. "The organization of the text is very unusual. Air resistance is discussed in the chapter on forces…. This organization is detrimental to student learning" (Zurcher, 2015). They expressed considerable care and concern for student learning, and suggested ways the text could be improved or educators could mitigate these issues.

Interface
Reviewer comments included attention to navigation, images, charts, and display features. The majority of reviews (83.7%) positively described the open textbook as free of interface issues, while 10.8% described some issues and 1.5% indicated that the interface was insufficient (Table 9).

Table 9
Coded Reviewer Comments on the Interface Code n % Example (+) No interface issues 602 83.7 "The text is a plain pdf, and the images within it all look fine. No problems with the interface at all" (Fountain, 2015) (-) Some interface issues 254 10.8 "As I mentioned above, there is no index, glossary or table of contents…. As with many pdfs, the product on the screen is not as crisp as what a student might view in a physical book with nicer graphics" (Krutz, 2015) (-) Insufficient interface 14 1.5 "Very poor. There was very little use of color, pictures, and other graphics" (Trombley, 2016) *Note. Counts do not total 100% because we excluded comments that did not respond to this area of concern.
Positive evaluations of interface were generally straightforward, sometimes indicated that reviewers had tested multiple electronic formats of the text, such as PDF and EPUB, and often evidenced that reviewers had checked navigation and hyperlinks. "The book has a solid interface…. All of the links within the table of contents and hyperlinks [sic] within the text function" (Wilcox, 2016). Reviewers also commented on how interface facilitates and enhances a reader's interaction with textbook content. "Colors for text, fonts, [sic] headings are all appropriate and help to focus the reader's attention to what is truly important" (Gort, 2017).
When reviewers evaluated the interface as flawed, they commented on how this negatively affected their reading experience and offered suggestions for improvement. "The book had a great deal of white space and frequent blank pages…. A bit of work to improve the design and make the book more visually interesting (colors, less wasted space, etc.), certainly wouldn't hurt" (Maurer, 2017). Their comments reflect the extent to which they value and expect typical interface elements and affordances of electronic publication including text searchability and hyperlinked navigation. "At 700+ pages, there's no table of contents and little in the PDF that allows for quick and easy browsing without intense scrolling. I'd recommend a hyperlinked TOC" (Marx, 2016). Reviewers often described very specific interface flaws, such as the absence of page numbers, that authors and publishers could resolve in future editions.
When reviewers described the interface as insufficient, they drew connections to the ways in which visual elements, and text formatting such as bolding or italicizing, can engage readers and support comprehension and learning. "The textbook contains only words…. The lack of color, images, and charts may make it difficult for students to remain engaged" (Sanders, 2017). They also mentioned formatting errors and approaches that might distract or deter readers and negatively impact learning. "Simply put; there are just to [sic] many errors in equation (both chemical and mathematical) formatting to make this text useable [sic]" (Philbin, 2014).

Grammatical Errors
Table 10 documents our finding that the majority (81.6%) of reviews described the grammar as sound. Just 13.4% of reviews indicated limited or minor grammar issues, and only 1.5% described significant issues.

Coded Reviewer Comments on the Topic of Grammatical Errors
Code n % Example (+) No grammar issues 786 81.6 "The grammar is sound" (Bess, 2016) (-) Limited or minor grammar issues 129 13.4 "Written in a conversational, informal style the book is by and large free of grammatical errors. There are about a dozen minor mistakes, such as concatenated words or repeated words" (Anghel, 2017) (-) Significant grammar issues 14 1.5 "The book still needs some work in this regard. Pronouns don't always agree with the antecedents, and I noted several shifts in voice in the text" (Jenkins, 2017) *Note. Counts do not total 100% because we excluded comments that did not respond to this area of concern.
Comments describing sound grammar indicated the absence of issues or even commended the grammar. "The grammar is excellent. It is written in a scholarly format but does not confuse readers with undefined jargon or superfluous words" (Tusing, 2017). Reviewers were often inclined to make related comments concerning the extent to which the textbook's language was formal or conversational. "I have not noticed any grammatical errors. In terms of style, I would say that it is colloquial, friendly English. The material is certainly technical but there is a consultative, inviting [sic] tone behind the technical discussion" (Fowler, 2015).
When reviewers found the grammar to be flawed, they cited specific issues. "At times, sentences run on, with the sentence looking more like a paragraph and having multiple commas.
Otherwise, the writing looks clean, although at quite a high level" (Weimerskirch, 2014). They often qualified the scope and extent of grammar issues. "There are occasional text and grammatical errors found in the book. However, these have been quickly corrected in the online version when a report was submitted" (Tiffany, 2017). Reviewers who described significant grammar issues frequently expressed concern with finding these issues in a published textbook.
"The book is written as one might spontaneously talk. Grammar is not a top priority of this friendly style of writing. Words are used improperly and punctuation is sometimes used improperly" (Tullis, 2017). They expressed concern about using substandard text with students.
"As an instructor in an English department, I do not want to put a textbook in front of my students that does not model consistent and clean sentences at a level they should aim for" (Parker, 2016).

Cultural Relevance
Responses to this prompt were more diverse than in other review sections, which seems unsurprising given that reviewers may have diverse cultural perspectives and that cultural relevance may be more subjective than other measures of quality. Many comments seemed to reflect the influence of the prompt: reviewers often described the text as neutral, not insensitive, or not offensive. Table 11 documents the results of our primary coding, which found that 66.6% of reviews described the content as culturally relevant.

Coded Reviewer Comments on the Topic of Cultural Relevance
Code n % Example (+) Relevant 641 66.6 "This textbook is inclusive and comprehensive and is written in a respectful tone" (Kompelien, 2016) (-) Somewhat relevant 236 24.5 "There was not a strong emphasis on cultural relevance. On a positive note there were no issues with cultural insensitivity either. General psychology textbooks tend to be a bit culturally neutral, however there could be added cultural implications to the topics" (Shelton, 2017)  While many of the comments we coded as (+) Relevant described the textbook as neutral or not culturally insensitive, some applauded clear attention to diversity. "Barkan does an excellent job of addressing social problems, which by nature can be controversial, in a manner that is neither culturally insensitive nor offensive. Barkan uses examples that are diverse, multicultural, and inclusive" (Jones, 2015). Reviewers also noted examples of attention to issues of diversity and inclusion: Dr. Collins really opens the opportunity to have lengthy discussions about social inclusiveness--how even in a single country, social issues affect how businesses relate to their marketplace. It was among the first examples of this level of dialogue I've seen in a textbook targeted at introductory business. (Gore, 2015) Additionally, reviewers commented on a perceived absence of problematic bias.  (Kimmons, 2015), the educators have not also provided free responses that may illuminate their quantitative ratings. The free responses we analyzed help us understand what educators value in discrete areas of concern such as grammar and cultural relevance, and how educators perceived the quality of these open textbooks in each area of concern and overall.
Although the reviewers often found flaws with the textbooks, these were most often minor, the majority of evaluators' comments indicated that these flaws did not render the textbook unacceptable for adoption in teaching, and reviewers often asserted that these flaws could be corrected. Additionally, many reviewers stated that similar flaws are frequently present in commercial textbooks, and that educators need to account for flaws and imperfect curricular alignment when adopting any textbook. This corroborates results from other studies that perceived textbook quality is influenced by the context for use (Woodward et al., 2017).
Reviewers found the open textbooks to be more flawed in terms of organization, flow, and writing consistency. In comparison, commercial textbooks may more typically have a single author voice resulting from substantial professional editing. Nonetheless, reviewers found the modularity of the open textbooks to be strong (making it straightforward for educators to extract, or reorder students' use of, specific sections). The overall quality of a textbook may represent a balancing act between quality characteristics with negative relationships, where increased modularity results in decreased consistency, for example.
One unexpected observation was the variation in reviewer comments concerning the consumption of open textbooks in a digital format. Some reviewers expressed concern that a digital textbook may not support student learning as effectively as print. The digital format of a textbook has been shown to have little impact on learning (Rockinson-Szapkiw et al., 2013) and whereas students may wait to or never purchase commercial textbooks due to cost, students may

STUDY LIMITATIONS
While the results of this study are promising, some limitations and delimitations are noteworthy. First, the reviewers may have had more positive bias toward open textbooks than the general educator population. These reviewers self-selected to complete an OEN workshop about OER that may have positively influenced their perceptions and evaluations of open textbooks. In many cases, the educators received a small stipend from their institution to recognize the time and effort they invested in attending a workshop and authoring a review. The presence of this stipend may have led some educators to be kind in their reviews. However, reviewers were explicitly encouraged to be honest in their reviews; they were advised that their reviews would be published as-is to the OTL, without editing; and the reviewer responses include clear criticisms. Second, as this study is delimited to the American context, these evaluations may have limited international applicability. Third, as we are actively engaged and immersed in research and the community discourse concerning OER, we likely make assumptions, and have specific biases, about OER. We attempted to separate our personal assumptions and ensure reliability by conducting a significant extent of double coding, and by comparing our coding.
Nonetheless, our biases and perceptions concerning OER may have influenced our interpretation of the reviews as we coded. Our limited application of the ( interpretation of the reviewers' free responses is open to human interpretation, even given our efforts to adhere to best practices of qualitative research and accommodations to human error. To counteract this possibility, and to ensure trustworthiness and transparency of our work and results, we have described our work in detail and included direct quotations from reviewers.

IMPLICATIONS FOR FURTHER RESEARCH
Although we often use peer review as a proxy for quality, other measures, such as student performance after using a textbook, are often considered to be valid.

CONCLUSION
In this study, we analyzed educator reviews of open textbooks across 10 areas of concern.
The results of this study were encouraging: overall, reviewers found the open textbooks to be of