Abstract
Objective. To determine the long-term change in the Altmetric Attention Score (AAS) and its components, as well as the impact of higher AAS on citation count for articles published in major pharmacy journals.
Methods. This study evaluated articles from pharmacy journals ranked in the top 10% according to their AAS in the year 2017. Correlation between the AAS and number of citations (through November 2020) was assessed using the Spearman’s correlation test. A Kruskal-Wallis 1-way analysis of variance was used to compare the AAS across journals.
Results. The median three-year AAS and citation count per article was 20 (25th, 75th percentile=15, 28) and 11 (6, 18), respectively. Between November 2018 and November 2020, there was no significant change in the median AAS for the 137 included articles. The only change in the AAS components was an increase in the number of Mendeley readers (22 [13, 34]). The median number of citations per article also increased (8 [4, 14]). We found a significant association between the three-year AAS and the three-year number of citations. The three-year number of Mendeley readers was associated with an increase in the 3-year number of citations. The mean three-year AAS was highest with articles published in the Journal of the American Pharmacists Association, while the mean three-year number of citations was highest for articles published in Pharmacotherapy.
Conclusion. Higher AAS scores appear to be associated with the number of citations for articles published in major pharmacy journals within three years of publication.
INTRODUCTION
Publishing in peer-reviewed journals is the primary process by which research is evaluated by one’s peers to determine its scientific merit.1 It is also an important consideration when making promotion and tenure decisions for faculty.2 The evaluation of a faculty’s scholarship is generally based on the quantity and quality of the articles published. The quality of peer-reviewed publications is typically dependent on the journal impact factor (JIF); however, the JIF largely reflects the reputation of the journal rather than that of its authors.3 The raw number of citations an article receives may also be considered an important metric, but the value of this metric is limited as there is large variation among disciplines as to what is a significant number of citations for an article. The Hirsch index, or h index, has largely been touted as a metric that evaluates both quantity and quality since it reports the number of articles that have been cited h number of times.4 For example, an author with 15 publications, each cited 15 times, would have an h index of 15. Of course, this favors more senior authors who have had more time to publish articles and accrue citations.
In addition to traditional metrics, there is an interest in recognizing online publications. Alternative metrics was coined as a term in 2010 and refers to many types of attention a published article receives online, including but not limited to mentions on social media platforms (eg, Twitter), blogs, Wikipedia, and news outlets.3 Journals have largely embraced alternative metrics and primarily use either Altmetric or Plum Analytics to source, aggregate, and report alternative metrics. One difference between these two platforms is that Altmetric provides a score, while Plum Analytics does not.5 The Altmetric Attention Score (AAS) serves as a quick and easy indicator of the amount and type of attention an article receives online. The score is based on an automated algorithm representing a weighted count of how much attention the article receives.6 More weight is given to news outlets, blog posts, policy documents, patents, and Wikipedia, while less weight is given to mentions on social media platforms. Further, more prominent news outlets with a national reach (eg, New York Times) are weighted more heavily than local news. A colored donut shape around the score represents the sources of attention. The author of the online post also matters as more credit is given when peers share your work than when the journal automatically shares a link to the article. Although alternative metrics have not replaced traditional measures of scholarship quality, perhaps they can serve as a complement to highlight the attention one’s scholarship receives online through social media and news outlets, which is not measured by traditional metrics.3
An area of uncertainty around alternative metrics is how effective they are at increasing citation counts, which affects both the JIF and the author’s h-index, and are still widely accepted measures of scholarship quality.7 Traditionally, one might only find an article if it was identified through a PubMed search, but more researchers are now sharing their articles online, especially through social media. This broader reach to the public and the scientific community could translate into more citations for articles that are shared and discussed online. There is little evidence regarding whether the AAS contributes to higher citation counts among pharmacy practice journals. Our own analysis of major pharmacy practice journals found no short-term correlation between the AAS and citation count.8 Therefore, the objective of this study was to determine if the AAS is associated with citation counts over an extended period of time in select major pharmacy practice journals.
METHODS
Our previous study8 included articles published in 2017 from pharmacy practice journals indexed in Web of Science or Scopus with a 2017 JIF of at least 1.9 Journals considered to focus primarily on international pharmacy practice or pharmacy education were excluded. We extracted data for all articles that ranked in the top 10% for each journal according to their AAS using Dimensions (as collected in November 2018), an online platform that provides citation counts and altmetrics data in aggregate form.10 Each article was reviewed to categorize its article type, study design, topic, and whether it was an open access article. We updated the AAS and relevant citation data for each included article through November 2020, providing up to three years of information from date of original publication. Sources of attention for each articles AAS were obtained from the Altmetric website.
Both AAS and the number of citations were log-transformed given skewed distributions. We used linear regression to evaluate the relationship between AAS and number of citations. The Kruskal-Wallis test estimated differences in AAS and number of citations across journals. We also explored non-linear associations using polynomial regression with 10-fold repeated cross-validation to identify the optimal model (highest explained variance, lowest mean squared error). We then conducted multiple regression to identify predictors of number of citations. Variables to include in the models were identified using least absolute shrinkage and selection operator (LASSO) regression.11 The optimal value for the tuning parameter (λ) to optimize model performance was selected using 10-fold repeated cross-validation. We performed all analyses using SAS 9.4 (SAS Institute) and R 4.0.3 (The R Foundation, https://www.r-project.org/) using the “caret” and “glmnet” packages, with a p value <.05 considered statistically significant.
RESULTS
Characteristics of the included articles have been published previously.8 As shown in Table 1, the median 3-year AAS and number of citations per article was 20 (25th, 75th percentile=15, 28) and 11 (6, 18), respectively. While many AAS components remained unchanged over time, including the median AAS, the median number of citations per article increased by 8 (4, 14) and the number of Mendeley readers increased by 22 (13, 34).
Long-term Characteristics of the Top 10% of Pharmacy Articles According to Altmetric Attention Score (N=137)
An increase in the log-transformed three-year AAS was associated with an increase in the log-transformed three-year number of citations (adjR2=0.06, p=.005) (Figure 1). Because the component of the AAS with the largest change from one-year to three-year follow-up was the number of Mendeley readers, we additionally examined its relationship with citation counts. An association between increasing log-transformed three-year number of Mendeley readers and an increase in the log-transformed three-year number of citations (adjR2=0.36, p <.001) is shown in Figure 2. The mean three-year AAS was highest with articles published in the Journal of the American Pharmacists Association (JAPhA) compared with other journals, with Research in Social and Administrative Pharmacy being the lowest (p<.0001). The mean three-year number of citations was highest for articles published in Pharmacotherapy, with American Journal of Health-System Pharmacy (AJHP) being the lowest (p=.013).
Relationship between log-transformed 3-year Altmetric Attention Score and log-transformed 3-year number of citations.
Relationship between log-transformed three-year number of Mendeley readers and log-transformed three-year number of citations.
The LASSO model identified the following predictors, which we then included in the multiple regression: log-transformed number of citations: log-transformed AAS, AJHP journal, Pharmacotherapy journal, opioid epidemic topic, pharmacy education topic, miscellaneous topic, systematic review/meta-analysis design, and case-report design. Predictors (coefficient intercept [95% CI]) associated with a higher log-transformed number of citations included log-transformed AAS (0.178 [0.0007 to 0.356]), Pharmacotherapy journal (0.595 [0.148 to 1.043]), opioid epidemic topic (0.385 [−0.099 to 0.870]), and systematic review/meta-analysis design (0.167 [−0.293 to 0.628). Those (coefficient intercept) associated with a lower log-transformed number of citations included AJHP journal (−0.229 [−0.616 to 0.157]), pharmacy education topic (−0.432 [−1.389 to 0.524]), miscellaneous topic (−0.144 [−0.682 to 0.394]), and case-control design (−0.770 [−1.915 to 0.374]).
DISCUSSION
Contrary to our previous study,8 a longer follow-up period resulted in an association between the AAS and the number of citations for articles published in major pharmacy journals in 2017. This longer follow-up period likely allowed time for more citations to accrue, resulting in a greater chance of identifying an association. We also observed no significant change in the AAS during this longer follow-up period, and Mendeley readers was the only component of the AAS associated with higher citation counts. While studies of journals from other disciplines have observed similar associations,12⇓⇓-15 we believe this is the first time such an association has been identified among pharmacy journals.
The sharing of articles online seems to occur primarily around the time the article is originally published. This is reasonable given that journals and authors are probably more likely to share newly published work rather than older articles. As such, we were not surprised to see the AAS did not change over the course of this longer follow-up period. Likewise, this phenomenon may also explain why we did not observe an association between the AAS and number of citations in our short-term analysis.8 When individuals see an article posted on Twitter or another platform, it is likely saved or tagged for later reading or use in their own work, which may not be published for months (or years) later resulting in a delay in the article shared online receiving a citation.
Given the AAS is based on multiple components, we explored which of these components were associated with higher citation counts. The only AAS component associated with higher citation counts were articles captured by Mendeley readers. Mendeley is a free reference manager with a built-in academic social network where individuals can share articles or entire reference libraries. As such, articles saved or shared by Mendeley readers were more likely to be cited since saving an article in a reference manager may suggest the user intended to cite the same article in a future publication of their own. While other reference managers exist (eg, EndNote), Mendeley is the only one captured by the AAS since it has a built-in social network.
In addition to Mendeley, Twitter has become a frequently used platform to share articles online.16 Although we did not find an association between Twitter promotion and citation counts, two large, randomized studies17,18 of articles published in cardiology and thoracic surgery journals demonstrated that Twitter promotion was associated with a significant increase in citation counts. However, neither study controlled for authors who may have had a significant online presence and self-promoted their published articles. The lack of an association between Twitter promotion and citation counts in pharmacy journals may be due to variability in each pharmacy journals’ presence and reach on Twitter, available support staff to manage the account, and approach to using Twitter to promote articles. Nevertheless, the body of evidence largely favors promoting published articles via Twitter to increase citation counts. Pharmacy journals may need to enhance their presence on Twitter or consider a dedicated Social Media Editor.
Our analysis also has implications for faculty and academic institutions. Faculty should consider sharing their published work online through social media. This may be particularly helpful to junior faculty who are trying to establish themselves and build their professional network. In addition to sharing at the time of publication, faculty may consider re-sharing their relevant articles in response to queries from their online peers or current events. There also appears to be a role for alternative metrics in the promotion and tenure process, and institutions should consider incorporating language around the role of alternative metrics in their promotion and tenure policies. Regardless, faculty who want to include alternative metrics in promotion and tenure dossiers may need to explain what they are and understand how they will be perceived by their department chair and institution.
CONCLUSION
There is increasing evidence that the AAS, which captures the attention an article receives online, is associated with increased citation counts. Until now, there has been no evidence to support such an association among articles published in pharmacy practice journals. Although our previous work showed no association within the short-term, we did find an association between the AAS and citation counts during a three-year follow-up period. Additional study is warranted to determine whether promotion on specific platforms (eg, Twitter) contributes to higher citation counts. There is now some evidence that authors and pharmacy journals should consider promotion of their articles online with the goal of raising the articles AAS score, which may result in future citations.
ACKNOWLEDGMENTS
The authors thank Madeleine Wagner for assisting with data collection.
- Received February 19, 2021.
- Accepted May 25, 2021.
- © 2022 American Association of Colleges of Pharmacy
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.