Using ScienceDirect APIs for Institutional Repositories

Introduction

Institutional Repositories are systems for collecting, preserving, disseminating and promoting the intellectual output of an institution. One of the most common objectives for institutional repositories is to promote the organization's research and teaching activities.

A major challenge for the managers of IR systems is making sure that all of the institute's research output is captured, up-to-date, accompanied with the right metadata and made available in line with Publisher Policies.

With the ScienceDirect APIs Elsevier aims to support institutional repositories with the following:

Repository objectives & use cases    How this objective can be addressed using Elsevier ScienceDirect APIs
Ensure your institutional repository reports on all articles affiliated authors published with Elsevier. Automate periodic downloads of structured metadata (including abstracts) for articles your authors published with Elsevier.
Ensure locally hosted full-text (green OA) accepted manuscripts of Elsevier articles are made available in line with publisher's hosting & posting policies, with minimum hassle. Responses of the ScienceDirect search API include the embargo end date, the date by which an AM may be made publicly available.
Ensure repository users are linked the to them best available version. Use the article DOI or PII to check if the user has access on ScienceDirect based on their IP, and if so, link the user to the full text of the article on ScienceDirect. Show links to locally hosted earlier versions in case the user does not have access to the final article on ScienceDirect.
Keep users on your repository site to discover and use more of the content available on the repository.

Immediately show the final published article to all users who have full-text access, and the first page for everyone, even when no version available in the IR itself.

Ensure readership metrics are captured and made available to authors.
Embed PDF of final articles on your repository. Users who have access based on IP address will be shown the final article; other users will see a first page preview. This API usage will be made available in reports for subscription customers and to authors.

Process for getting access to ScienceDirect APIs

Elsevier RESTful APIs provide access to ScienceDirect data to allow developers working on IRs systems to write programs that automatically extract data from ScienceDirect periodically, and add that data to their systems. The ScienceDirect data that is available for use in IRs via APIs is subject to the Elsevier content use policies.

Getting access to the ScienceDirect APIs for institutional repositories takes few simple steps:

  1. Register for an API key here.

  2. Register for the institutional repository program here and submit your API key: https://www.elsevier.com/solutions/sciencedirect/forms/institutional-repository-managers?rURL=Direct. The Elsevier integration support team will add the ScienceDirect IR settings to your API key, and confirm back. If your institution already has a Scopus API key, you can submit this API key to have the settings for the SD IR services added.

  3. Develop your IR software. Detailed technical documentation on the ScienceDirect API integration can be found here. Institutions using DSpace open source repository software can download the plugins for DSpace we have developed. For more information visit (https://github.com/atmire/Elsevier).

Instructions for using APIs for the supported use cases


Note: API keys used in this document are example API keys. Please make sure to replace them with your API key.


1. Identifying publications affiliated with your institution


Initial repository population with article metadata


Important Note: We recommend Repositories from institutions that subscribe to Scopus to use the Scopus Search API to retrieve cross-publisher metadata. For more information please see Scopus IR/CRIS/VIVO How To Guide.


The ScienceDirect Search API (interactive documentation is available here, detailed interface documentation is available here) can be leveraged to identify articles that are written by authors that are affiliated with your institution. In order to do that you can construct a field restricted search by affiliation as follows:

http://api.elsevier.com/content/search/scidir?query=aff%28broad%20institute%29&httpAccept=application/xml&apiKey=[my_api_key]
where If the institution has changed names over the years, the API request can be constructed to accommodate name variants in the affiliation search as follows:

http://api.elsevier.com/content/search/scidir?query=aff%28broad%20institute%20MIT%29%20or%20aff%28broad%20institute%20Harvard%29%20or%20aff%28broad%20institute%29&httpAccept=application/xml&apiKey=[my_api_key]
The number of search results that can be retrieved via a single API request is anywhere from 25 to 200, depending on institutional access rights and API service levels. For most institutions this is sufficient to retrieve articles published in the last month, which appears a suitable frequency to keep the repository updated. Institutions that would like to retrieve more results, for example because they would like to download metadata for articles published over a longer period, we suggest to narrow down the result set by using publication year restriction. For example: the repository aims to host articles published after year 2000. In this case, the API requests for initial repository population should contain pub-date year restriction for each year such as 2000, 2002, 2003, ..., 2014, 2015:

http://api.elsevier.com/content/search/scidir?query=aff%28broad%20institute%29%20and%20pub-date%20is%202014&httpAccept=application/xml&apiKey=[my_api_key]

Note that date restrictions can also be made for explicit dates, either matching or within a particular range. For example, the comparsions for pub-date can utilize the date format YYYY-MM-DD when using the operators BEF (before) and AFT (after), which can be used to denote a publication date prior to or after the specified date(s) such as pub-date AFT 2014-06-30 AND pub-date BEF 2014-10-01 for all articles in the third quarter of 2014.

http://api.elsevier.com/content/search/scidir?query=aff%28broad%20institute%29%20and%20pub-date%20AFT%202014-06-30%20AND%20pub-date%20BEF%202014-10-01&httpAccept=application/xml&apiKey=[my_api_key]

Note that these operators BEF and AFT are only applicable for specific fields, such as dates.
If you are planning to make use of article abstracts, you will need to specify view=COMPLETE on the API request as shown below:

http://api.elsevier.com/content/search/scidir?query=aff%28broad%20institute%29%20and%20pub-date%20is%202014&view=COMPLETE&httpAccept=application/xml&apiKey=[my_api_key]

Please check if search returned non-zero results by examining <opensearch:totalResults> counts. To programmatically iterate over search results set please use URL in href element of the <link ref="next" provided in the search API response. Here is example:

<opensearch:totalResults>123</opensearch:totalResults>
<opensearch:startIndex>0</opensearch:startIndex>
<opensearch:itemsPerPage>25</opensearch:itemsPerPage>
<opensearch:Query role="request" searchTerms="aff%28broad+institute%29+and+pub-date+is+2014" startPage="0"/>
<link ref="self" href="http://api.elsevier.com:80/content/search/scidir?start=0&count=25&query=aff%28broad+institute%29+and+pub-date+is+2014&httpAccept=application/xml&apiKey=[my_api_key]" type="application/xml"/>
<link ref="first" href="http://api.elsevier.com:80/content/search/scidir?start=0&count=25&query=aff%28broad+institute%29+and+pub-date+is+2014&httpAccept=application/xml&apiKey=[my_api_key]" type="application/xml"/>
<link ref="next" href="http://api.elsevier.com:80/content/search/scidir?start=25&count=25&query=aff%28broad+institute%29+and+pub-date+is+2014&httpAccept=application/xml&apiKey=[my_api_key]" type="application/xml"/>
<link ref="last" href="http://api.elsevier.com:80/content/search/scidir?start=98&count=25&query=aff%28broad+institute%29+and+pub-date+is+2014&httpAccept=application/xml&apiKey=[my_api_key]" type="application/xml"/>

Update repository with newly published article metadata


Upon completing initial repository population, the update process can use the following API request including a search for pub-date after the last year used for initial repository population:

http://api.elsevier.com/content/search/scidir?query=aff%28broad%20institute%29%20and%20pub-date%20aft%202014&view=COMPLETE&httpAccept=application/xml&apiKey=[my_api_key]

Metadata


We suggest storing metadata about Elsevier article, including article identifier called PII and a link to ScienceDirect article (returned in the ScienceDirect Search API response), when populating institutional repository. PII will be used to call Article Entitlement Retrieval API and Article Retrieval API as described later in this document.

Example or PII and ScienceDirect link returned in ScienceDirect Search API response:

<pii>S0049-3848(14)00284-9</pii>
<link ref="self" href="http://api.elsevier.com/content/article/pii/S0049384814002849"/>

Please find metadata field mappings below:
Field Name ScienceDirect API response element
Creator <dc:creator>
Date <prism:coverDate>
Description <dc:description>
Identifier <dc:identifier>
Rights <prism:copyright>
Source <prism:publicationName>
Title <dc:title>
Type <pubType>
If you have any questions regarding data being returned in API response please contact us.

2. Determine whether user has access to a full-text article


To determine the best available article version to be presented to the user we need to find out whether the user has access to the full-text version of the article on ScienceDirect using Article Entitlements API.

Important Note: ScienceDirect Article Entitlements API accepts the following article identifiers: PII, DOI, EID, scopus_id and pubmed_id as described in corresponding API specification. Institutions that subscribe to Scopus may use the Scopus Search API to retrieve cross-publisher metadata that includes article identifiers as described in Scopus IR/CRIS/VIVO How To Guide. The repository may use any authoritative source of the article metadata to obtain one of the article identifiers accepted by the ScienceDirect Article Entitlements API.


Here is an example of the PII returned by ScienceDirect Search API for an article: <pii>S0049-3848(14)00284-9</pii>

The PII should be used to determine the end user's article access using Article Entitlement Retrieval API as shown below, the API request must be made from the user's browser (client side):

http://api.elsevier.com/content/article/entitlement/pii/S0049-3848(14)00284-9?apiKey=[my_api_key]

If the user has access to the full text of the article, the Article Entitlement Retrieval API response will contain <entitled>true</entitled> or <entitled>open_access</entitled>. If the user does not have access, the API response will contain <entitled>false</entitled>. If the article was not found, the HTTP status code will be 404, <entitled> element will not be present in the response payload while <entry status="not_found"> will be returned.

It is possible to determine the user's access to multiple documents using a list of document identifiers as shown in example below:

http://api.elsevier.com/content/article/entitlement/pii/S0049-3848(14)00284-9,S0955286315000716,S2211419X14001062?apiKey=[my_api_key]

Important Note: The Article Entitlement Retrieval API should be called from user's browser by integrating it directly with the IR's web page via JavaScript. In order to comply with cross-origin security policy, we will ensure that your IR's domain is added to the API key configuration to enable W3C CORS support. You can request to add IR domains by emailing us your API key and a list of authorized domains.


3. Determine whether embargo dates are applicable to Accepted Manuscript


To determine whether embargo dates are applicable to Accepted Manuscript version of the article, the Article Hosting Permissions API is used. The article PII is used to determine hosting permissions status via Article Hosting Permissions API as shown in examples below.

Example Article Hosting Permissions API request:

http://api.elsevier.com/content/article/hostingpermission/pii/S0955286315000716?apiKey=[my_api_key]&httpAccept=text/xml

Example Article Hosting Permissions API response:

<hosting-permission-response xmlns="http://www.elsevier.com/xml/svapi/hostingpermissions/dtd" xmlns:xocs="http://www.elsevier.com/xml/xocs/dtd" xmlns:tb="http://www.elsevier.com/xml/common/table/dtd" xmlns:sb="http://www.elsevier.com/xml/common/struct-bib/dtd" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:ja="http://www.elsevier.com/xml/ja/dtd" xmlns:ce="http://www.elsevier.com/xml/common/dtd" xmlns:cals="http://www.elsevier.com/xml/common/cals/dtd" xmlns:bk="http://www.elsevier.com/xml/bk/dtd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:prism="http://prismstandard.org/namespaces/basic/2.0/" xmlns:jav="http://www.niso.org/schemas/jav/1.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:ali="http://www.niso.org/schemas/ali/1.0/">
<document-hosting-permission status="found">
  <dc:identifier>http://dx.doi.org/10.1016/j.jnutbio.2015.02.010</dc:identifier>
  <prism:url>http://api.elsevier.com/content/article/pii/S0955286315000716</prism:url>
  <prism:doi>10.1016/j.jnutbio.2015.02.010</prism:doi>
  <pii>S0955-2863(15)00071-6</pii>
  <pii-norm>S0955286315000716</pii-norm>
  <link rel="scidir" href="http://www.sciencedirect.com/science/article/pii/S0955286315000716"/>
  <timestamp>2015-10-23T13:43:14.623752-04:00</timestamp>
  <hosting-platform type="institutional_repository">
    <document-version>
      <jav:journal_article_version>AM</jav:journal_article_version>
      <hosting-allowed audience="Internal">
        <allowed-audience>
          <prism:url>http://api.elsevier.com/affiliation/affiliation_id/115791654</prism:url>
          <dc:identifier>AFFILIATION_ID:115791654</dc:identifier>
          <affiliation-name>Molecular Infectious Disease Research Centre</affiliation-name>
        </allowed-audience>
        <allowed-audience>
          <prism:url>http://api.elsevier.com/affiliation/affiliation_id/60020351</prism:url>
          <dc:identifier>AFFILIATION_ID:60020351</dc:identifier>
          <affiliation-name>Chang Gung University</affiliation-name>
        </allowed-audience>
        <allowed-audience>
          <prism:url>http://api.elsevier.com/affiliation/affiliation_id/60003934</prism:url>
          <dc:identifier>AFFILIATION_ID:60003934</dc:identifier>
          <affiliation-name>Chang Gung Children's Hospital</affiliation-name>
        </allowed-audience>
        <allowed-audience>
          <prism:url>http://api.elsevier.com/affiliation/affiliation_id/60031913</prism:url>
          <dc:identifier>AFFILIATION_ID:60031913</dc:identifier>
          <affiliation-name>Chang Gung Memorial Hospital</affiliation-name>
        </allowed-audience>
      </hosting-allowed>
      <hosting-allowed start_date="2016-06-10" audience="Public"/>
      <required-license>http://creativecommons.org/licenses/by-nc-nd/4.0/</required-license>
    </document-version>
    <document-version>
      <jav:journal_article_version>VoR</jav:journal_article_version>
      <hosting-not-allowed/>
    </document-version>
  </hosting-platform>
</document-hosting-permission>
</hosting-permission-response>

The structure of the API response for the embargo date is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<hosting-permission-response>
  <document-hosting-permission status="found">
    ...
    <hosting-platform type="institutional_repository">
      <document-version>
        <jav:journal_article_version>AM</jav:journal_article_version>
        <hosting-.../>

There are 3 possible Article Hosting Permissions API responses:
  1. Hosting of the AM version is not permitted as indicated by <hosting-not-allowed/> element.
  2. Embargo end date is not yet available as indicated by <hosting-permissions-forthcoming/> element.
  3. Embargo end date is indicated in the API response with an actual starte_date element. For example: <hosting-allowed start_date="2015-06-10" audience="Public"/> .

Depending on the Article Hosting Permissions API response, the following rules are used to dynamically determine whether AM version of the article can be presented to various audience types:

Table 1.
Article Hosting Permissions API response example What to be displayed to an internal IR user (a user affiliated with institution)
<hosting-not-allowed/> IR can show hosted accepted manuscript (if available) and a PII link to ScienceDirect
<hosting-permissions-forthcoming/> IR can show hosted accepted manuscript (if available) and a PII link to ScienceDirect
Start date is in the past
<hosting-allowed start_date="2015-06-10" audience="Public"/>
IR can show hosted accepted manuscript (if available) and a PII link to ScienceDirect
Start date is in the future
<hosting-allowed start_date="2017-06-10" audience="Public"/>
IR can show hosted accepted manuscript (if available) and a PII link to ScienceDirect

Table 2.
Article Hosting Permissions API response example What to be displayed to a public IR user (a user not affiliated with institution)
<hosting-not-allowed/> IR can show article metadata and a PII link to ScienceDirect
<hosting-permissions-forthcoming/> IR can show article metadata and a PII link to ScienceDirect
Start date is in the past
<hosting-allowed start_date="2015-06-10" audience="Public"/>
IR can show hosted accepted manuscript (if available) and a PII link to ScienceDirect or DOI link
Start date is in the future
<hosting-allowed start_date="2017-06-10" audience="Public"/>
IR can show article metadata and a PII link to ScienceDirect or DOI links

4. Ensure repository users are linked to the best available article version


Depending on the response from the Article Entitlement API and Article Hosting Permissions API the repository should dynamically decide which links should be displayed to a user. To assist with this task please refer to table below.

Table 3.
Article Entitlement API response Links to be displayed
User has access to full text
<entitled>true</entitled>
<entitled>open_access</entitled>
PII-based link to a full-text article on ScienceDirect and embedded article PDF on the IR site (using API link via iframe)
User does not have access to full text
<entitled>false</entitled>
Link to IR hosted accepted manuscript (if available) based on the rules specified in Table 1 and Table 2 using Article Hosting Permissions API response
An error occurred or document not found
HTTP/1.1 404 Not Found
<entry status="not_found">
Link to IR hosted manuscript (if available) and DOI link

How to create the links described in Table 1, 2 and 3?


PII link to www.sciencedirect.com

The ScienceDirect Search API response will contain a link to an article on ScienceDirect platform for each search results entry. Here is example:

<link ref="scidir" href="http://www.sciencedirect.com/science/article/pii/S0049384814002849"/>

The URL "http://www.sciencedirect.com/science/article/pii/S0049384814002849" from the href element in <link ref="scidir" href= ... > is the actual link to the article on ScienceDirect.

Alternatively, PII-based ScienceDirect article links can be constructed as follows:

http://www.sciencedirect.com/science/article/pii/[PII]
where [PII] should be replaced with an actual article PII (such as S0049-3848(14)00284-9 or S0049384814002849) returned in ScienceDirect Search API response and stored along with article metadata as described in section 1 above.

API link to embed the article PDFs on the IR site

The ScienceDirect Search API response will contain a link labeled <link ref="self" href= ... > to an article via APIs for each search results entry. Here is example:

<link ref="self" href="http://api.elsevier.com/content/article/pii/S0049384814002849"/>

The URL "http://api.elsevier.com/content/article/pii/S0049384814002849" from the <link ref="self" href= ... > href element is the actual link to the article via APIs. Please use that URL to construct Article Retrieval API request by adding PDF format and your API key:

http://api.elsevier.com/content/article/pii/S0049384814002849?httpAccept=application/pdf&apiKey=[my_api_key]&cdnRedirect=true

Embedding of the article PDF on the IR site can be accomplished using an iframe as shown in the example below:

<iframe width="700" height="500" frameborder="0" scrolling="no" marginheight="0" marginwidth="0" src="http://api.elsevier.com/content/article/pii/S0049384814002849?httpAccept=application/pdf&apiKey=[my_api_key]&cdnRedirect=true"></iframe>


Please note that all Elsevier APIs requests can be implemented using javascript. In order to comply with cross-origin security policy, IR's domain must be added to the API key configuration to enable W3C CORS support. Please email us your API key and a list of authorized domains for IR.


If you have questions contact us.