ScienceDirect Search API Migration

Summary

Due to changes to our search back-end, the existing ScienceDirect Search API has been deprecated (and referenced as such within this guide). However, the API will remain accessible through December 13, 2018 to allow time to migrate to our new search API options.

Beginning in September, two new ScienceDirect search interfaces will be offered to users as migration options. These interfaces are based on improved search technology, and each one is designed to support different use cases:

  1. ScienceDirect Search API v2 (available now)
    - Replicates the search experience on www.sciencedirect.com
    - Suitable for federated search and free text search applications
  2. Article Metadata API (tentative availabilty: October 12, 2018)
    - Provides advanced metadata "lookup" style searches
    - Suitable for IR and TDM applications

Both new interfaces follow the RESTful principles users have come to expect, support our standard authentication methods, and in some cases use similar or identical query syntax and response payloads. In other cases, the query syntax and response payloads differ substantially from the deprecated ScienceDirect Search API.


The new ScienceDirect Search API v2 is designed for the following use cases:

  • Federated search applications
  • General free text searches

This API replicates the search experience on ScienceDirect. The search results from this API will precisely match those from the web application.

We offer the new ScienceDirect Search API v2 in two RESTful flavors: PUT (native) and GET (legacy).

Resource URI HTTP Verb Legacy Support Recommended Purpose
https://api.elsevier.com/content/search/sciencedirect PUT No Yes The native method for this API, with the most robust capabilities and support. This is the recommended method.
https://api.elsevier.com/content/search/sciencedirect GET Yes No Provided as a legacy option for those migrating who wish to evaluate the API using familiar query syntax, or for those who need time to understand the PUT option. This is provided for convenience but not recommended, as the GET method is an emulation layer not natively supported by the API.

Understanding the new PUT interface

The new ScienceDirect Search API v2 natively supports the HTTP PUT method for submitting requests. While different from the GET method supported by the deprecated search API, the PUT requests are quite simple and particularly well-suited to federated search applications that use web forms.

To clearly illustrate the difference, examples of the old and new methods will be shown. First, an example of a GET request using the deprecated ScienceDirect Search API:

https://api.elsevier.com/content/search/scidir?query=title(Articulation+Disorders)+AND+openaccess(1)+and+pub-date+aft+20180601

Note that the GET request's parameters are contained within the URL, e.g. title(Articulation+Disorders). This is different from a PUT request, which contains no parameters in the URL, but rather places them in a structured form within the request body.

Example PUT request URL:

https://api.elsevier.com/content/search/sciencedirect

Example PUT request body:

{
  "title": "Articulation Disorders",
  "filters": {
    "openAccess": true
  },
  "loadedAfter": "2018-06-01T00:00:00Z"
}

The example request body uses a JSON structure, standard for this new API. Imagine a web form containing a field for article title, a drop-down for selecting "Open Access" or not, and a date chooser to specify articles that appeared on ScienceDirect only after the given date. The user inputs would map directly to the request body structure, e.g. the article title inputted by the user would map to "title": "Articulation Disorders" in the JSON request body.

More detailed information, examples, and request / response schemas can be found in the ScienceDirect Search API v2 documentation.


Understanding the new GET interface

While not recommended as your primary choice, we've provided an emulated GET layer as a convenience for testing and evaluation purposes, or for those who require additional time to understand and implement the native PUT interface.

The GET interface works the same way as the deprecated API; parameters must be supplied in the request URL. The GET method uses the same field names as the deprecated ScienceDirect Search API, and those fields get mapped to the new ScienceDirect Search API v2 fields. This provides some legacy support and backwards compatibility, but because we're emulating this functionality, it's not as robust as the new API's native PUT method.

Here's an example of a GET request using the new API:

https://api.elsevier.com/content/search/sciencedirect?query=all(Articulation+Disorders)+AND+pub-date+aft+2017

As mentioned previously, note that the GET method's field names emulate the deprecated ScienceDirect Search API for legacy support. They do not match the native field names associated with the new ScienceDirect Search API v2. In the section below, these differences are outlined in detail.


Available search fields

The chart below shows the available search fields and filters available in the new ScienceDirect Search API v2 interface and how they correspond to the fields in the deprecated API. The "new" field names and filters may only be used with the PUT interface -- the recommended method. The "old" field names and filters may only be used with the GET interface and are a subset of what's available in the PUT interface.

Old Field New Field Description Validation Rules Notes
authors authors The article / book author(s) search field Max 250 characters This field searches both first AND last names.
pub-date date Year (or year range) in which an article or chapter was published. Must match the following regex pattern:
^[0-9]{4}\s*(-\s*[0-9]{4})?$
N/A highlights Determines whether highlights for search terms should be returned. Default: false. true or false
start offset The position from where search results begin. Default: 0. Permitted values: 0-6000 The syntax differs slightly, but this is used to paginate through search results in the same fashion as "start" with the deprecated API.
count show The number of results to retrieve per request. Default: 25. Permitted values: 10, 25, 50, 100 The syntax differs slightly, but this is used to paginate through search results in the same fashion as "count" with the deprecated API.
sort sortBy How results should be sorted. Default: relevance. Permitted values: relevance, date
N/A openAccess Flag that can be used to filter results to only include open access content. Permitted values: true or false The deprecated API has no direct equivalent filter.
vol-issue issue The issue numeric search field. This field supports both issue numbers and issue ranges Must match the following regex pattern:
^[0-9]{1,15}\s*(-\s*[0-9]{1,15})?$
What was a combined field in the deprecated API is now a separate field in the new API.
orig-load-date AFT loadedAfter Field that can be used to search only for articles loaded onto the ScienceDirect website after a certain date. Must be a combined date in ISO 8601 format:
yyyy-MM-dd'T'HH:mm:ssX
Functions similarly to using orig-load-date in combination with the AFT qualifier with the deprecated API.
pages, pagefirst, pagelast page The page search field. This field supports both page numbers and page ranges. Must match the following regex pattern:
^([0-9A-Za-z\.]{1,15})\s*(-\s*([0-9A-Za-z\.]{1,15}))?$
Combines the functionality of several fields used by the deprecated API.
srctitle pub Searches the title of a journal or book. Max 250 characters
all qs The general search field for searching over all article / book chapter content (excluding references). Max 250 characters A free text search using the GET interface is equivalent to using "qs" with the PUT interface.
title title Max 250 characters Searches the title of an article or book chapter.
vol-issue volume The volume search field. This field supports both volume numbers and volume ranges. Must match the following regex pattern:
^[0-9]{1,15}\s*(-\s*[0-9]{1,15})?$
What was a combined field in the deprecated API is now a separate field in the new API.

More detailed field specifications can be found in the ScienceDirect Search API v2 documentation.


Supported request syntax

Boolean operators

The uppercase AND, OR, and NOT operators can be used within a query string on selected fields to construct boolean queries:

{
  "authors": "Crick AND Watson",
  "title": "Molecular Structure OR DNA",
  "qs": "NOT London"
}

Lowercase (or mixed case) and, or, not are treated as search terms, not Boolean operators. Note that when searching multiple fields as in the above example, the search implicitly ANDs these fields. Explicit ORs across fields is not supported.

Prohibit operator

A hyphen immediately before a word can be used to exclude results containing that word:

{
  "qs": "depression -economics"
}

The above search retrives content containing the word "depression" but not the word "economics."

Parentheses

Parentheses can be used to clearly specify how queries should be interpreted:

{
  "qs": "(stem cell research OR human embryonic) AND cambridge university"
}

Quotes

Quotation marks can be used to search for exact phrase matches:

{
  "title": "\"Cardiothoracic surgery training\""
}

Filters

Filters can be used to restrict results. The filter parameters must be contained within the "filter" element:

{
  "title": "Physics",
  "filter": {
    "openAccess": true
  }
}

Currently, the only filter available is for Open Access articles.

Pagination & Sorting

To retrieve the full search result set, one must use pagination. The pagination parameters "offset" and "show" are used for this purpose. To sort results, the "sortBy" element must be used. Both pagination and sorting parameters must be contained within the "display" element:

{
  "qs": "Management of drip irrigation",
  "display": {
    "offset": 50,
    "show": 25,
    "sortBy": "date"
  }
}

Valid "offset" values: 0 <= offset <= 6000. Valid "show" values: [10, 25, 50, 100]. Sorting can be done by relevancy (default) or date.


Example requests

An author search, sorted by date descending:

{
  "authors": "holmes",
  "display": {
    "sortBy": "date"
  }
}

Results 50-75 of a general free text search:

{
  "qs": "Management of drip irrigation",
  "display": {
    "offset": 50,
    "show": 25
  }
}

A search for open access articles / book chapters available on ScienceDirect since June 2018 and containing the words "Articulation" and "Disorders":

{
  "title": "Articulation Disorders",
  "filters": {
    "openAccess": true
  },
  "loadedAfter": "2018-06-01T00:00:00Z"
}

Example request response

The response payload contains familiar article metadata. One such sample response is shown below:

{
    "resultsFound": 7670845,
    "results": [
        {
            "authors": [
                {
                    "order": 0,
                    "name": "Xiao He"
                },
                {
                    "order": 1,
                    "name": "Xing Chen"
                }
            ],
            "doi": "10.1016/j.infsof.2018.07.010",
            "loadDate": "2018-07-17T00:00:00Z",
            "openAccess": false,
            "pages": {
                "first": "98",
                "last": "117"
            },
            "pii": "S0950584918301538",
            "publicationDate": "2018-07-17",
            "sourceTitle": "Information and Software Technology",
            "title": "Testing Bidirectional Model Transformation Using Metamorphic Testing",
            "uri": "https://www.sciencedirect.com/science/article/pii/S0950584918301538",
            "volumeIssue": "Volume 98"/”Volume 4, Issue 7”,
        },
        {
            "authors": [
                {
                    "order": 0,
                    "name": "Zhongxue Chen"
                },
                {
                    "order": 1,
                    "name": "Qingzhong Liu"
                },
                {
                    "order": 2,
                    "name": "Kai Wang"
                }
            ],
            "doi": "10.1016/j.ygeno.2018.07.010",
            "loadDate": "2018-07-17T00:00:00Z",
            "openAccess": false,
            "pages": {},
            "pii": "S0888754318300193",
            "publicationDate": "2018-07-17",
            "sourceTitle": "Genomics",
            "title": "A genetic association test through combining two independent tests",
            "uri": "https://www.sciencedirect.com/science/article/pii/S0888754318300193"
        }
    ]
}

Schemas

Request body schema

{
    authors: string,
    date: string,
    display: {
        highlights: boolean,
        offset: integer,
        show: integer,
        sortBy: string
    },
    filters: {
        openAccess: boolean
    },
    issue: string,
    loadedAfter: string,
    page: string,
    pub: string,
    qs: string,
    title: string,
    volume: string
}

Response body schema

{
  "resultsFound": integer,
  "results": array,
  "message": string
}

Each search result is contained within the "results" array and follows the schema outlined below.

Search result object schema

{
    authors: array,
    doi: string,
    loadDate: string,
    openAccess: boolean,
    pages: object,
    pii: string,
    publicationDate: string,
    sourceTitle: string,
    title: string,
    uri: string,
    volumeIssue: string
}

The new Article Metadata API is designed for the following use cases:

  • Advanced metadata "lookup" style searches
  • IR and TDM applications

The Article Metadata API will offer more advanced search functionality than the new ScienceDirect Search API v2; however, the search experience and results will not match www.sciencedirect.com. For this reason, the new ScienceDirect Search API v2 is recommended for federated search applications.

If you have no requirement to match the ScienceDirect web search experience and need advanced search functionality, than you should consider the Article Metadata API. IR and TDM applications will benefit from its more advanced fields, filters, and Boolean options. The capabilities of this API will also be expanded over time as the use cases dictate.

The Article Metadata API will be offered as one RESTful interface:

Resource URI HTTP Verb Legacy Support Recommended Purpose
https://api.elsevier.com/content/metadata/article GET Yes Yes Provides advanced lookup functionality suitable for IR and TDM applications.

Understanding the Article Metadata API

While well suited to federated search and general free text searches, the new ScienceDirect Search API v2 does not replicate all of the features and functionality of the deprecated ScienceDirect Search API. The latter was also used to populate institutional repositories and as a discovery tool for some TDM applications. These use cases require lookups on specific metadata and in some cases, more complex Boolean options.

Below is a typical example request rom the deprecated ScienceDirect Search API:

https://api.elsevier.com/content/search/scidir?query=affiliation(MIT)+AND+available-online-date+AFT+20001201

Using the new Article Metadata API, the request syntax and response payload would be identical to the above example, except for the URI:

https://api.elsevier.com/content/metadata/article?query=affiliation(MIT)+AND+available-online-date+AFT+20001201

Note the difference in the resource URI, i.e. /metadata/article. The request is otherwise structured in the same manner using the same parameters. Initially, the most frequently used fields for these use cases will be supported, with more to be added in the future.

The Article Metadata API response payload will be very similar, as well (sample data shown):

{
    "search-results": {
        "opensearch:totalResults": "38",
        "opensearch:startIndex": "0",
        "opensearch:itemsPerPage": "1",
        "opensearch:Query": {
            "@role": "request",
            "@searchTerms": "affiliation(MIT) AND available-online-date AFT 20001201",
            "@startPage": "0"
        },
        "link": [
            {
                "@_fa": "true",
                "@ref": "self",
                "@href": "https://api.elsevier.com/content/metadata/article?start=0&count=1&query=affiliation%28MIT%29%2BAND%2Bavailable-online-date%2BAFT%2B20001201",
                "@type": "application/json"
            },
            {
                "@_fa": "true",
                "@ref": "first",
                "@href": "https://api.elsevier.com/content/metadata/article?start=0&count=1&query=affiliation%28MIT%29%2BAND%2Bavailable-online-date%2BAFT%2B20001201",
                "@type": "application/json"
            },
            {
                "@_fa": "true",
                "@ref": "next",
                "@href": "https://api.elsevier.com/content/metadata/article?start=1&count=1&query=affiliation%28MIT%29%2BAND%2Bavailable-online-date%2BAFT%2B20001201",
                "@type": "application/json"
            },
            {
                "@_fa": "true",
                "@ref": "last",
                "@href": "https://api.elsevier.com/content/metadata/article?start=37&count=1&query=affiliation%28MIT%29%2BAND%2Bavailable-online-date%2BAFT%2B20001201",
                "@type": "application/json"
            }
        ],
        "entry": [
            {
                "@_fa": "true",
                "link": [
                    {
                        "@_fa": "true",
                        "@ref": "self",
                        "@href": "https://api.elsevier.com/content/article/pii/S0045794907000727"
                    },
                    {
                        "@_fa": "true",
                        "@ref": "scidir",
                        "@href": "https://www.sciencedirect.com/science/article/pii/S0045794907000727"
                    }
                ],
                "dc:identifier": "DOI:10.1016/j.compstruc.2007.02.001",
                "prism:url": "https://api.elsevier.com/content/article/pii/S0045794907000727",
                "dc:title": "Preface",
                "dc:creator": "K.J., Bathe",
                "prism:publicationName": "Computers & Structures",
                "prism:issueName": "Fourth MIT Conference on Computational Fluid and Solid Mechanics",
                "prism:issn": "00457949",
                "prism:volume": "85",
                "prism:issueIdentifier": "11",
                "prism:coverDate": [
                    {
                        "@_fa": "true",
                        "$": "2007-06-01"
                    }
                ],
                "prism:coverDisplayDate": "June–July 2007",
                "prism:startingPage": "627",
                "prism:doi": "10.1016/j.compstruc.2007.02.001",
                "openaccess": "0",
                "openaccessArticle": false,
                "openArchiveArticle": false,
                "openaccessUserLicense": null,
                "pii": "S0045-7949(07)00072-7",
                "authors": {
                    "author": [
                        {
                            "@_fa": "true",
                            "given-name": "K.J.",
                            "surname": "Bathe"
                        }
                    ]
                },
                "prism:teaser": null
            }
        ]
    }
}