- General questions about the policy
- Other types of outputs
- Deposit requirements
- 'Deposit on acceptance' requirement and initial flexibility on deposit timings
- Discovery requirements
- Access requirements
- REF environment 'open research' section
- Gold open access exception
- Audit and compliance
- Arrangements for the next REF
Updates to the REF open access policy FAQS
These FAQs have been updated to reflect the REF 2021 initial decisions, published on 1 September 2017.
What changes have been made?
FAQ 7 outlines that a unit’s open access strategy, including where this goes beyond the REF OA policy requirements, will be included in REF2021 as a section in the environment template titled ‘open research’. This update is published in the REF2021 initial decisions document.
When will the FAQs be updated again?
We expect to update the FAQs based on the further decisions published in ‘Decisions on staff and outputs’ on 21 November 2017. Please see paragraphs 37-40 in the decisions document for detail.
The REF OA policy, FAQs, and the technical requirements document state that further guidance will be provided by panels. This will be developed in 2018. The REF timetable outlined in the initial decisions document provides further detail on panel recruitment, and when we expect the panels to meet and develop criteria for consultation.
View the previous FAQs
See a PDF version of the FAQs prior to the changes made on 1 September 2017 on the basis of the REF initial decisions.
1.1. Why is this policy being introduced?
The REF process drives the allocation of around £1.7 billion of public money for research each year. The funding bodies that operate the REF believe that the outputs of this research funding should be disseminated as widely and freely as possible. Open access research brings substantial benefits to researchers, students, institutions, governments, public bodies, professionals and practitioners, citizen scientists and many others. Open access makes research more efficient and impactful.
1.2. Do I need to publish in open access journals to be eligible for the REF?
No. The policy allows authors to comply via either the green or the gold route. The basic policy requirement is that papers are deposited in a repository, and almost all journals and conferences already allow authors to make an open-access deposit of some version of their paper in a repository (often the accepted manuscript or a version of the paper at least equivalent to the final peer-reviewed text).
1.3. Does this policy mean that only journal articles and conference proceedings can be submitted to the REF?
No. We fully anticipate that the next REF will operate in the same way as REF2014 in accepting the full range of research output types for assessment.
1.4. Are papers accepted for publication between the start of the REF period and 1 April 2016 now ineligible for submission to the next REF?
No. The open access policy only covers outputs accepted from 1 April 2016 onwards. Outputs accepted before that date are not expected to meet the open access requirements, but these outputs will still be eligible for submission to the REF (as long as they fulfil any other REF eligibility criteria).
1.5. What if I do not want to provide access to my accepted manuscript?
We ask that access is provided to the version of the article that contains all academically necessary changes arising from peer review and the academic editorial process. Accepted manuscripts do not typically contain the subsequent non-academic alterations arising from copyediting and typesetting, nor do they typically show the journal page numbers and other publication livery present in the published version of record, but for many people wishing to access research findings they do represent an academically sound version of the output.
We would encourage authors to provide access to the most up-to-date version of their output, but most journals and conferences do not allow the published version of record to be made open access via deposit in repositories. Authors not comfortable with providing access to their accepted manuscripts are advised to ensure that the manuscript is updated to reflect any changes that are perceived to be essential, or to ask their publisher for permission to deposit the published version of record.
2.1. Am I required to make my books open access?
No. We encourage open access to be extended to all output types (see FAQ 7.1 for further information), but we do not require books to be made open access. This policy only applies to journal articles and conference proceedings. We have reviewed the opportunities and challenges in making books available open access in an independent report by Professor Geoffrey Crossick, published in early 2015.
2.2. Will creative writing articles that are published in academic journals need to fulfil the requirements?
Yes, all articles published in scholarly journals with an ISSN are within scope.
2.3. Will articles in non-academic serial publications (for example, newspapers and magazines) need to fulfil the requirements?
No, the policy applies to scholarly publications only. Newspapers and magazines not intended for a scholarly audience are out of scope of this policy.
2.4. What about conference proceedings published with both an ISBN and an ISSN, or in journals with only an ISBN?
Outputs published in journals with an ISBN are within scope.
The distinction between ISSNs and ISBNs is intended to separate two types of conferences:
- (a) conferences that publish online-only, journal-like series of proceedings (typically in the sciences)
- (b) conferences that publish books or book-like outputs (typically in the humanities).
Following feedback received during consultation, we intend type (a) to be within scope of the policy, and type (b) out of scope, with the ISSN/ISBN distinction providing a useful rule-of-thumb in most cases.
We note, however, that there are some cases when conference proceedings will have both an ISSN (for the series) and ISBN (for individual issues). Given the above, institutions should reasonably be expected to use their own professional judgement to determine whether an individual output is of a type intended to be covered by our policy, bearing in mind that REF panel members will also have the ability to query whether an output has been classified appropriately.
In many cases we believe outputs with an ISSN and an ISBN can meet the requirements, as they are published in a venue with a compliant self-archiving policy, and we would therefore encourage authors to ensure these outputs meet the OA criteria. Where this has not been possible, the submitting institution should record this as an ‘other’ exception in their REF submission.
3.1. Who is responsible for depositing the output?
The author is responsible. It is a feature of this policy that it places a responsibility on authors to deposit their work and consider their open access options. We see this as the clearest way to increase awareness and uptake of open access within the research community.
3.2. What about for papers with multiple authors?
We do not have a strong view on which author should deposit the output, as long as the paper is deposited by one of the authors. We see no substantial drawbacks to more than one author depositing the output. We see it as institutions’ responsibility to ensure that the outputs they are submitting to the REF meet the requirements of this policy.
3.3. How will the policy requirements treat papers which are posted to a subject repository?
If the posted paper meets the deposit requirements, and the repository itself otherwise meets the requirements of the policy, then the posting would fulfil the requirements. Note that the posting of pre-prints (also known as author originals or submitted manuscripts) would not meet the deposit requirements in most cases.
The submission process for future REF exercises (after the next exercise) could potentially involve harvesting of journal articles and conference papers from institutional repositories. We would recommend that institutions keep a record of any outputs posted in subject repositories to ensure they have a complete record of their research output in advance of preparations for the next and future REF exercises.
4.1. What is meant by the date of acceptance?
The date of acceptance is the point at which the author is notified that:
- their output has been reviewed by the journal or conference (normally via peer review)
- all academically necessary changes have been made in response to that review
- the article is ready to be taken through the final steps toward publication (normally copy-editing and typesetting).
By this point, the paper should have been updated to include all changes resulting from peer review as well as any changes of an academic nature requested by the journal editor or conference organiser. At this stage, the journal editor or conference organiser normally notifies the author that their paper has been ‘firmly’ accepted (as opposed to any earlier point of ‘provisional’ acceptance, for example conditional on major or minor revisions being made) and the paper is ready for copy-editing or typesetting; it is the date of this notification that should be taken to mean the date of acceptance.
The author’s final, accepted manuscript is the one that has been agreed with the editor at that point. The accepted manuscript is not the same as the copy-edited, typeset or published paper – these versions are known as ‘proofs’ or ‘versions of record’ and publishers do not normally allow authors to make these open-access.
Note: the diagram shown here is dedicated to the public domain for open reuse.
4.2. Some journals and conferences publish papers ‘as submitted’ with no peer review, editorial input, copy-editing or typesetting. What constitutes the point of acceptance in these circumstances?
Where the journal or conference publishes papers ‘as submitted’, the date of acceptance is the date that the journal or conference confirms that the article has been received from the author and will subsequently be published in the journal or proceedings.
Where papers are not peer reviewed but are still copy-edited and typeset, the point of acceptance is the point that the journal or conference confirms receipt of the article and its readiness for taking through the publication process.
Where papers are peer reviewed but not copy-edited or typeset, the point of acceptance is the point that the peer review and editorial process has completed.
4.3. What is meant by ‘updated peer-reviewed manuscript’ (Policy, paragraph 20)?
In some cases, authors may make minor changes to the manuscript following acceptance (possibly as a result of copy-editing, as figures become finalised, or as rights to third-party material become secured). The author may choose to deposit the updated manuscript, augmenting or replacing the originally deposited file, and the policy allows this. As long as these changes do not result from peer review, the required date for initial deposit will not change.
4.4. Our CRIS automatically harvests metadata from SCOPUS and other sources, which saves us from having to enter metadata for publications into our CRIS or repository manually. If records are also being created by authors at an earlier point, how can we avoid creating duplicate records, or creating lots of manual work to link metadata to deposits?
It may be desirable for deposited manuscripts to have a complete metadata record that matches the publication’s metadata exactly, but this is not necessary for open access. The journal title, paper title and author name(s) should be enough to allow discovery, to prevent or resolve duplicates, and to allow the CRIS to feed metadata across.
The author can complete these fields easily when they deposit the paper, and we are advised that the most commonly used repository software does not allow the paper to be uploaded unless these fields are completed. We are also advised that repository software can detect and merge duplicates.
However, we recognise that universal solutions in this area are still in development. Discussions with all stakeholders have identified a number of possible solutions, including the use of unique researcher identifiers, publisher provision of digital object identifiers (DOIs) at acceptance, and the direct updating of records by publisher systems, all of which can help streamline this process.
4.5 During the first two years of the policy, what date should be taken as the ‘date of publication’ for deposit purposes?
We stress the importance of early deposit and strongly encourage this. During the first two years of the policy, we therefore expect deposit to take place within three months of the 'early online' date of publication, and ideally even earlier. However, we believe it is reasonable for HEIs to use the publication dates provided by their existing systems and workflows to inform deposit timescales. We would strongly encourage HEIs to adopt deposit-on-acceptance practices now to prepare for the policy shift on 1 April 2018.
4.6 What should we do if the date of publication is imprecise?
Where an output’s publication date is provided in the format YYYY-MM, we will expect deposit to take place within three months of the last day of the given month. For example, if an article’s publication date were given imprecisely as 'July 2016', we would expect publication within three months of 31 July 2016, that is by 31 October 2016. If the output’s publication date is even less precise (for example, 'summer 2016'), we advise depositing on acceptance. An article’s acceptance date has the clear advantage of precision.
4.7 What should the acceptance date be in journals which offer an open peer-review once the article has been published?
In this case the acceptance date is the date that the journal confirms that the article will be published.
5.1. When do outputs need to become discoverable?
We would expect outputs to become discoverable as soon as possible after deposit to allow for maximum visibility and use of the deposited work.
When depositing on acceptance: If the paper cannot be made discoverable until it is published, the repository record need not become discoverable (‘live’) until publication. In these circumstances, we would expect the output to be discoverable as soon as possible after the point of first publication (including any early online publication), but we encourage early discoverability. For the purposes of reporting, outputs of this nature should be considered to be following Route 2 in the access requirements of the policy.
5.2. What do institutions need to retain and report as evidence for meeting the discovery requirements?
No evidence needs to be retained about the discovery requirements. During any audit, the REF team will check that the requirement has been met by checking for the presence of a repository record of the output (for example, via a web search).
6.1. Can you explain why CC BY-NC-ND is mentioned as an acceptable licence?
We have not asked that outputs be made available under a specific licence. The open licensing landscape is complex and shifting rapidly. Publisher rules for self-archiving vary widely and are under continual review. While stable open licences have emerged, some authors, particularly in the humanities, have expressed significant concerns about some of these.
To provide some clarity now, we have instead set out a number of things that we would expect any reader with internet access to be able to do: to read the output, to download it, and to search electronically within it, all freely and without charge. This level of access is a realistic minimum and is broadly achievable now for most outputs.
To be more specific, we have advised that outputs licensed under CC BY-NC-ND satisfy this minimum, as would outputs licensed under CC BY, and other more permissive open licenses.
It is our long-term expectation that further sorts of reuse beyond this minimum will become increasingly possible, which is why we are seeking information in the REF2021 environment template about where a unit goes beyond the REF policy requirements (see paragraph 34 of the open access policy and FAQ 7.1).
6.2. How do these licensing requirements interact with other funders’ policies?
They complement each other. Authors that have received funding from the Research Councils, the Wellcome Trust or other funders that demand more permissive licenses can publish under these licenses and still fulfil our minimum requirements.
6.3. From what start point is the embargo period active, and how can this be calculated?
Embargo periods are normally calculated from the date of first publication, including online publication, but publishers may set their own policy. If the paper is not published by the time it is deposited, the embargo end date must be entered into the repository record at a later point. Institutions may choose to ask for ‘closed’ deposits from authors, and when they know the output has been published (perhaps assisted by a publications index), they may set the embargo period based on information found in SHERPA/RoMEO. Over time, publications metadata will evolve to include embargo information, allowing for this step to be completely automated. Publishers are committed to implementing the NISO-approved <license_ref> tag to provide the publication date, the embargo end date and any licence metadata for research outputs via CrossRef. At the discretion of publishers, this may cover green open-access; in those cases, repositories will be able to ingest these metadata from CrossRef automatically, meaning deposits can be made accessible at the end of the embargo period without any additional manual work.
7.1. What aspects of OA should submitting unit’s include in the environment statement section titled ‘open research’?
The REF initial decisions published 1 September 2017 (paragraph 29) notes that the Environment template will include a section on ‘open research’ which should detail the submitting unit’s open access strategy, including where this goes beyond the REF OA policy requirements, and wider activity to encourage the effective sharing and management of research data. This decision was based on responses to the recent consultation on the next REF. REF panels will set out further guidance on the ‘open research’ element of the environment template in the panel criteria which will be developed next year.
The intent of the REF OA policy is to provide a set of minimum requirements for OA, while providing an environment where researchers and HEIs can choose to move beyond the minimum requirements. This may include:
1. Enabling open access for outputs beyond the scope of the minimum policy requirements
2. Licencing of content that allow use beyond the minimum of ‘search within the text, read it and download it without charge’
3. Embargo periods that are shorter than the maxima set by the policy.
Previously published FAQs 7.2 and 7.3 have been removed from this webpage. The panels will be developing criteria in 2018 which will detail how submitting unit’s can demonstrate they have gone beyond the REF policy intent as part of the ‘open research’ element of the environment template.
8.1. What is a repository?
We do not intend to certify repositories as compliant with the policy – the range and diversity of repositories worldwide makes this unfeasible. Jisc has developed a thought piece which outlines principles by which to assess whether arrangements to store and share research outputs constitute a “repository”. This piece was written in the context of the REF open access policy and provides guidance for HEIs considering repository options. ‘What is a repository?’ can be found here.
8.2. What work is being done to allow repository systems to adapt to these changes?
With Jisc, we have been discussing the technical steps that repositories and CRISs need to take to enable authors to comply with the policy. This will include the adoption of the RIOXX metadata application profile, supplemented by a number of REF-specific metadata fields that may be implemented in repositories or CRIS systems. The information and audit evidence that we anticipate will be captured by repository and CRIS systems have now been published under 'technical resources' on our open access policy page.
8.3. Which subject repositories meet the requirements of the policy?
We do not intend to certify repositories as compliant with the policy for REF 2021 – the range and diversity of repositories worldwide makes this unfeasible.
We recognise that institutions will want some assurance that external repositories are retaining information on deposits that might be required for a future REF submission or audit. The information requirements for repositories and the expected audit requirements for the post-2014 REF have now been published and are available under 'technical resources' on our open access policy page. We are aware of discussions that are taking place between the HEI community and several major external repositories.
8.4. Under what circumstances would we need to move outputs from one institutional repository to another?
Where an individual being submitted to the REF has published outputs while employed by a different UK higher education institution (HEI), the submitting institution may wish to assure itself that it has gathered enough information about these outputs to ensure that the relevant submitted outputs meet the requirements of this policy. We anticipate that, in time, most of this information will be available from publicly available metadata, but some information about the initial deposit may not be publicly available.
We recognise that some institutions may not wish to inquire about particular outputs held within another institutional repository during the REF submission phase. Therefore, we will not require that the new institution demonstrate that outputs have met the deposit requirements for these outputs, including in any audit, and while we may ask for evidence of outputs meeting the access requirements, we only plan to do this on the basis of openly available information.
Some institutions have stated that they wish to deal with this issue in advance by holding copies of the repository records and full text of all research in their own repository. We are therefore content for institutions to undertake a bulk transfer of repository records and files on employment of new staff, as long as there is no interruption in access. We recognise that unique researcher identifiers such as ORCID can facilitate this process.
8.5. Some publishers ask that institutions sign agreements before allowing access to outputs in the repository (specifying that embargo periods will be respected etc.) Must institutions sign these in order to create the conditions for compliance?
While institutions are likely to introduce processes in response to our policy, we wish to emphasise that decisions about whether outputs, individuals or institutions are submitted to any REF exercise are made jointly and voluntary by authors and institutions. Participation in the REF is not compulsory, and we know of a number of institutions who were eligible to take part in REF2014 that chose not to participate.
Furthermore, there is no contractual agreement or funding arrangement in place between the UK HE funding bodies and individual authors, and our policy offers significant flexibility over where authors can choose to deposit their work should they wish to meet the REF open-access requirements.
We do not see a compelling reason, therefore, why institutions should sign any agreements with publishers to govern the management of deposits in their institutional repository. We further recommend that institutions give due consideration to the potential negative implications of signing agreements that create significant administrative burdens and legal risks.
8.6. My institution does not have a repository. Can I use the institutional website to provide access to my outputs?
Most institutions do have an institutional repository. Some are using a CRIS system (for example, PURE, Symplectic Elements or Converis) that has repository-like functionality. Some systems are not generally referred to as ‘repositories’ by colleagues; they might also be called ‘research archives’ or have another special name. Authors should check with their open access contact for details of local arrangements.
A number of small or specialist institutions have no repository arrangements in place (for example, institutions where authors do not typically publish their work in journals or proceedings). In these cases, we would encourage institutions to explore options for establishing a repository, which has benefits for the institution beyond this REF policy. However, we recognise that the administrative costs of setting up and managing a repository for might be prohibitive for some smaller institutions. Therefore, if an institution has no repository arrangements in place, and is expecting its REF submission to comprise less than 50 per cent or fewer than 250 journal articles or conference proceedings, then the institution may use departmental or institutional webpages to provide discoverability and access to the outputs in order to meet the requirements of the policy.
9.1. In the Policy, paragraphs 39b and 39c, what is meant by ‘most appropriate publication for the output’, and how will this be checked?
We see this as a matter for the individual researcher and institution to judge, though the determination of the form of the REF submission is of course an institutional responsibility. We have yet to develop detailed submission and audit requirements for the post-2014 REF. But we expect we would ask institutions to show that a process was in place to allow the author or the institution to examine and consider the range of venues that allowed deposit within the rules.
It is important that authors are aware of the rules and give serious consideration to their choice of publication venue in light of them. We therefore want to see limited uptake of these two exceptions. (It should also be noted that the outputs to which these exceptions apply must still have fulfilled the deposit requirements.)
We do not accept that journal impact factors (or the use of specific lists of acceptable journals) are an appropriate factor in the selection of publication venue.
9.2. Don’t the exceptions open up the policy to abuse by authors and publishers?
A sample of outputs submitted to the 2014 REF showed that 96 per cent of outputs could have complied with the access requirements had they been in place sooner (and 100 per cent of outputs could have complied with the deposit requirements). We therefore believe that the risk of abuse is small.
If we see evidence that publishers and authors are seeking to circumvent these rules by taking disproportionate advantage of the policy’s exceptions, we will revisit these provisions with the intention of removing them.
9.3. How will the REF panels be judging exceptions submitted as ‘other’?
This has not yet been decided. We expect the panels to develop criteria in 2018. Criteria on judging exceptions submitted as ‘other’ will be informed by the evidence collected in the survey on the delivery of open access policies which is currently underway as of summer 2017.
9.4. If an institution wishes to submit an output accepted for publication by an individual while at a different UK HEI, but that previous HEI had failed to comply with the policy, would there be consequences for the previous HEI?
9.5. When might an output not be able to be deposited ‘for security reasons’?
In some circumstances, and in some publications, aspects of an author’s identity – particularly their institution – are not made public because their research area is sensitive or controversial. In these circumstances, the author or institution may consider that depositing the work would put the author or institution at risk. We recommend use of the appropriate exception in these circumstances.
9.6. What provision will be in place for authors unable to comply with the policy due to equality and diversity issues?
We are making decisions on submitting staff and outputs for REF2021, with an announcement due in autumn 2017. This announcement is likely to introduce changes to the individual staff circumstances process which was introduced in REF2014.
The REF will continue to take account of equality and diversity issues. The funding bodies will seek to ensure that equality is embedded in all relevant elements of a future exercise, including areas covered by the Equality Act 2010 and other relevant legislation. As with all of our work, the funding bodies take very seriously the responsibility to take account of equality and diversity in the REF processes.
Where there are equality and diversity issues that cause difficulties complying with the open access policy, institutions should make use of the relevant exception in the policy, including the 'Other' exception.
9.7. What circumstances might mean that the individual being submitted to the REF encountered a delay securing the final, peer-reviewed text?
This is only likely where the paper has more than one author, and the individual being submitted to the REF was not responsible for corresponding with the publisher. We know that some institutions are concerned that this exception is open to abuse by authors. We do not wish to see extensive use of this exception and would expect institutions to take due care to prevent abuse. We would expect authors to make their open-access responsibilities known to other authors at an early stage, including seeking joint agreement among UK authors about deposit arrangements.
9.8. If a publisher actively disallows deposit, but has a ‘hybrid’ gold OA option, is the author expected to pay the APC in order to get permission to deposit?
No. We do not wish for authors and their institutions to pay publishers for permission to deposit their work. If the publication does not allow open access to deposited works, and it is the most appropriate publication for that work, there is a relevant access exception in the policy. (This assumes the author is not already publishing via the gold route.)
9.9. What exactly does ‘short-term failure’ mean? Can you provide a timeframe?
We do not wish to be specific about this. However, we do see a difference between ‘planned’ and ‘unplanned’ downtime. We would expect any ‘planned’ downtime should be considerably shorter.
9.10. The policy has an exception in place to deal with those individuals who were not employed by a UK HEI at the point that their work was submitted for publication. What constitutes employment?
The individual must have a contract of employment with a UK HEI with intensity of 0.2FTE or greater. Individuals not under contract, or with a contract of intensity less than 0.2FTE, will be covered by the relevant exception in the policy.
10.1. What is the purpose of the gold deposit exception in the policy and when can it be used?
This exception is for outputs which are published via gold open access in accordance with other funders’ requirements and definitions. We will continue to audit by checking availability on publishers’ websites for a sample of outputs. This deposit exception was incorporated into the policy in July 2015 in response to feedback from the sector.
10.2. If the publication has provided access through the gold route, do authors still need to deposit their manuscripts?
No. We would encourage papers to be deposited, but we do not require it. There is an exception in the policy to deal with these outputs.
10.3. If the publication provides ‘retrospective’ gold open access to outputs (that is, later than the date of publication), will these outputs then be compliant with the policy?
No. We expect outputs published via the gold route to be available immediately upon publication. FAQ 10.1 outlines how the gold deposit exception should be used.
11.1. What supporting information and evidence of meeting the deposit, discovery, access and exceptions should institutions retain?
We have developed a list of information that we expect institutions to retain, which can be found under 'Technical resources' on the open access research policy guide.
11.2. How will the RIOXX application profile help us to satisfy the REF requirements?
The RIOXX profile is designed primarily to fulfil Research Councils UK's requirements for metadata collection and reporting on open access. It is compatible with the information and audit requirements for the open-access policy in the next REF, which also overlap with RIOXX. The REF information requirements document shows all areas where overlaps exist between RIOXX and REF.
To fulfil the metadata requirements for the open access policy in the next REF, institutions should ensure they are able to collect the information set out in the information requirements document. These include a number of fields (including relating to REF OA exceptions) that RIOXX does not cover. This document is available under 'technical resources' on our open access policy page.
Work to implement the requirements in repository and CRIS systems is under way. This will include work that Jisc are carrying out, and various user-led initiatives. For further information, please contact firstname.lastname@example.org
11.3. What freedom will the REF panels have to request audits of compliance with the OA policy above and beyond what has been set out in the audit requirements so far?
There will be an audit process for the OA policy in the next REF, as set out initially in the REF audit requirements document. This document is available under 'technical resources' on our open access policy page.
The REF panels will be permitted to request audits of particular outputs, but the following areas of the open-access policy are exempt from panel-initiated audit:
- acceptance date
- technical exceptions
- deposit date
- deposit exceptions
- discovery requirements
- embargo end date
- free to read/download/perform in-text search
- access exception b (embargo period)
- access exception c (publisher disallows OA).
12.1. What version of output will be assessed by panels in the next REF?
It is our current intention that the panels will assess published versions of record, not accepted manuscripts, in the next REF. In establishing this policy, we have assumed that the next REF will operate in the same way as REF2014.
12.2. Are category C staff in scope?
All outputs that meet the definition are within scope. This includes outputs being submitted that were published by individuals employed by a UK HEI at the point of acceptance.
Further decisions on the arrangements for submitting staff to REF 2021 will be made in autumn 2017.