Economic Analyses of Federal Scientific Collections: Methods for Documenting Costs and Benefits

Federal collections have been created to serve agency missions and, in a few cases, to comply with legislative and regulatory mandates. The benefits generated by federal scientific collections can take monetary and non-monetary forms, and are usually indirect and delayed. The value chains that connect costs to benefits are generally difficult to document. This report describes five methodologies that are available for describing and estimating the benefits federal collections generate. Departments and agencies can use the methods described for evidence-based decisions concerning policies and management practices for their institutional collections.


Introduction
P hysical objects form the basis of research in many scientific disciplines. Organisms, soil, medical samples, meteorites, and thousands of other types of objects provide the evidence that is central to the missions of many Federal departments and agencies. These missions include research, regulatory responsibilities, and complying with legislation that serve the Nation' s interests. In many cases, departments and agencies decide to retain objects for long-term preservation in scientific collections, in anticipation of future use or in compliance with regulation or legislation. The long-term support for collections is a commitment of Federal resources without clear evidence of future returns on these investments. History has shown that some of these objects prove to be valuable, even critical, in solving mission-related challenges. They may help to cure diseases, save agricultural crops, avoid natural disasters, and provide other tangible benefits to the Nation. Many others may not have been used since being added to a long-term collection.
Faced with their uncertain future value, how can Federal collection officials decide which objects to preserve and which to discard? How can departments and agencies justify the cost of creating and maintaining scientific collections?
Are there evidence-based approaches that can reduce the costs and increase the benefits associated with Federal scientific collections?
In 2005, the Office of Science and Technology Policy (OSTP 1 ) created an Interagency Working Group on Scientific Collections (IWGSC) 2 to explore and solve the common challenges that face Federal departments and agencies with scientific collections. Their 2009 report included a survey of more than 300 collections in 14 departments and agencies, documenting their size, uses, management practices and financial support (IWGSC 2009). The report also put forward recommendations, including the development of better methods for documenting the budget support needed by these collections. In 2010, OSTP directed all Federal departments and agencies with collections to implement this recommendation (OSTP 2010)  This report presents the findings of a year-long IWGSC study to provide Federal departments and agencies with standard methods for documenting costs and benefits related to their scientific collections. 4 These findings enable Federal scientific collections to do the following: • Document the costs related to operating their long-term institutional collections, whether or not they have separate line items in the budgets of their organizational structure; • Identify how these costs are allocated to the six standard services provided by institutional collections; • Review their philosophies and practices regarding user fees and other forms of cost recovery, and consider how they may be affecting collection use and the benefits stemming from this use; • Consider new methods for documenting the benefits generated by use of their collections, especially the five methods presented in the report, with examples from Federal collections: ◌ Tracing the value chains from collection use to new products, processes, or other economic activity; ◌ Describing the roles their collections have played recently in breakthrough discoveries and meeting major societal challenges; ◌ Associating collections and the services they provide with the mission-critical uses they have played in the past, and are being maintained to serve if needed again; ◌ Documenting the pattern of collection use and the financial assets that users are investing in the collection; and ◌ Gathering data on the losses the Nation' s economy and well-being would suffer if the collections and the services they provide didn't exist.

Scientific Collections as a Marketplace
Taken collectively, the collections in any discipline resemble a marketplace with providers, customers, and limited supplies of goods and capital. In this context, "goods" refers to the objects in scientific collections, not other goods and services provided by institutions with collections (e.g., jobs created, increased tourism, museum gift shop items sold; see American Alliance of Museums, 2017). Providers bear the costs of making goods available, and consumers want the highest quality goods at the lowest cost possible.
The marketplace analogy is not perfect for two principal reasons. First, the prices that consumers pay for access to goods are not the result of competition or other market forces. Only a minority of collections owned and operated by Federal departments and agencies charge fees that are set to recover costs, and they are never set to generate profits. Most Federal collections charge fees that recover only a portion of costs borne by providers, and they often charge only for shipping-or for nothing at all. Second, the benefits generated from the goods provided by collections are generally bestowed on society, not the consumers themselves. They take the form of new knowledge in the public domain, or improved public health and security, or protection against future shocks to the environment, food supply, and public safety. Relatively few Federal collections provide goods that private companies use to generate revenue and profits. Schumann (2014) provides a clear summary of the distinctions between valuations of public and private goods; Smale and Koo (2003) provide a taxonomy of values generated by scientific collections used for plant breeding. Scientific collections have been described by some authors as "global public goods" and have been discussed in the context of the "public commons" (e.g., Halewood, 2013 for plant genetic resources; Kothamasi, Spurlock, and Kiers, 2011 for agricultural microbial resources; and Reichman, Uhlir, and Dedeurwaerdere, 2015 for microbial culture collections).
This literature focuses on intellectual property law pertaining to collections but does not explore methods for estimating their costs and benefits, and was not considered directly germane to this report.
This study explores the other factors that determine the costs borne by providers and the benefits generated from the use of collection-based goods obtained by consumers. A clearer understanding of these factors and the relationships among them may equip Federal collections to make evidence-based policies and better management decisions (Graves 2003).

Box 1. Glossary of Terms
Accessioning is the process of transferring ownership of objects into an institutional collection. Collections may incorporate the records of the objects into the registry of that collection and objects may be physically transferred into a collection at the time of accessioning, but these procedures vary.
Deaccessioning is the process of ending legal ownership of objects in an institutional collection, either by transferring ownership to another institution or destroying the objects.
Institutional collections are scientific collections that are made available for use by qualified researchers, companies, government agencies and other qualified users (see original definition in IWGSC 2013a). They can be non-renewable or renewable (defined below), and they are generally managed by collection professionals.
Non-renewable collections consist of objects that have been collected or fabricated at a specific time and place and are no longer being produced. They cannot be replaced with identical objects so they are maintained and preserved long-term for future use. Objects in these collections may undergo some degree of destructive analysis, but they are completely consumed only under rare circumstances.
Project collections are scientific collections that are managed and used by scientists who may have obtained the physical objects for specific research or another purpose, or by others working on that or related research (see original definition in IWGSC 2013a). Objects in project collections are sometimes completely consumed by destructive analyses or discarded at the end of the project.
Renewable collections consist of living organisms that can generate replicas of themselves, or man-made objects that can be fabricated. As a result, objects can be completely consumed by destructive analyses without exhausting the supply.
Renewable collections are made available in response to user demand and can be disestablished if demand wanes.
Scientific collection is used herein as shorthand for "object-based scientific collection": a group of physical objects that are used for research, development, education, and other activities related to disciplines in physical, life, earth, and planetary sciences as well as archaeology, physical anthropology and applied sciences such as engineering, agriculture, and veterinary science.
Scientific collections may include the maps and notes related directly to physical objects but would not include libraries, document archives, or data repositories not associated with physical objects.

Types of Federal Collections
I WGSC (2009) included a survey of Federal scientific collections that found 14 departments or agencies with approximately 300 scientific collections. They cover the full spectrum of scientific disciplines and the majority of these departments or agencies own and manage their collections long-term. IWGSC has since created the Registry of U.S. Federal Scientific Collections (USFSC) 5 that now contains data on many hundreds of scientific collections in 19 departments or agencies.
Despite the great diversity of disciplines in which Federal collections are maintained, there are only a few types of collections from the perspective of their operating principles (see Box 1 for definitions). Non-Federal collections can generally be assigned to one of these categories and many of them are eligible for Federal support through NSF grants and other Federal funding opportunities.
Most, but not all, of the objects in Federal collections begin as objects in "project collections." While a project is in progress, access to the collection and information about its contents are generally controlled by the researchers working on them. They make decisions about consuming an object, in part or in whole, through destructive analyses, choosing not to preserve a part for future use. Eventually, one of several things occur: the project can be completed with or without publication of the results; the project leaders can retire or pass away; or the department in which the project was conducted could be reorganized, relocated, or abolished. In each case, decisions need to be made about what to do with the collection.
Many objects in project collections are accessioned into "institutional collections" (sometimes referred to as "archival collections"). IWGSC agencies have developed policies governing their institutional collections 6 and these are available on the IWGSC Clearinghouse. Ownership of and responsibility for these objects passes to the institutional collection and control of the objects normally passes from researchers to professional collection managers. Their mission is to preserve the contents of the collection and to make them available for uses that serve the broader agency mission and generate benefits for society. Most institutional collections will disseminate information about the contents of the collection and will facilitate their use by qualified researchers. There may be restrictions on the use of certain objects in a collection based on national security, public safety, or the terms of collecting permits and/or material transfer agreements. Examples include informed consent given by human subjects at the time samples are collected, and biological samples collected and exported under international agreements such as the Convention on Biological Diversity, the Nagoya Protocol on Access and Benefit Sharing, the Food and Agriculture Organization' s International Treaty on Plant Genetic Resources for Food and Agriculture, and the Convention on International Trade in Endangered Species.
Most institutional collections are "non-renewable." They include inanimate objects, preserved dead organisms, or parts thereof (e.g., frozen tissue or body fluid samples). These collections normally have procedures by which researchers can request permission to perform destructive subsampling and analysis of a sample or specimen. Since the mission of institutional collections is the preservation of objects for future study, collection managers require strong justifications based on the scientific importance of the proposed analysis. Permission is rarely granted to consume the last remaining portion of a sample or specimen.
In contrast, "renewable" collections consist of organisms that reproduce or replenish themselves (e.g., cell cultures; viable microbial, plant and animal germplasm) or inanimate objects that can be replicated. Destructive sampling and analyses of these objects are common and are not issues of concern for renewable collections because replacements can be grown or manufactured. These are sometimes referred to as "research resources" though IWGSC considers them scientific collections.

Ownership and Stewardship of Federal Collections
I n addition to directing agencies to establish adequate and sustainable operating budgets for collections, the America COMPETES Act and OSTP (2010) directed agencies to develop and disseminate policies on the management and use of Federal collections. OSTP (2014) specified the components that should be included in these collections policies and the IWGSC Clearinghouse now presents the policy documents that comply with these requirements. Taken together, the IWGSC (2009) recommendations, America COMPETES, OSTP (2010), and OSTP (2014) direct agencies to harmonize their collections policies, budgets, and practices with agency missions as expressed in authorizing legislation and regulations.
When Federal departments and agencies obtain and add non-renewable objects to project collections, the objects normally meet some immediate mission-related need. Project collections do not encumber agency budgets beyond the resources allocated for research projects. As described above, objects in project collections can be consumed through destructive analyses when researchers decide that immediate project goals outweigh concerns about potential future use.
Within the limits placed on the use of objects by collecting permits, consents, and material transfer agreements, objects in project collections can be viewed as consumable research supplies. In contrast, when agencies accession objects into their institutional collections they accept long-term responsibility and related costs for maintaining the objects long-term for future use, as stipulated in their collections policies. Complete destructive sampling of an object is permitted only under very specific conditions. Figure 1 illustrates the diverse pathways that objects can follow before they are accessioned into Federal institutional collections. Each of the pathways leading to a Federal institutional collection can be viewed as a long-term financial obligation by the receiving agency. However, not all of the objects being added to Federal institutional collections begin as project collections obtained and owned by that agency.
Some renewable collections provide users with living organisms for research and development of agricultural products and industrial processes (see Boxes 2 and 3). Users rely on these collections to discover and develop traits found in nature' s diversity, so specimens are collected based on how they expand the collection' s diversity. Accessioning them into an institutional collection represents a significant financial commitment because of the costs of maintaining living populations. Other renewable collections offer precisely standardized objects to users for use as calibration standards, or highly characterized and uniform organisms for controlled experiments. This second group of renewable collections goes through an initial period of planning, development, testing, and quality control before the standardized samples or specimens can be offered to users. Planning may involve market surveys and analyses or workshops to determine market demand and to determine the exact properties and characteristics that users want the samples or specimens to have. The development phase involves activities like chemical synthesis, genetic engineering, or selective breeding to produce samples or specimens that will satisfy user demands. Testing and quality control follow for confirmation. We consider renewable collections as project collections during this initial phase. We consider them institutional collections once testing has been completed and they have been made available to users. T his study explores costs associated with institutional scientific collections and standard methods that can be used to estimate and document those costs. This approach is essential and the logical predicate to evaluating benefits generated by collections-the returns on investments made by taxpayers in those collections. The evaluation of costs takes into consideration differences among institutional, project, renewable, and non-renewable collections.

Project Collection Costs
Project collections have a wide range of cost categories. The costs in each category, measured on a per object basis, vary over many orders of magnitude among collections. This variation results from the fact that project collections involve collecting activities, preserving the objects and preparing them for analyses, conducting a diverse array of analyses, interpreting data, and preparing publications. The logistics for project collections are sometimes simple, involving one or a few researchers collecting in local areas with no requirements for collecting permits other than the agreement of landowners.
Others are far more complex, involving inter-institutional and international agreements or treaties that govern collecting, transfer and use of research material. For example, several different authorities may have legal responsibility for issuing permits or consents to: collect threatened and endangered species; obtain samples from human subjects; or collect and transfer research material among countries or from territories beyond national boundaries (e.g., Antarctica, the open oceans, or outer space). Significant effort and expense may be needed to obtain permits or consents before the first collecting activity can begin or access to and use of samples can be granted. Once permission to collect has been granted, the cost of obtaining samples or specimens can be as high as NASA space missions, deep-sea drilling operations, or expeditions into remote regions like tropical forests or the deep sea, to name a few. Analytical techniques can involve a wide range of other instrumentation and techniques, including expensive infrastructure such synchrotron beams, CT scanners, DNA sequencers, and mass spectroscopy.
For these reasons, project collections do not lend themselves to standard methodologies for estimating costs. There are simply too many types of costs and too many factors that affect these costs to produce cost estimates that can be compared meaningfully. For these reasons, cost estimation for project collections is not treated in this report.

Institutional Collection Costs
Unlike project collections, there are relatively few types of costs involved in the operation of institutional collections. The costs per object within each cost category will vary according to: • the type of preservation needed for non-renewable collections (e.g., ultra-cold versus refrigerated; dry versus alcoholpreserved at room temperature), and processes for reproducing renewable collections; • geographic location, which affects salaries (due to cost of living variation), facilities and utility expenses (due to rental, construction and utility rates); • size of the collections (affecting economies of scale); and • the services provided by the collection.
The first three sources of variation are relatively straightforward and can be taken into account when comparing costs among institutional collections. The most important source of variation in per object operating costs is differences among collections in the services they provide. That is, the more different services a collection provides, and the more of each service it provides, the higher the cost per object. Cost comparisons among institutional collections will only make sense if differences in the services provided are considered. Like costs, the types of benefits generated by institutional collections vary with the number of services they provide (see below, Implications for Policies and Management).

Services Provided by Institutional Collections
This study identified six different services that Federal institutional collections may elect to perform, depending on the missions of their institutions and the resources available to them (see Table 1). NSF provides support for all of these services to non-Federal collections in the biological sciences. The exact nature of some services will differ between renewable and non-renewable collections, as portrayed in Table 1  The first service, accessioning, is the core service that institutional collections perform. The accessioning function is integral to the definition of institutional collections. As one study participant said, "It' s what we do." Accessioning is both a service and a critical decision-point for collection managers because adding objects to an institutional collection is a commitment of space and support that usually has no clear time limit. In a few cases, legislation and/or regulations require that certain objects must be accessioned into Federal scientific collections. In other cases, agencies must make decisions based on the potential costs and benefits of long-term maintenance (see below, Implications for Policies and Management).
Preservation and maintenance ensure the security of objects and their fitness for use by future users. This is the second most common service provided, though collections within an institution may receive different levels of this expensive service. Environmental controls on temperature, humidity, exposure to sunlight and other factors can be critical in preventing the deterioration that would render objects useless for research or other activities (Stauderman and Tompkins, 2016). Some collections require highly specialized preservation (e.g., cryo-preservation, alcohol immersion). For renewable collections of living specimens, maintaining and producing healthy and viable organisms are major costs. Beyond these two basic services, agencies vary widely in providing the other four services, depending on each agency' s mission and the support available. Documenting additions to collections involves the transfer of data and metadata (e.g., collecting location and date, collector' s name) from project collections and researchers or from other institutional collections. These data and metadata may contain errors and ambiguities that persist unless they are corrected at this stage or at a later stage through data curation. Metadata from renewable project collections can include instructions for maintaining living organisms, as well as data on collectors and collecting localities.
Providing access to users can include loan and visitor programs as well as digital access through web portals with information, digital images, and even trait data from the objects in the collection. This is a key service for increasing the benefits generated by institutional collections and their extramural users. These services make communities of users aware of the objects in collections and the rules governing their access and use. NSF is supporting Advancing Digitization of Biological Collections, 10 a 10-year funding initiative for non-Federal collections, to raise the visibility, discoverability, and use of collections through digitization. Collection managers are responsible for promoting access and use, while also safeguarding objects to ensure their long-term availability and fitness for use. Information about and access to some collections are not made publicly available. These include collections of virulent pathogens, crime investigations, and others that could pose a threat to public health and national security.
Data curation has developed rapidly in a few types of collections over the past decades, especially those with largescale digitization initiatives. Taken together, digitization and data curation have moved these collections into the realm of "big data." Data curation activities usually include: • Detecting and correcting errors; • Developing and implementing community-driven data standards and ontologies; • Developing automated translation of analog data to digital format; • Automating data quality control; • Developing and using standardized specimen/sample identifiers; and • Linking digital collection records (using standardized identifiers) to publications citing samples/specimens and data derived from samples/specimens in public databases. For example, DNA sequences in GenBank include references to the voucher specimens from which the sequences were derived; not all collections include references to related GenBank records. Park, 3+ million), as well as being featured in magazine articles and television programs. NSF' s support for scientific collections is awarded partly on the basis of their broader impact, including increasing public awareness of science.

Cost Categories and Accounting
The types of expenditures related to delivering these services fall into a small number of cost categories. Table 2 shows which cost categories are relevant to each of the services offered by collections. These associations between cost categories and particular services do not vary significantly among types of collections, geographic location, or agencies, though the costs within categories will vary. Viewing overall operating costs in the context of the services provided enables collection managers to make evidence-based management decisions, especially when the benefits arising from particular services are considered (see Implications for Policies and Management, below). Baker et al. (2014) presented cost data in terms of a similar breakdown of services provided by a non-Federal collection. The IWGSC (2009)  Presentations and discussions during this study confirmed that the availability of dedicated funding for the operation of institutional collections varies widely among agencies and even among collections in the same agency. Relatively few institutional collections have distinct budgets that support the cost of providing the services described above. In these few cases, the operating costs of the institutional collection are well documented and provide a solid basis for evaluating costs relative to the benefits generated by the collection. agency through an Interagency Agreement and the other is managed by a non-Federal contractor (see Table 3A for examples). These collections created systems for documenting and recovering all operating costs (see Method for Reconstructing Collection Budgets, below).
More commonly, a single agency budget will combine the operating costs of one or more institutional collections along with unrelated costs, such as: collecting activities; management of project collections; research activities including analyses of objects; and preparation of publications. Other agencies have considered collection-related budgets as too small to segregate so they have included them in much larger organizational units, sometimes unrelated to research. In these latter cases, special efforts are needed to identify the costs associated with the institutional collection (see examples, Table 3B).

Method for Reconstructing Collection Budgets
Agencies that do not have separate budgets for their collections or use contractors for collection management can document their operating costs in another way. NSF-ICF uses the following method to develop the budget for its Interagency Agreement with NSF. The International Cooperative Administrative Support Services is a similar system used by the U.S.
State Department to divide the cost of administrative services provided by the State Department (e.g., office space, IT support, utilities, motor pool) among the Federal agencies housed in each embassy. Table 2 shows the cost categories associated with each of six services that can be offered by an institutional collection.
This framework allows collection managers to reconstruct their operating budgets using the following procedure: • Itemize the full-time equivalents, square footage, number of computers and other measures of resource utilization in each cost category of the services they provide; • Calculate personnel costs directly using the compensation of the staff members involved in each service; • Use the budget of the parent organization to determine total support for space, utilities, IT and security services, and other cost categories used by the collection; and • Prorate the portion of space, utilities, security and other services used by the collection, using the appropriate utilization measure (e.g., number of computers for IT services, square footage for facility space and security).

A. Institutional Collections with Dedicated Budgets B. Institutional Collections without Dedicated Budgets
Two of the three NHANES institutional collections are stored and managed by non-CDC repositories under a contract with CDC. The budgets submitted by the contractors and paid by CDC document all operational costs other than personnel costs of CDC staff that oversee the contracts. These operating costs include storing and maintaining samples and distributing samples to users whose proposals have been approved after CDC staff review.
The third NHANES institutional collection includes samples directly accessioned into the CDC Biorepository after being collected from health survey participants. Management of the NHANES collection (storage, maintenance, and shipment of samples to qualified users) and several other institutional collections is provided by the CDC Biorepository. It can be difficult to know the operating costs of any particular collection in a centralized facility.
The National Science Foundation Ice Core Facility (NSF-ICF) 13 belongs to NSF but it is housed at the Denver Federal Center, where it is managed by the U.S. Geological Survey (USGS) and supported by an NSF-USGS Interagency Agreement. The agreement's budget represents the operating budget of the collection, including personnel costs of USGS staff.
The Food and Drug Administration (FDA) of the Department of Health and Human Services (HHS) has collections of food-borne pathogens and other food safety collections 14 that serve the agency's regulatory mission. Since the services provided by the collections serve regulatory activities that have separate budget line-items (e.g., seafood safety, marketplace inspections, food and feed safety, outbreak response), support for collections is distributed among several budget lines.
ARS has non-renewable institutional collections (e.g., preserved insects, fungi) used for agricultural research. They include objects relevant to several USDA commodity programs that contribute funds for research and collection operations. Combining support for research and collections obscures the true operating costs of the collections. 15 Once the services/cost categories have been established and their associated costs prorated from higher organizational

Cost Recovery
This study considered user fees as a mechanism for reducing costs, not for generating benefits or returns on investments.
Accordingly, we address the issue of cost recovery in this section on costs, rather than in the following section, which focuses on benefits. We found the full range of agency philosophies concerning cost recovery.
At one end of the spectrum, many agencies do not charge users any access fees. There is general concern among these agencies that charging users for access will reduce user interest, especially among potential users with limited resources (e.g., students, small institutions). Based on one year of records from three Canadian biobanks, Albert, Bartlett, Johnston, Schacter and Watson (2014) argued that cost recovery is probably limited to 5-25% of total operating costs. USDA/ARS maintains germplasm of agriculturally important plants, animals, microbes, and insects. ARS has no statutory authority to charge user fees for access, with one exception. 16 The 1990 Farm Bill established the USDA National Genetic Resources Program (NGRP), which includes renewable national germplasm collections of agriculturally important plants, animals, microbes, and insects. The Bill stipulated that the NGRP' s germplasm be made available to users free-of-charge. NSF-ICF is also proactive in promoting use and does not seek cost recovery through user fees.
Other agencies recover some or all operating costs. NSF' s Living Stock Centers Program 17 does not require grantees to recover full operating costs; they are encouraged to become more sustainable through user fees. The National Cancer Institute (Odeh, Miranda and Rao, et al. 2015) and the University of British Columbia 18 have developed software tools that help biobanks determine cost recovery fees based on data from cost categories. The NHANES Biospecimen Program charges a set fee per sample to cover some operational and transactional costs: collecting, storing, and processing samples; preparation of data files; and personnel costs associated with the process of reviewing proposals for access to samples. User fees are not intended to recover full operating costs.
At the other end of the spectrum, the National Institute of Standards and Technology (NIST) is required by statute (15 USC 275c) to recover costs related to its Standard Reference Materials (SRM) Program. NIST maintains clear documentation of the costs associated with developing, marketing, producing, and distributing SRMs. Estimating the number of units they expect to sell each year allows the program to set prices that will recover the required operational costs (Research   Triangle Institute, 2000).

M
any, perhaps all, IWGSC agencies have experience in justifying their requests for collection support by describing in various ways the benefits their collections generate for the agency and taxpayers. Their efforts to describe these benefits can use monetary terms (e.g., commercial revenue generated, cost reductions, productivity improvements). Rates of return on investment and benefit to cost ratios can be calculated in these cases, assuming that operating costs are also well known. Other benefits are specific but qualitative, and these are often described as impacts. IWGSC (2009) included a series of side-bar examples of these benefits, each identified with areas of impact such as public health (Horowitz et al., 2010;DiEuliis et al., 2016), environmental quality (Lawrey, 1993), or public safety and national security.
Other previous attempts to place a value on scientific collections have used qualitative terms (Suarez and Tsutsui 2004) or the cost of creating the collections .
In reviewing methods used for evaluating benefits generated by scientific collections, the study' s priority was finding evidence-based methods, regardless of whether the evidence was monetary, qualitative, or descriptive. The evidence used by methods reviewed during the study include: survey responses from users; market value data; historical records of collection use; expenditures by users on objects from collections; and potential savings to society through emergency mitigation.
The following sections describe five methodologies available to Federal agencies. Each one views the impacts of collections from slightly different perspectives and comes with particular assumptions. Each one has strengths and weaknesses; some are more time-and labor-intensive than others. Agencies may find that employing several approaches in combination is useful.

Technology/Knowledge Transfer
There is a substantial body of research into the monetary impact of federally funded research in science and technology.
Many studies have attempted to trace products in the marketplace and the jobs and wealth they generate back to their origins in government grants and research labs (see review by Ammon, Salter, and Martin, 2001). These studies must address the delays inherent in the R&D process leading from discovery through application, proof of concept, patenting, product development and testing, licensing, manufacture and eventual marketing. They must also somehow apportion the eventual market value of new products among all the contributing links in this value chain. For example, new crop varieties have been developed from accessions obtained from plant germplasm collections (see Box 2) but the monetary value of the new crops was also due to crop breeding that may have involved crops already in production, improvements in farming technique, and other inputs (Rubenstein, Heisey, Shoemaker, Sullivan, and Frisvold, 2015). Partitioning the net value of the new crop among these inputs would require replicate treatments that isolate and control for each contributing factor (e.g., Evenson and Gollin, 1997;Güereña, Lehmann, Thies, Enders, Karanja and Neufeldt, 2015). Since the funds invested in the original research were unavailable for other uses during this delay, appropriate discount rates must be applied to adjust for the actual cost of the research (including opportunity costs) prior to calculating returns on investment.
This same approach can be applied to scientific collections, which are often the basis of research leading to discover- Collections of living microbial strains (often referred to as "culture collections") are renewable collections that are used for basic research in microbiology, and for applied research in a variety of commercial areas (e.g., agribusiness, pharmaceuticals). Furman and Stern (2011) discuss the economic valuation of Biological Resource Centers such as the ARS Culture Collection.
Agencies involved in safeguarding public health face several challenges in demonstrating the benefits generated by the use of institutional collections. There is little doubt that they contribute to the prevention, detection, and cure of diseases, but the precise pathways between the use of collections and tangible, measurable benefits can be difficult to follow (see Box 4).
Economic analyses of technology/knowledge transfer that include benefit to cost estimates involve considerable effort and expense. They are normally done by contractors who specialize in economic studies. As an alternative, agencies can

Box 2. National Plant Germplasm System
NPGS is a distributed network of 25 plant genebanks and support labs cooperatively operated by USDA/ARS, State Agricultural Experiment Stations, and Land-Grant Universities. The repositories receive, preserve, maintain, and distribute germplasm samples and associated information to support agricultural production by making germplasm available to users around the world, including researchers, plant breeders, growers, and other qualified users. NPGS is a renewable collection so preservation and maintenance involve propagating and testing plants to ensure their health and viability. User requests for germplasm samples vary widely, including requests for highly characterized accessions that are already cultivated as crops. These may be used in cultivation or in plant breeding aimed at developing new and more productive cultivars.
Other users request wild plant relatives of crops that are poorly characterized and have no history as crop plants. These may be used for basic research and to identify, isolate and incorporate useful traits into new plant varieties.
The benefits generated by NPGS take many forms: increased agricultural output; reduced losses to pests and environmental stress; increased knowledge that may contribute to food security; and increased potential to respond to food insecurity. Some of these can be measured quantitatively (Bretting, 2018), but sources other than NPGS also played important roles in generating these measurable benefits.
Photo courtesy of USDA ARS use qualitative terms to document the use of their collections by commercial entities, or non-commercial users who are working on commercial development projects. This can be accomplished by adding data gathering from users of a collection to other data curation tasks.

Success Stories
Several of the sidebar examples in IWGSC (2009) are "success stories"-case studies in which collections helped to solve a problem or prevent losses that represented millions to billions of dollars. For example, USDA's collections of agricultural pests help prevent catastrophic crop losses or trade wars over disagreements about imports with dangerous insects (see Box 8). Vaccines developed using samples in collections can curtail epidemics and save lives and avoid productivity losses, documented in "cost-of-illness" economic studies (Byford, Torgerson, and Raftery, 2000). Other examples are used

Box 3. ARS Culture Collection
The ARS Culture Collection (also known as the Northern Regional Research

Lab Collection [NRRL]) is one of the largest public collections of bacteria and fungi in the world. It is housed within the Mycotoxin Prevention and Applied Microbiology Research Unit at the National Center for Agricultural
Utilization Research in Peoria, Illinois. NRRL's intramural research focuses on advancing agricultural production, food safety, public health, and economic development. Data about strains in the collection provided by depositors, users, and intramural staff are added to NRRL's public database, which improves the community's ability to find samples for further study. NRRL includes two collections.
• The "Open Collection" contains 90,000 microbial strains that are owned by USDA and are made available to academic and commercial researchers without charge. These isolates represent a broad sample of biological diversity collected over more than a century. On average, 4,000 microbial cultures are distributed by the NRRL each year at no cost to users. If obtained from private culture centers, users would be charged approximately $1 million for these samples. In 2018, NRRL isolates were provided to government, academic, and industry scientists across the U.S. and 42 other countries. NRRL contributes directly and indirectly to technology development and business enterprises. Direct benefits can be traced through patents that cite NRRL samples. NRRL samples contribute indirectly to technology development through the new knowledge presented in research publications that cite them. NRRL isolates have been cited in more than 65,000 scientific publications, as well as 7,500 patents. A formal economic analysis of the monetary value of these direct and indirect contributions has not been attempted.
Photo courtesy of USDA ARS routinely by agencies to highlight the potential value of collections that can provide high-impact solutions to applied problems. However, the benefits generated by success stories are often in areas other than the one for which they were collected.
That is, the benefits are not the products of the day-to-day work of the collection or even the mission of the agency that owns the collection. For example, IWGSC (2009) included a sidebar about geologic rock cores that were collected for oil and gas discovery but are also proving valuable for mapping and predicting earthquakes. These success stories can be effective for public relations and for raising awareness about collections, but they are not often viewed as compelling evidence of returns on investment or estimators of benefit to cost ratios.
Economists sometimes draw an analogy between success stories and winning lottery tickets. Both involve a small investment (for a lottery ticket or a few objects in a collection), a very large payoff, and very low probabilities of success. They also point out some important differences. Lotteries have a known delay between purchasing a ticket and when the winner is chosen, but the waiting time until the next collection-based success story is unknowable. In addition, the value of a winning lottery ticket is known when the winner is drawn, but there is no way to predict the value of a solution that might be found in a collection. Finally, a lottery has at least one guaranteed winner each time the winning number is picked, but there' s no guarantee that the solution to a critical problem is waiting somewhere in a collection. The random, unpredictable nature of success stories may limit their value in demonstrating returns on investment. The following example of success stories, and the subsequent example of option values, may suggest ways that agencies can highlight benefits that are valuable but unpredictable.
The National Park Service (NPS) protects and preserves an extraordinary range of habitats and life forms, from high alpine mammals to marine microorganisms. A well-known success story involves a bacterial species, Thermus aquaticus, collected and described from a hot spring in Yellowstone National Park. It was found to produce an enzyme that catalyzed

Box 4. NHANES Value Chain
The National Health and Nutrition Examination Survey (NHANES) is a unique public health program that obtains blood and urine samples at  Bishop (1982) and Fisher and Haneman (1986) described "option value" as the benefit of having something available in case it is needed in the future, even though the probability of future use and its future value are unknown. This concept of options is used in the sale and purchase of stocks, wines, and works of literature that might become the basis of commercial films (to name a few examples). Option value can be thought of as an insurance policy that is only worth something in the occurance of an unforeseen event. Option values are therefore related to the values claimed in success stories, described above, with one important difference. As used here, option values are directly related to agency missions and the everyday uses of collections.

Option Value
The missions of many Federal collections include preventing or mitigating threats to public health and safety, such as risks to the nation' s food supply. For example, NPGS is a resource for developing new and better crops (see Box 2). FDA' s Foodborne Pathogen Collections are critical for ensuring the safety of food in the marketplace (see Box 6).
The U.S. has witnessed many such threats to our food security: declines in agricultural output of specific crop varieties, foodborne disease outbreaks, and crop failures due to the introduction of insects, mold, fungi, and other agricultural pests. The same is true for epidemics, airline crashes due to bird strikes, earthquakes and other threats to public safety described in IWGSC (2009). The economic impacts of many of such events have been estimated. Agencies could compile focused knowledge bases of past events that illustrate the scope of potential losses that face the country. These past losses illustrate the option value of maintaining collections that could prevent or mitigate these losses.

Value Added by Users
Success stories and option values are based on rare and unpredictable uses of collections. In contrast, this approach to documenting benefits is based on normal, everyday activities in collections. "Providing access to users" is one of the services listed in Table 1. Since this service increases the number of researchers who use a collection, it can be viewed as the service which has the greatest impact on the potential benefits generated by a collection. Collections often explain their value in terms of growth (accessions per year), or activity (numbers of visitors who use the collections, or numbers of specimens/ samples distributed to users), but these are input measures, not indicators of outputs or benefits. Some collections try to report the publications and/or datasets generated by users, though collecting this information from users is often difficult.
Published articles and datasets are important outputs, but prior to their use by others it is difficult to assess their beneficial value beyond the professional standing of the authors. Collections will often highlight important discoveries made by users of the collection, similar in some ways to success stories described above.
The Core Research Center (CRC) of the U.S. Geological Survey 26 has been proactive in promoting community use of the collection and increasing the discoverability of objects in the collection through data curation (see Box 7). Their policies

Box 6. FDA Foodborne Bacteria Collections
The The FDA has not attempted to trace monetary or other impact of the use of its collections. These impacts are generated through new and improved capabilities of public health agencies, hospitals, universities, and private companies that prevent and respond to disease outbreaks caused by foodborne pathogens. However, the use of FDA collections by these sectors is known and can be considered in the context of the foodborne disease outbreaks each year and their cost to the Nation.
Photo courtesy of HHS/FDA and practices are comparable to those of companies that re-invest profits in their R&D efforts in order to increase future productivity. CRC' s "virtuous cycle" is based on promoting use of the collection, capturing and incorporating the outputs from this use into the collection, and thereby increasing future use. CDC' s NHANES Biospecimen Program also collects the results of laboratory testing done by intra-and extramural users and connects these data to the NHANES database. A list of publications resulting from studies using NHANES institutional collection samples are provided on the NHANES Biospecimen website.
When the results of analyses and sample preparations done by users are integrated back into a collection and its public databases through data curation, future users are more likely to find the samples, specimens, and data they need.
The collection' s value to those users has increased. Accordingly, this study views the expenses borne by users for analytical procedures and sample preparations as "co-investments" in the collection and a form of return on investment.
Patterns of user demand are the basis for understanding and documenting co-investments in collections. For example, user demand for objects in a collection might be high soon after they are added to the collection but decline soon after. Alternatively, user demand might be unrelated to the length of time since objects were accessioned. To determine the degree to which collections retain the user interest that drives co-investment, the study obtained historical data on user demand over time from several Federal collections.
USGS' s CRC has accepted donated cores from 1974 to the present, with most coming from around the 1980s. The core samples requested from 2016 to 2018 showed this same age distribution (see Figure 2).
The Smithsonian' s National Museum of Natural History (NMNH) has more than 125 million specimens and samples of plants, animals, fossils, rocks, minerals, and human artifacts. NMNH, which began computerizing its loan records in the 1960s, provided information on more than 120,000 loan requests from nine scientific departments. Figure 3 shows loan data from the Invertebrate Zoology Department, which had the most complete data.

Box 7. USGS Core Research Center (CRC)
CRC is an institutional collection 27 that contains approximately 10,000 rock cores and more than 50,000 borehole cuttings, 95% of which were donated by private companies whose intramural use of the cores did not justify the costs of storage and maintenance. USGS evaluates the quality and rarity of donations before accepting them into the Denver, CO based repository. The collection includes cores    2 and, clearly, 3 show that users request access to samples and specimens in proportion to their representation in the collection. There is no evidence that their value to users diminishes over time. This suggests that the value added through co-investment by users over time could be substantial.
Some agencies with collections are proactive in documenting their collections and promoting their extramural use. In order to document patterns of use and co-investment by users, agencies will need to have systems for capturing data on: • Research visitors and requests for loans and sample distributions, and • Publications and datasets released on public data repositories.
To incorporate the value generated by users into the collection (making them co-investments that add value), the collection must also provide data curation services. This would involve: • Receiving and curating sample preparations done by users and returned to the collection; • Adding metadata about the returned sample preparations to the digital records of the original sample or specimen; • Capturing the metadata, digital identifiers, and web addresses of publications and datasets released on public data repositories; and • Incorporating these metadata, identifiers, and addresses into the digital records of the original sample or specimen.

Counter-Factual Scenarios
Counter-factual scenarios are well-established devices for economic analyses. They shift the frame of reference normally used to evaluate the value of scientific collections. Rather than exploring the value of collections to users, counter-factual scenarios explore the costs to users of not having access to the collections. Two examples are presented here.
In a study of USDA's Animal and Plant Health Inspection Services (APHIS; see Box 8 and Lichtenberg, Olson and Lawley, 2009), the agency was the principal user of the collection. The absence of the collection would have had a direct and clear effect on the agency' s ability to fulfill one of its core mission responsibilities. This allowed the study to use programmatic data to estimate the financial impact of not having access to the collections.
Companies were the focus of a study of the benefits generated by NIST' s SRM Program (see Box 9 and Martin, Gallaher and O'Connor, 2000). The study relied on user surveys and interviews to estimate the financial impact on the companies if they did not have SRMs. NIST does not consider the SRM Program a scientific collection, though it has many of the characteristics of renewable collections as described in Box 1, and it provides some of the services described in Table 1. NIST' s use of counter-factual scenarios may therefore be instructive for Federal scientific collections.
Economic analyses that employ counter-factual scenarios are data-rich but labor-intensive. These studies are normally done by contractors rather than agency staff to avoid any appearance of conflicts of interest and to increase credibility. The APHIS study (Box 8) relied on programmatic data because the absence of collections (the counter-factual premise) would result in clear consequences (rejection of certain imports at ports of entry). The NIST study (Box 9) relied on structured interviews of users because there were many possible consequences to users if SRMs did not exist.
The Paperwork Reduction Act limits the burden placed on non-Federal survey respondents by restricting Federal surveys to nine requests for information. OMB can issue waivers based on formal requests and some agencies have been granted waivers for cause. Complying with the Government Performance and Results Act by conducting economic analyses is one possible basis for requesting OMB waivers. of what constitutes "value," "benefits," and "returns on investment." They may be looking for cost savings, or ways of increasing cost-effectiveness, or seeking ways to make management and policies more evidence-based. No single evaluation method can give clear and simple benefit-to-cost ratios that will address these questions. The methods described here will shed new light on these matters in different ways, and the following table of strengths and weaknesses may help agencies select appropriate methods.

Box 8. Counter-Factual Scenario for USDA's Animal Plant Health Inspection Services
Customs and Border Protection (CBP) staff inspect incoming agricultural shipments at U.S. ports of entry for potential agricultural pests. When they encounter evidence of insects, fungi or other potential pests, they collect, preserve and send them to USDA/APHIS area identifiers, located at larger ports and plant inspection stations around the United States. APHIS area identifiers provide the first layer of authoritative identification for commonly intercepted and unambiguously identifiable pests.
When the area identifiers cannot identify them with certainty, they send samples to APHIS National Identification Services (NIS) for identification. NIS has a staff of National Taxonomists APHIS inspection and USDA shipment records showed that $180 million in imports fell into these categories over one year (mid-2006 to mid-2007). All inspection-related APHIS and ARS research and collection costs totaled $27 million during this period, resulting in a benefit to cost ratio of 4.87.

USDA APHIS Image Gallery
Photo by Jim Young, USDA

Box 9. Counter-factual scenario for NIST's Standard Reference Materials Program
NIST produces and sells more than 1,300 types of highly characterized and standardized SRMs to industry, academia, and government-including companies that develop, manufacture, and or use analytical instruments that are critical to assuring quality, verifying accuracy of measurements, and ensuring compliance with Federal regulations. The SRM Program shares similarities with renewable collections as defined in this report. The approach taken by NIST using economic impact analyses for the SRM Program highlights an opportunity for institutional collections.
An economic analysis was commissioned by NIST to study the value of SRMs developed to measure the sulfur content of fossil fuels (Research Triangle Institute, 2000). The study interviewed representatives of nine companies from industries (e.g., coal processing, oil refining, steel production) that purchased the sulfur SRMs, asking them to report

Implications for Policies and Management
T he methods described here for documenting and estimating costs and benefits can equip agencies to make two evidence-based decisions: • What kinds of objects should they accession each year, and how many of them?
• Which services should they provide, and how much of them?
Some agencies operate under authorizing legislation (including the organic acts that created them), in which requirements to retain Federal ownership of certain objects is specified or implied. This may limit an agency' s ability to make evidence-based decisions on what kinds and how many objects they accession. The following discussion explores the tensions and trade-offs caused by unfunded mandates, and how these mandates may come into conflict with the value of understanding operating costs as called for in America COMPETES.
In containing costs and generating benefits, the managers of Federal institutional collections and their agency leadership face three structural constraints on policies and management: A. All agencies with collections face decisions on the intake and the removal of objects from their collections in order to maximize potential use and future benefits (within budget constraints), while minimizing the risk of not having important objects when need for them arises. Agencies lose the ability to make informed decisions when legislation or policies mandate that an agency must obtain and keep certain objects; B. Attracting resources (facilities, staff, and funding for operating costs) enable collections to generate benefits that advance the agency' s mission, but budget processes and environments are often zero-sum or declining. Unfunded mandates to obtain and keep certain objects limit an agency' s abilities to make informed decisions about resource allocation; and C. Expanding the range of collection-related services provided by a collection can generate benefits that advance the agency' s mission, but offering too many services (some of which may be mandated by law) may limit the quality and impact of each service provided.

Constraint A
Decisions about which objects to obtain, retain, and discard are difficult because future use and impact are unknown.
Knowledge of future operational costs per object and agency priorities for areas of future benefits provide valuable guidance for these decisions. When agencies are required by legislation to maintain and preserve whole categories of objects, they lose control over future costs and the ability to pursue particular benefits.
NIH has developed policies concerning the management of their project collections; USGS has done the same for its working collections. These policies 30

Constraint B
Federal appropriations are the primary source of support for the services provided by Federal collections, so funding levels limit the amount of services the collections can provide. The growth, maintenance, and preservation of collections generate most of their costs and without adequate support, these basic services can limit the ability to offer other services that generate tangible benefits, especially user access, data curation, and education and outreach.
Legislative, regulatory, and policy mandates have significantly increased the number of objects that agencies must accession, maintain, and preserve, thereby limiting the resources available for support of objects that serve other parts of an agency' s mission. For example, DOI' s collections contain tens of millions of scientific objects, many of them added to DOI collections in compliance with legal and policy mandates.
Financial and management arrangements can be difficult when ownership and stewardship are assigned to different institutions. Many DOI collections are housed and managed by non-Federal repositories, often those of the researchers who made the collections. DOI retains ownership of these collections but non-Federal institutions are their stewards. DOI' s appropriations have not been adequate for supporting collection services. 33 In addition, these non-Federal institutions are not eligible to receive NSF funding for projects that would improve collections, or portions of collections, that are owned by the Federal Government.

Constraint C
Beyond the basic services of accessioning, documenting, maintaining, and preserving, collections face important and difficult decisions about which services to offer. If a collection or agency tries to offer other services, they may not be able to devote the resources needed to generate benefits, and they are at risk of being accused of "mission creep." Even basic services such as preservation may suffer. Once policies about accessioning and services are set, it is sometimes necessary to reduce collection growth and services if increases in funding and other resources do not keep pace. Such reductions are likely to reduce the benefits generated, so they are very difficult to explain to collection stakeholders.
Decisions about which services a collection should offer are at the center of the relationship between costs and benefits. That is, the services provided by a collection drive its operational costs, the types and amounts of benefits the collection can be expected to generate, and the appropriate methods used to document and estimate those benefits. Collections that perform only the basic service-accessioning material-have lower costs that are easier to document, but benefits will be more difficult to generate if no other services are provided.
Beyond the basic service of accessioning, each additional service increases costs, but they may also increase the types and amounts of benefits a collection can generate: • Preserving and maintaining a collection extends the time that objects will survive and can produce reliable analytical results; • Documenting additions to a collection will facilitate use by intramural researchers; • Providing access to users will expand the user base to the extramural research community, including other countries and scientific disciplines; • Data curation will add value to the collection and permit the collection to document the benefits of this co-investment; and • Increasing public understanding through education and outreach through collections can create societal benefits beyond research and development.

Recommendations
T his report was based on examples and experiences obtained from Federal scientific collections, but the following recommendations may be applicable to scientific collections in general.
A. The framework of services, costs, and benefits described here provides collections and organizations which own collections with an approach for greater evidence-based policy formulation and management decision-making. Federal institutional collections should consider testing and adopting them as tools for improving operations, as well as for documenting and explaining the value of their collections to taxpayers.
B. IWGSC member agencies should consider testing one or more of the methods presented here for documenting the benefits generated by collections. Several would require new data collecting efforts and added expense. CDC NHANES (Box 4) and USGS/CRC (Box 7) are collecting and providing access to value added data provided by users; USDA/ APHIS (Box 8) is using agency data for analysis in a counter-factual scenario.
C. In choosing among the methods presented here for documenting benefits, officials should consider their mission and the types of benefits their collections can generate. For example, i. Collections that contribute more directly to economic development might favor technology/knowledge transfer and/or counter-factual scenarios. The former can identify the collection uses related to successful innovations or outcomes, and the latter can document the costs of not having the collections; ii. Collections that contribute to societal benefits by preparing for environmental shocks (e.g., disease outbreaks, major crop failures) might find success stories and option value more useful. Collection managers can use the former method to highlight recent events in which their collections came into use, while the latter method can describe the costs of similar shocks in the past.
iii. Collections that primarily contribute new knowledge in the form of public data and academic publications might prefer value added by users and counter-factual scenarios. As described above, collections can gather data on patterns of collection use and co-investment by users, and can ask selected users (through surveys) what they would have done if they did not have access to the collection.
D. Groups such as the IWGSC, the International Society for Biological and Environmental Repositories (ISBER), the Society for the Preservation of Natural History Collections (SPNHC) and others should continue in their roles as forums for information exchange and sharing of best practices as they apply the methods described here. As collections and organizations which own collections begin to generate reports and other documents concerning costs and benefits, the IWGSC Clearinghouse 34 can continue to serve as a useful platform for information exchange.