The Smithsonian Institution
Browse
Trizna Digital Data 2022 Presentation.pdf (2.77 MB)

Making BHL scientific illustrations searchable for non-scientists

Download (2.77 MB)
presentation
posted on 2024-03-15, 04:07 authored by Michael TriznaMichael Trizna

Presented at 2022 iDigBio Digital Data Conference

The Biodiversity Heritage Library has uploaded over 300,000 open source images extracted from its vast collection to the popular image hosting site Flickr. The images -- most of which are colorful scientific illustrations of organisms from across the tree of life -- are meticulously organized by source publication, and labeled with taxonomic tags. However it is very difficult to search by general descriptions like "a watercolor illustration of an insect with flowers" that might not be picked up by the tags. To enable these types of searches, I used OpenAI's CLIP (Contrastive Language–Image Pre-training) deep learning model to produce a vector representation of all images from the Flickr dataset. The CLIP model is trained on image-text pairs, so we can then encode text search terms with the same model, and find nearest neighbors from the BHL image set. Finally, a front-end interface was created using the Python framework Streamlit to allow for searching and exploration. Source code can be accessed at https://github.com/MikeTrizna/bhl_flickr_search.

History

Usage metrics

    Office of the Chief Information Officer

    Categories

    Licence

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC