Knowledge Graph Embedding & GAN for Medieval Manuscript Studies
We will develop innovative AI techniques to “augment” the historical content of texts and images to study cultural meanings and representations of music and sound in the Middle Ages. Our aim is to develop techniques to both (i) recognize or at least detect textual elements related to performance (words, phrases, etc.) in texts between the 5th and 15th century, and (ii) identify performances (musicians, instruments) in images. The first challenge is to incorporate different data sources lacking explicit contextual linking. Because of the complexity of the subject — the variety of sources, languages,
and artistic contexts — we would like to continue implementing more complete transcultural and diachronic knowledge graphs. Since, knowledge graph embedding techniques have not been exploited in the medieval musicology domain; we will explore different approaches using purely knowledge graph embedding, visual embedding or combined embedding to calculate the similarity between items. Another major challenge in Deep Learning is the need for vast amounts of labeled data for training the images. In
this project, we will use Generative Adversarial Networks (GAN) and Neural Network Style Transfer (NNST) for the generation of large training databases and introduce a pioneering approach towards the generation and completion of historical images containing musical instruments. Furthermore, we will introduce the idea of using GAN for domain transfer between these two domains, i.e., historical representations to “real-life” representations and vice versa. This will be useful for data augmentation as well as image enhancement and completion. Natural Language Processing (NLP) techniques will extract references to musical performances in texts: multilingual for medieval languages and multiple versions of
descriptions are a novel challenge.
Keywords
Knowledge graphs, Ontologies, Cultural Heritage, IIIF, Knowledge graph embedding, Graph Neural Networks, Generative Adversarial Networks (GAN), Neural Network Style Transfer (NNST).
Contact
Dr Xavier Fresquet
Deputy Director of SCAI
phone: + 33 1 44 27 76 05
email: xavier.fresquet@sorbonne-universite.fr
4, Place Jussieu
75005 Paris