Exploiting Double Representation and Gated Multi-Head Cross-Attention for Polymers Properties Prediction

Abstract

The accurate prediction of chemical properties in polymers is a significant challenge in Materials Science, often hindered by the complexity of polymer structures and the limitations of traditional modeling approaches that rely on classical string notations like P-SMILES. In this respect, a multimodal approach where also structural information of polymers is taken into account, can lead to improved performance. The present work tackles this challenge with a novel neural framework that integrates a graph-based embedding into the string embedding of a Transformer neural network, aiming at performance improvement by leveraging structural information provided by the graph representation. We called our approach Double Representation (DR) and in this work we explore the relationships between string- and graph-based polymer description by evaluating two distinct Transformer architectures on different polymer data sets that are reported in the literature. Our results reveal that the DR approach markedly improved the performance of the ChemBERTa model across various polymer properties. The introduction of a particular Gated Multi-Head Cross Attention operation to combine the two embeddings has improved the results by leveraging the strengths and advantages of both representations. These findings not only contribute to the advancement of predictive modeling in Polymer Science but also have significant implications for material design and development, setting the stage for future research aimed at refining property prediction methodologies through innovative data integration techniques