An Improved Neuro-Symbolic Architecture to Fine-Tune Generative AI Systems
(2024) CPAIOR 2024
Files
978-3-031-60599-4.pdf
Open Access - Adobe PDF
- 15.08 MB
Details
- Authors
- Abstract
- Deep generative models excel at replicating the mechanisms that gen- erate a specific set of sequential data. However, learning the underlying con- straints preventing the generation of forbidden sequences poses a challenge. Recently, RL-Tuner, a reinforcement learning framework designed for the ad hoc fine-tuning of a neural model to adhere to given constraints, was enhanced to learn from the output of two constraint programming models. The first model computes a score representing the number of constraint violations from the cur- rently generated token while the second model provides the marginal probability of that token being generated if no additional violation is allowed. In this paper, we significantly enhance the latter framework in three ways. First, we propose a simplified architecture that requires only a single constraint programming model. Second, we evaluate constraint violations in a more accurate and consistent man- ner. Third, we propose a reward signal based on belief propagation on this new model that further improves performance. Our experiments, conducted on the same learning task of music generation, demonstrate that our approach surpasses the previous framework both in terms of convergence speed during training and in post-training accuracy. Additionally, our approach exhibits superior general- ization to longer sequences than those used during training.
- Affiliations
Citations
Yin, C., Cappart, Q., & Pesant, G. (2024). An Improved Neuro-Symbolic Architecture to Fine-Tune Generative AI Systems. CPAIOR 2024, Uppsala, Sweden, May 28–31, 2024, Proceedings, Part II, 14743, 279-288. https://doi.org/10.1007/978-3-031-60599-4_19 (Original work published 2024)