A unified framework for Singing Voice Synthesis (SVS) and Conversion (SVC), addressing the limitations of existing approaches in cross-domain SVS/SVC, poor output musicality, and scarcity of singing data. This framework enables control over multiple aspects, including language content based on lyrics, performance attributes based on a musical score, singing style and vocal techniques based on a selector, and voice identity based on a speech sample. 

An SVS framework that leverages a cascade of diffusion models to generate realistic singing across multiple languages, styles, and techniques from scores and lyrics.

A large, studio-quality, monophonic singing dataset, covering111 songs(224 versions), 8 singers, 12.8 hours, 3 languages, and various singing styles (including creative improvisations). It involves style transfer demonstration: 80 songs covering at least two distinct singing styles performed by the same singer. Detailed human annotations such as phrase segmentation, lyrics phoneme-audio alignment, performance MIDI, score.

With Music Frameworks (a hierarchical music structure representation) and new musical features, we combine music domain knowledge with deep learning, and factor music generation into sub-problems, which allows simpler models, requires less data and achieves high musicality.

What is missing in deep music generation? A study of repetition and structure in popular music, that illustrate important music construction principles by the analyses of two popular music datasets (Chinese and American).  It offers challenges as well as opportunities for deep-learning music generation and suggest new formal music criteria and evaluation methods.

Introduces new algorithms for identifying a two-level hierarchical music structure based on repetition. Automatically detected hierarchical repetition structures reveal significant interactions between structure and chord progressions, melody and rhythm. Different levels of hierarchy interact differently.

Designed a stylistic music generation system that is able to capture structure, melody, chord progression and bass styles from one or a few example music, and imitate the styles in a new piece using statistical machine learning models. 

HCMP is an emerging computer music system that can perform live music in association with human performers, with goal of creating highly autonomous artificial performers that can fill human roles

Designed series of computational models for common pipa performance techniques using "analysis-by-synthesis" method, leading to much more realistic synthesized performances

Developed mobile web app with SuperCollider that lets people use mobile gestures (speeds, ranges, directions) to control melodies (such as ringtones) in real-time as if they were playing musical instruments, adjusting pitch, volume, tempo, accompaniment and special effects etc., allowing group of people to form ringtone orchestra