Lemmy.zip

2,936 readers
202 users here now

Welcome to Lemmy.zip - a community for like minded people to come and have a chat about almost anything. From games to tech, to anything else, come and have a chat.

If you're new and would like to join Lemmy.zip, please fill in the sign up form. Email verification is required. (Please check your spam folder!)

Once you're signed up, come and introduce yourself in our Home community!

Useful Links

m.lemmy.zip - Our Mobile/Tablet site
old.lemmy.zip - A familiar desktop experience!
a.lemmy.zip - A desktop-first UI
t.lemmy.zip - An alternative Mobile/Tablet UI with lots of features!
Join our Matrix Chat!

Instance Rules

To maintain the high standard of discourse and interaction we all value, each user must adhere to the guidelines outlined in our Code of Conduct. This set of rules is designed not just to maintain order but also to ensure a safe and inclusive environment for everyone to share their thoughts and ideas.

What to Expect in Our Code of Conduct:

Respectful Communication: We strive for positive, constructive dialogue and encourage all members to engage with one another in a courteous and understanding manner.
Inclusivity: Embracing diversity is at the core of our community. We welcome members from all walks of life and expect interactions to be conducted without discrimination.
Privacy: Your privacy is paramount. Please respect the privacy of others just as you expect yours to be treated. Personal information should never be shared without consent.
Integrity: We believe in the integrity of speech and action. As such, honesty is expected, and deceptive practices are strictly prohibited.
Collaboration: Whether you're here to learn, teach, or simply engage in discussion, collaboration is key. Support your fellow members and contribute positively to shared learning and growth.

If you enjoy reading legal stuff, you can check out legal.lemmy.zip.

Funding

If you would like to contribute to the upkeep of Lemmy.zip, please head over to OpenCollective.
Anything you're happy to donate is very highly appreciated!
You'll even get your name in the Thank You thread.

If you want to use PayPal, you can donate via Ko-Fi:

Server

founded 1 year ago

ADMINS

Demigodrick

Sami

ZippyBot

Music ControlNet: Multiple Time-varying Controls for Music Generation (musiccontrolnet.github.io)

submitted 1 year ago by BrikoX to c/interestingshare

0 comments fedilink

Text-to-music generation models are now capable of generating high-quality music audio in broad styles. However, text control is primarily suitable for the manipulation of global musical attributes like genre, mood, and tempo, and is less suitable for precise control over time-varying attributes such as the positions of beats in time or the changing dynamics of the music. We propose Music ControlNet, a diffusion-based music generation model that offers multiple precise, time-varying controls over generated audio. To imbue text-to-music models with time-varying control, we propose an approach analogous to pixel-wise control of the image-domain ControlNet method. Specifically, we extract controls from training audio yielding paired data, and fine-tune a diffusion-based conditional generative model over audio spectrograms given melody, dynamics, and rhythm controls. While the image-domain Uni-ControlNet method already allows generation with any subset of controls, we devise a new strategy to allow creators to input controls that are only partially specified in time. We evaluate both on controls extracted from audio and controls we expect creators to provide, demonstrating that we can generate realistic music that corresponds to control inputs in both settings. While few comparable music generation models exist, we benchmark against MusicGen, a recent model that accepts text and melody input, and show that our model generates music that is 49% more faithful to input melodies despite having 35x fewer parameters, training on 11x less data, and enabling two additional forms of time-varying control.

Music ControlNet: Multiple Time-Varying Controls for Music Generation (musiccontrolnet.github.io)

submitted 1 year ago by [email protected] to c/[email protected]

0 comments fedilink

There is a discussion on Hacker News, but feel free to comment here as well.