EDIT: As recursive pointed out, you're not simply looking for a way to read/write midi files. I misunderstood. If you're trying to extract notes from a mp3/wav file, you won't have much luck finding any library that does that. It's not as simple, since you're working with essentially analog data instead of digital. I've seen some application that will at least show you a spectrograph of your audio, and if the audio is clean enough and only a single instrument, you can see what appears to be the notes. But you'd have to figure it all out manually. Izotope RX was the VST I was thinking of.
I had this problem maybe 5 or 6 years ago and eventually started writing my own (not as easy as it sounds). I got something working to read, but not write, but eventually ran out of time to work on the project.
I would try NAudio. I can't specifically vouch for it, as I haven't personally used it. However, the feature list says it has extensive support for reading and writing MIDI files.