It all depends on what your web site is doing. We have .NET servers running on 32 GB RAM, and the worker process gladly takes all it wants. It needs it really.
Are you running a lot of background threads, stored an abhorrent amount of data in the session/global application / static methods?
Are connection strings, readers, file I/O, etc. being closed properly?
.NET garbage collection works great, but you have to do your part. Garbage collectors (waste disposal engineers) are not going to go into your house and collect it for you, you have to at least walk it to the curb... or close / null / dispose of objects you're not using.
Update 1:
What is happening is the ASP.NET worker process is creating a buffer. When it hits a certain amount of memory used, it will decrease over time, but it likes to allocate memory so it doesn't have to go out of it's way to fetch it when it needs it.