Yes, you have to loop through all of your elements.
That's how nearly all hit testing/object picking works.
Now there are a lot of optimizations you can do, like making sure that the mouse coordinate is in the rectangular bounds of the object before you test its real bounds (unless they're all rectangles, which would make everything convenient and fast, even if there are 10,000 objects). If you have a million objects, you could also split up their location into quadrants, or otherwise add a layer of isolation.
But at the end of the day you need to loop through every relevant object and ask if its bounds intersect the mouse point. There's no way of getting around that. It ought to be fast, and if you've implemented something that is very slow then you should post it so we can take a look and give you pointers.
If you need to see a live example I made a simple tutorial on picking and moving shapes in a canvas exactly for explaining situations like this.