If you haven't worked much with Android before, specifically with capturing and overlaying its video stream data I would say that creating an augmented reality application would be very difficult. I worked on a project capturing and analyzing live video streams through android for on-device analysis and found it significantly challenging.
If you're looking for an alternative I would suggest a static mapping application that showed a blueprint-like layout of the library, your current location at the terminal that provides the Dewey Decimal listing, and the calculated location of the book they just looked up. You can even add a section about the Dewey Decimal classification system and become your librarian's best friend.
If you really want to go for live navigation indoors your best bet may be through a combination of the compass and accelerometer data so you can infer direction and distance as a user moves through the environment. There are some good papers around in regard to this but you may need to go through the IEEE or ACM libraries to look them up.