Author of RFC 7033 here.
WebFinger was a work-in-progress for several of years and underwent a number of changes during that time. RFC 6415 was the first attempt to standardize the concept of WebFinger, which included host-meta and LRDD. The process of discovery using RFC 6415 was complicated by the fact that one needed to perform two queries and then merge information from each of the queries to create a resulting set of link relations. Also, there had been a move for a while toward JSON. WebFinger had used XML, but RFC 6415 Appendix A introduced a JSON encoding. People wanted that to be the only encoding.
Working with the original authors of RFC 6415 and others in the WebFinger community, a group of us in the IETF worked to simplify the process, make the move to JSON as the content encoding, ensure the solution was secure (HTTPS only), and get agreement on a URI scheme for querying user account information (the "acct" URI).
So with RFC 7033 we have a secure, simple, one-query mechanism for discovery that works basically like this:
$ wfinger paulej@packetizer.com
What this "wfinger" client would do is find the domain "packetizer.com" and then issue the following query (using curl just to make the example clear):
$ curl https://packetizer.com/.well-known/webfinger?resource=acct%3Apaulej%40packetizer.com
Note that any URI scheme may still be used with WebFinger -- that concept was not lost. So as intended with the original WebFinger, it was possible to query for information about web pages (e.g., www.packetizer.com) or other types of content. Here is one example:
$ curl https://packetizer.com/.well-known/webfinger?resource=http%3A%2F%2Fwww.packetizer.com
That would return link relations and other metadata about the page "http://www.packetizer.com".