Most major browsers process a page and create a document hierarchy or document tree based on the elements in the page, according to the DOM. When applying CSS, they typically iterate the document tree, working on a per-element basis and performing right-to-left matching on the candidate selectors in a stylesheet for each element, then apply CSS rules based on these matches according to the CSSOM. This means for a selector like body > h1.span for example, a browser first checks each element to see if it's an h1 with a class of span, then checks if it's a direct descendant of body.
Depending on the implementation, there may be some optimizations to filter more likely non-matching cases. For instance, checking the namespace, tag name, ID, or class name, before attempting any other matching routines, as those are the most common ways to distinguish elements.
This per-element matching pattern allows a browser to effectively "subscribe" a selector to changes in the DOM, because all a browser would need to do (I imagine) is to look at changes in the DOM and apply rules and reflow the elements that were changed accordingly.
That is over-simplifying it, though. It also mainly refers to selector matching in a stylesheet. It doesn't describe every selector implementation, because the spec doesn't define implementations.
For example, browsers implement the Selectors API (document.querySelector() et al) differently, even though it uses CSS selectors for querying the DOM. In particular, there is no subscription model whatsoever; the node lists returned by Selectors API methods aren't dynamically updated. From §6.2 Finding Elements:
The NodeList object returned by the querySelectorAll() method MUST be static, not live ([DOM-LEVEL-3-CORE], section 1.1.1). Subsequent changes to the structure of the underlying document MUST NOT be reflected in the NodeList object. This means that the object will instead contain a list of matching Element nodes that were in the document at the time the list was created.
jQuery seems to perform right-to-left matching of selectors as well, according to some answers on this site, but I haven't found any supporting sources. It also has a number of optimizations such as body and ID selectors being read first, etc. And like the Selectors API, jQuery returns a static list of nodes that matched at the time its selector engine was invoked; it won't subscribe to changes in the DOM and update the node list accordingly (if you need to subscribe to DOM changes and delegate event handlers, you need to use .on() or a similar method with a selector).
It's worth noting that one major optimization that jQuery employs is to defer to the Selectors API for matching selectors first. This means it uses the browser's native implementation rather than JavaScript. If the selector is valid CSS and supported, document.querySelectorAll() returns a node list. If it errors out otherwise, jQuery falls back to its own selector engine, Sizzle, for querying the DOM.
Again, it's not strictly identical to browsers' implementation of CSS selectors, in particular because jQuery selectors and CSS selectors are not the same thing, even though one is an adaptation of the other.
