3

Help! In carefully stepping through irb to control a browser (Firefox and Chrome) using the Watir library, it seems the xpath addresses are too shifty to rely on. Eg. one moment, the xpath for one's balance seems consistent, so I use that address in my script. Sometimes it works, but too often crashing with "element not found" although every time I manually step through, the webpage data is there (firebug inspect to confirm).

Yes, using Ajax-reliant sites, but not that changing....bank websites that pretty much remain the same across visits.

So question....is there some way watir-webdriver can simply give me a long, verbose dump of everything it sees in the DOM at the moment, in the form of an xpath tree? Would help me troubleshoot.

Chuck van der Linden
  • 6,660
  • 2
  • 28
  • 43
Marcos
  • 4,796
  • 5
  • 40
  • 64
  • Marcos, What result do you need? What do you mean by "tree dump"? – Dimitre Novatchev Dec 19 '11 at 20:33
  • What I had in mind was a hierarchy so that I might see all possible nodes available to me (in the form of xpath addresses) on a particular page – Marcos Dec 19 '11 at 22:01
  • Marcos, then the link in my updated answer gives you a solution that produces all such XPath expressions. – Dimitre Novatchev Dec 19 '11 at 22:30
  • Mucho thanks Dimitre, when I am ready that wealth of info will prove very useful. At the moment my primitive Ruby script is feeling slightly intimidated by the prospect of applying XSLT tranformations(??)to get that debugging output..its author who is not really a programmer found some tutorials in your link but has a learning curve ahead....Meanwhile I'm working on sanitizing & pastebin-ing the HTML source behind passworded https site to get help with my problem – Marcos Dec 20 '11 at 09:12
  • `xsltproc --html buildPath.xsl page.html` Terrific! Eventually went with that linux program after hours wasted fighting gems (libxslt-ruby, and several others until libxsl finally compiled on install,but still no require to satisfy XSLT::Stylesheet.new) However that takes way too long to process >20m and appears to freeze without output unless I verbose it. Proud that I learned new concepts though, before this morning had never even combined the letters X-S-L :) Wikipedia gave me good beginner's intro. – Marcos Dec 20 '11 at 12:14
  • Markos, do you mean that the transformation takes 20m (m as in minutes)?, If so, can you send me the source XML? I am using on a daily basis this transformation on fine-grain comparisond of XML documents and the comparisons are instantaneous. – Dimitre Novatchev Dec 20 '11 at 13:12
  • @Dimitre My sample input varies but is above-average size 2.5MB, took 24minutes 1st time I measured. Gets updated 100's of times/ day. Recently I ran `time xsltproc -o page.html.xpaths.log --timing --html buildPath.xsl page.html` with results `gem1.9.1 Running stylesheet and saving result took 852052 ms real 14m13.651s user 5m32.129s sys 0m2.604s ` using these files: your buildPath.xsl [http://pastebin.com/wDZcCaNW, input page.7z [http://ul.to/92xowy84 minus any sensitive bank/id data, and my output page.html.xpaths.log.7z [http://wfurl.com/f2ad204 . HTML parser had ~4 errors – Marcos Dec 20 '11 at 23:52
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/6022/discussion-between-marcos-and-dimitre-novatchev) – Marcos Dec 20 '11 at 23:55
  • Marcos, The code you are using was never intended to be used in producing the XPath expressions for every node in a big XML file. I can try to make an efficient implementation for you, however before starting on this I need a clear specification of the problem from you -- either edit the question, or better, ask a new question and notify me via a comment. In particular, you need to state explicitly that you need an XPath expression for every node and specify the types of nodes, for which such XPath expression is needed (root, element, text, attribute, proc.instruction, namespace, comment). – Dimitre Novatchev Dec 21 '11 at 15:46

2 Answers2

2

The big answer is to not use xpath, but instead use watir as the UI is intended to be used.

When it comes to a means to specify elements in browser automation, by and large Xpath is evil, it is SLOW, the code it creates is often (as you are finding) very very brittle, and it's nearly impossible to read and make sense of. Use it only as a means of last resort when nothing else will work.

If you are using a Watir API (as with Watir or Watir-webdriver) then you want to identify the element based on it's own attributes, such as class, name, id, text, etc If that doesn't work, then identify based on the closest container that wraps the element which has a way to find it uniquely. If that doesn't work identify by a sibling or sub-element and use the .parent method as a way to walk 'up' the dom to the 'parent container element.

To the point of being brittle and difficult readability, compare the following taken from the comments and consider the code using element_by_xpath on this:

/html/body/form/div[6]/div/table/tbody/tr[2]/td[2]/table/tbody/tr[2]/td/p/table[‌​2]/tbody/tr/td[2]/p/table/tbody/tr[3]/td[2]

and then compare to this (where the entire code is shorter than just the xpath alone)

browser.cell(:text => "Total Funds Avail. for Trading").parent.cell(:index => 1).text

or to be a bit less brittle replace index by some attribute of the cell who's text you want

browser.cell(:text => "Total Funds Avail. for Trading").parent.cell(:class => "balanceSnapShotCellRight").text

The xpath example is very difficult to make any sense of, no idea what element you are after or why the code might be selecting that element. And since there are so many index values, any change to the page design or just extra rows in the table above the one you want will break that code.

The second is much easier to make sense of, I can tell just by reading it what the script is trying to find on the page, and how it is locating it. Extra rows in the table, or other changes to page layout will not break the code. (with the exception of re-arranging the columns in the table, and even that could be avoided if I was to make use of class or some other characteristic of the target cell (as did an example in the comments below)

For that matter, if the use of the class is unique to that element on the page then

browser.cell(:class => 'balanceSnapShotCellRight').text

Would work just fine as long as there is only one cell with that class in the table.

Now to be fair I know there are ways to use xpath more elegantly to do something similar to what we are doing in the Watir code above, but while this is true, it's still not as easy to read and work with, and is not how most people commonly (mis)use xpath to select objects, especially if they have used recorders that create brittle cryptic xpath code similar to the sample above)

The answers to this SO question describe the three basic approaches to identifying elements in Watir. Each answer covers an approach, which one you would use depends on what works best in a given situation.

If you are finding a challenge on a given page, start a question here about it and include a sample of the HTML before/after/around the element you are trying to work with, and the folks here can generally point you the way.

If you've not done so, work through some of the tutorials in the Watir wiki, notice how seldom xpath is used.

Lastly, you mention Firewatir. Don't use Firewatir, it's out of date and no longer being developed and will not work with any recent version of FF. Instead use Watir-Webdriver to driver Firefox or Chrome (or IE).

Community
  • 1
  • 1
Chuck van der Linden
  • 6,660
  • 2
  • 28
  • 43
  • 2
    While first conceding that I don't know Watir and admitting that there may be native methods that make this easier, your general comments on XPath range from FUD to plain wrong. – Wayne Dec 19 '11 at 20:01
  • 2
    I disagree, we've seen many a case where selecting by xpath was far slower than not using xpath. Likewise I could point you to many a case where you cannot tell from the xpath what is being interacted with unless you have the dom for the page memorized, where the watir code was clear and easy to read. As to brittle, that is your initial complaint. and I've seen far too many cases were someone was selecting everything via long detailed xpath chains that broke the moment a designer asked the devs to re-arrange the page slightly. That may be misuse of xpath, but it is common nevertheless. – Chuck van der Linden Dec 19 '11 at 20:12
  • 2
    Post a sample of the HTML, Indicate clearly what element you are trying to interact with. Also post the Watir code you are using that uses xpath, I'll show you how I'd do it without xpath, and you can judge which would be easier to break, which is easier to read, and try a performance test yourself to see if there is any speed difference or not. – Chuck van der Linden Dec 19 '11 at 20:14
  • Thanks Chuck! Inside [http://wfurl.com/063d2c6] line st.page.html:263 is still giving me grief trying to read the value `$177.78`. Using Dimitre's XSLT method I was able to exhaustively list the xpath nodes into st.page.xpaths.log:754, which was fantastically eye-opening. Nevertheless `/html/body/form/div[6]/div/table/tbody/tr[2]/td[2]/table/tbody/tr[2]/td/table[2]/tbody/tr/td[2]/table/tbody/tr[2]/td[2]` or the one I read from Firebug Inspect Element--Copy Xpath: `/html/body/form/div[6]/div/table/tbody/tr[2]/td[2]/table/tbody/tr[2]/td/p/table[2]/tbody/tr/td[2]/p/table/tbody/tr[3]/td[2]` or – Marcos Dec 23 '11 at 21:56
  • even FirePath's `".//*[@id='ctl00']/div[6]/div[1]/table/tbody/tr[2]/td[2]/table/tbody/tr[2]/td/p/table[2]/tbody/tr/td[2]/p[1]/table/tbody/tr[2]/td[2]"` all die with Watir::Exception::UnknownObjectException. How would you recommend I grab that value reliably from that HTML page? Maybe considering its adjacent cell has the value `Total Funds Avail. for Trading`? – Marcos Dec 23 '11 at 22:07
  • 1
    Got it. Thanks for pointing me in that direction Chuck! The approach I'm now using is `$browser.element(:text => "Total Funds Avail. for Trading").parent.element(:class => "balanceSnapShotCellRight").text` Also, Merry Christmas :) Wonder if I get the Scrooge Badge for scratching this itch today – Marcos Dec 25 '11 at 17:20
  • If you know what 'column' (e.g. cell number within the row) then you could identify it using the index of the cell, but the way you are doing it is decent, althought I normally do not use `.element` except in situations where the element type is not yet supported by watir. Normally if the object is a cell (td) then I would use `.cell` (or `.td`). Especially as there are some methods specific to some element types which arenot supported by generic `.element`. the Watir object model closely parallels the HTML spec. using the method names that return the specific element is more readable. – Chuck van der Linden Dec 27 '11 at 08:54
  • The link you posted does not work unless I want to sign up for their service. I'd suggest perhaps starting a new question and include the section of HTML you are trying to work with in the question and indicate which element you are trying to locate etc. – Chuck van der Linden Dec 27 '11 at 08:56
  • updated the answer a bit, but I think that cause it to be unchecked as the answer – Chuck van der Linden Dec 27 '11 at 09:27
  • Don't know how I only now noticed your followups since Xmas. Think I gave Dimitry's link the check for closest answer to original question, weird how I don't see that post here now. Sorry about inaccessible download link, they had me convinced "shared" meant public. Will quit using that service. Anyway problem solved though. Good insight about avoiding .element, but prefer that to shifty col #. I didn't see any other subnode of that .td to unambiguously search text through--they don't use many name/id tags where would be useful to me to cleanly predict which object contains my search target. – Marcos Jan 12 '12 at 16:42
  • You can specify .cell or .td the same way you are using .element there, with the class name. they are not limited to an index or column number. – Chuck van der Linden Jan 12 '12 at 22:07
  • @Marcos, The reason not to use .element is you don't know for sure what you will get, especially if you have (or it changes to) nested elements wrapping a single piece of text, like a span or div within a cell. going after .cell makes it more assured that the .parent will be the table row. Using class to be sure of the right cell is good, better than index. e.g. `$browser.cell(:text => "Total Funds Avail. for Trading").parent.cell(:class => "balanceSnapShotCellRight").text` (otherwise if you have to use index, maybe define a constant, to give you one thing to update? ) – Chuck van der Linden Jan 12 '12 at 22:19
  • @ChuckvanderLinden, **This statement in your post gives us correct information only about you -- not about XPath**: "*Xpath is evil, it is SLOW, the code it creates is often (as you are finding) very very brittle, and it's nearly impossible to read and make sense of*." **Such situations can be avoided if one tries never to make absolute statements**. :) – Dimitre Novatchev Jan 15 '12 at 20:46
  • @DimitreNovatchev, 'nearly' and 'often' are not absolutes. In terms of the content expressed there, one merely needs to compare the xpath and watir api code here in the comments. You tell me which one is easier to make sense of, and which will be more or less brittle if the DOM changes or the position of an element in a table is altered. Ask the questioner if he's happier trying to do this stuff via convoluted xpath methods, or via the Watir API using the tactics I showed him that involve no xpath. – Chuck van der Linden Jan 16 '12 at 17:31
  • @ChuckvanderLinden: The questioner has shown explicitly his choice of the best approach -- just see which answer he selected -- almost one month ago ... – Dimitre Novatchev Jan 16 '12 at 17:35
  • @DimitreNovatchev He's using both things it appears See his 'got it' comment above. I respect your skill with stuff like Xpath and XSLT, there are some very useful things that can be done with that. But with respect to browser automation, how you select elements, and the readability of the code, xpath is a poor choice compared to using an API with an object model that follows HTML and can be used to create code that is far less brittle, far more readable, and generally executes faster. I'll edit the answer to make that a tiny bit clearer if that helps. – Chuck van der Linden Jan 16 '12 at 17:42
  • @ChuckvanderLinden: There was a wave of discovery (a lot of blog articles from many respected people) 7-8 years ago about the benefits of using XPath/XSLT over DOM for XML processing -- these should still exist and be easily found. Or are you saying that WATIR has its specialized XML processing features and doesn't leave the programmer to deal with DOM? – Dimitre Novatchev Jan 16 '12 at 17:50
  • @DimitreNovatchev, we're not talking about XML, we're talking about HTML. Watir (Web Application Testing in Ruby) is about automating web browsers for functional testing. I don't think I'd ever propose to use it for processing XML, that's like using a spreadsheet for a database, might be possible, but why do it that way? If you look at the original question, it's about finding better ways to select elements that are not brittle. He was doing that by xpath, which is as I said a poor way. dumping the HTML as xpath was just further down that path. – Chuck van der Linden Jan 16 '12 at 18:01
  • Sounds to me like bad usage of `xpath`. How is this xpath bad? `td[contains(text(),"Total Funds Avail. for Trading")]/sibling::td[@class="balanceSnapShotCellRight"]` – akostadinov Sep 05 '16 at 09:26
  • it's not bad. if all the xpath I had encountered years ago in test code was that clean I think my feelings about xpath would be different. I use mostly CSS selectors in the code (in JS) I write these days, but still resort to xpath for things CSS doesn't support like 'parent'. My attitudes towards xpath have mellowed over the years since this was posted. – Chuck van der Linden Sep 08 '16 at 05:45
1

You just need to output the "innerXml" (I don't know Watir) of the node selected by this XPath expression:

/

Update:

In case that by "dump" you mean something different, such as a set of the XPath expressions each selecting a node, then have a look at the answer of this question:

https://stackoverflow.com/a/4747858/36305

Community
  • 1
  • 1
Dimitre Novatchev
  • 240,661
  • 26
  • 293
  • 431
  • 1
    That's the HTML inside an object, not what I think he's asking for. – Chuck van der Linden Dec 19 '11 at 20:01
  • @ChuckvanderLinden: So, what do you thing the OP means by "dump"? – Dimitre Novatchev Dec 19 '11 at 20:32
  • I think he wants all the stuff OUTSIDE the object.. preferably in xpath format from the Body tag down to the object in question. Similar to what is along the bottom edge of the chrome developer tool when you right click an element and pick 'inspect element' when using chrome. – Chuck van der Linden Dec 19 '11 at 20:43
  • @ChuckvanderLinden: Then the link I provided in the update provides a complete solution. – Dimitre Novatchev Dec 19 '11 at 20:44
  • Yes I agree that seems a lot closer to what he is asking for (if not exactly what he is asking for) Impressive use of XSLT – Chuck van der Linden Dec 19 '11 at 21:20
  • @ChuckvanderLinden: Yes, this is useful -- based on this code I have a simple but detailed, node-based method of comparison of two XML documents, which I use in testing XSLT transformations. – Dimitre Novatchev Dec 19 '11 at 21:23
  • This answer's link leading to Dimitre's buildPath.xsl code helped me list the HTML page's DOM tree nodes in the form of xpath addresses, with the command `xsltproc --html buildPath.xsl page.html` Mucho thanks guys for help with phrasing and concepts along the way. – Marcos Jan 15 '12 at 20:40