Background
The target pages's request is initiated by the js after clicking the link.
So I plan to use selenium + webdriver to simulate the clicking and fetch the page data.(I use chrome driver first for debug)
the parent page is as follows:
It is obvious I can't directly get the links' urls from the page.
The following steps go like this:
- define the list of the links
- traverse the list element, click the link and get the corresponding page
- jump to the child page through the page handle, get the target data and go back to the parent-page
- continue to traverse until the loop over
Question
According to the above logic, the error occurs at the second time.
The 1st question:
org.openqa.selenium.StaleElementReferenceException: stale element
reference: element is not attached to the page document
My code:
int pageIndex = Integer.parseInt(driver.findElement(By.xpath("//*[@id=\"mainContent\"]/div[2]/div[2]/div[1]/table/tfoot/tr[2]/td/div/ul/li[1]/span/font[3]")).getText());
int pageSize = Integer.parseInt(driver.findElement(By.xpath("//*[@id=\"mainContent\"]/div[2]/div[2]/div[1]/table/tfoot/tr[2]/td/div/ul/li[1]/span/font[2]")).getText());
while (pageIndex <= pageSize) {
pageIndex++;
WebElement tbody = driver.findElement(By.ByXPath.xpath("//*[@id=\"mainContent\"]/div[2]/div[2]/div[1]/table/tbody"));
List<WebElement> links = tbody.findElements(By.cssSelector("a[class=ng-binding]"));
for (WebElement link : links) {
WebDriver window;
System.out.println("-------------- voucherNo: " + link.getText());
scrollToElementAndClick(link);
currentWindow = driver.getWindowHandle();
//get all windows
Set<String> handles = driver.getWindowHandles();
for (String s : handles) {
//current page is don't close
if (s.equals(currentWindow) || s.equals(parentWindow))
continue;
else {
window = driver.switchTo().window(s);
window.manage().window().maximize();
window.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
window.manage().timeouts().pageLoadTimeout(60, TimeUnit.SECONDS);
//get all tables
String pageSource = window.getPageSource();
String jsonArray = parseDTO(pageSource);
System.out.println(jsonArray);
//close the table window
window.close();
}
//swich to current window
driver.switchTo().window(currentWindow);
}
}
// click next page
if (pageIndex <= pageSize) {
WebElement nextPage = driver.findElement(By.xpath("//*[@id=\"mainContent\"]/div[2]/div[2]/div[1]/table/tfoot/tr[2]/td/div/ul/li[3]/a"));
scrollToElementAndClick(nextPage);
//set next page to current page
driver = driver.switchTo().window(driver.getWindowHandle());
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
driver.manage().timeouts().pageLoadTimeout(60, TimeUnit.SECONDS);
}
}
I have searched similar questions in the stackoverflow, but the solutions don't work. And I go to the official website to see the error statement
The reason should be when I switch to the subpage, the parent page refreshed. Although the links can be seen on UI, the list of the links, which is defined before, doesn't correspond to the current parent page (I understand this way, if I misunderstand that, please pointed it out).
So I modified the code:
while (pageIndex <= pageSize) {
pageIndex++;
WebElement tbody = driver.findElement(By.ByXPath.xpath("//*[@id=\"mainContent\"]/div[2]/div[2]/div[1]/table/tbody"));
List<WebElement> links = tbody.findElements(By.cssSelector("a[class=ng-binding]"));
int size = links.size();
for (int i = 1; i <= size; i++) {
String href = String.format("//*[@id=\"mainContent\"]/div[2]/div[2]/div[1]/table/tbody/tr[%s]/td[2]/a", i);
WebElement link = driver.findElement(By.xpath(href));
WebDriver window;
System.out.println("-------------- voucherNo: " + link.getText());
scrollToElementAndClick(link);
currentWindow = driver.getWindowHandle();
//get all windows
Set<String> handles = driver.getWindowHandles();
for (String s : handles) {
//current page is don't close
if (s.equals(currentWindow) || s.equals(parentWindow))
continue;
else {
window = driver.switchTo().window(s);
window.manage().window().maximize();
window.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
window.manage().timeouts().pageLoadTimeout(60, TimeUnit.SECONDS);
//get all tables
String pageSource = window.getPageSource();
String jsonArray = parseDTO(pageSource);
System.out.println(jsonArray);
//close the table window
window.close();
}
//swich to current window
driver.switchTo().window(currentWindow);
}
}
// click next page
if (pageIndex <= pageSize) {
WebElement nextPage = driver.findElement(By.xpath("//*[@id=\"mainContent\"]/div[2]/div[2]/div[1]/table/tfoot/tr[2]/td/div/ul/li[3]/a"));
scrollToElementAndClick(nextPage);
//set next page to current page
driver = driver.switchTo().window(driver.getWindowHandle());
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
driver.manage().timeouts().pageLoadTimeout(60, TimeUnit.SECONDS);
}
}
The links' Xpath change regularly:
//*[@id="mainContent"]/div[2]/div[2]/div[1]/table/tbody/tr[1]/td[2]/a
//*[@id="mainContent"]/div[2]/div[2]/div[1]/table/tbody/tr[2]/td[2]/a
//*[@id="mainContent"]/div[2]/div[2]/div[1]/table/tbody/tr[3]/td[2]/a
//*[@id="mainContent"]/div[2]/div[2]/div[1]/table/tbody/tr[%s]/td[2]/a
Then the 2nd quetion occurs:
Caused by: org.openqa.selenium.NoSuchElementException: {"errorMessage":"Unable to find element with xpath
Emm..I can't understand why I can't get the webElement. It is right down on the page.
Addition
well, I notice in that question, the links url can be get from the page. The accepted answer of the question uses a string list to store link href. and use ((JavascriptExecutor) driver).executeScript("window.open(arguments[0]), myhref); to execute them. But in my case, I can't get the urls. I need to click the link one by one.
