How to get all urls from google search (first page only) then get the index of a specified url from that list, using C# Selenium?

Question

Hey guys using Selenium in C# with NUnit testing and chrome, how would I get all the the main urls (web urls) in a list from the first page of a google search, then get the index number of a specified URL from that list?

Answer 1

I'm not sure exactly how you know what url you want to search, but here it is. I've added a dictionary method just in case you want to store the index and the url for later usage.

using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using System;
using System.Collections.Generic;
using System.Linq;

namespace Selenium
{
    public class Tests
    {
        public IWebDriver driver;


        [SetUp]
        public void Setup()
        {
            driver = new ChromeDriver(".");
        }

        [Test]
        public void TestUrlsIndex()
        {
            driver.Navigate().GoToUrl("https://google.com");

            //accept privacy
            driver.FindElement(By.XPath("//div[text()='I agree']")).Click();

            //type something and press Enter
            driver.FindElement(By.Name("q")).SendKeys("Retrieve Index" + Keys.Enter);

            //get all a elements 
            var a_webElements = driver.FindElements(By.XPath("//div[@class='g']/div/div/a"));


            //print the index for all
            for (int i = 0; i < a_webElements.Count; i++)
            {
                Console.WriteLine($"The index is {i} for the url {a_webElements[i].GetAttribute("href")}");
            }

            //when you know the url and want to return its index
            var index = ReturnIndexOfAnUrl(a_webElements, "https://stackoverflow.com/questions/41217310/get-index-of-a-row-of-a-pandas-dataframe-as-an-integer");



        }

        //when you want to search the url and return its index
        public int ReturnIndexOfAnUrl(IList<IWebElement> webElements, string searchHref)
        {
            return webElements.ToList().FindIndex(x => x.GetAttribute("href").Equals(searchHref));

        }

        //when you want to store the index and its corresponding url
        public Dictionary<int, string> ReturnIndexAsIndexAndUrlAsValue(IList<IWebElement> webElements)
        {
            var dictionary = new Dictionary<int, string>();

            for (int i = 0; i < webElements.Count; i++)
            {
                dictionary.Add(i, webElements[i].GetAttribute("href"));
            }

            return dictionary;
        }


        [TearDown]
        public void TearDown()
        {
            driver.Close();
        }

    }
}

You should see something like

The index is 0 for the url https://stackoverflow.com/questions/41217310/get-index-of-a-row-of-a-pandas-dataframe-as-an-integer
The index is 1 for the url https://www.geeksforgeeks.org/how-to-get-rows-index-names-in-pandas-dataframe/
The index is 2 for the url https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html
The index is 3 for the url https://thispointer.com/python-find-indexes-of-an-element-in-pandas-dataframe/
The index is 4 for the url https://www.geeksforgeeks.org/how-to-get-rows-index-names-in-pandas-dataframe/
The index is 5 for the url https://www.geeksforgeeks.org/python-list-index/
The index is 6 for the url https://stackoverflow.com/questions/17426521/list-all-indexes-on-elasticsearch-server
The index is 7 for the url https://github.com/MikkelSchubert/paleomix/issues/36
The index is 8 for the url https://www.programiz.com/python-programming/methods/list/index
The index is 9 for the url https://www.sqlshack.com/gathering-sql-server-indexes-statistics-and-usage-information/
The index is 10 for the url https://dba.stackexchange.com/questions/141293/query-to-retrieve-index-details-for-a-particular-column-in-all-tables
The index is 11 for the url https://en.wikipedia.org/wiki/Database_index
The index is 12 for the url http://www.bcsea.bt/indexnumber
The index is 13 for the url https://aip.scitation.org/doi/10.1063/1.4752753