0

I used this code for crawl question and anwser of Google People Also Ask. I want use that for create idea for writer.

But I can't get exactly that element, in this time I was try with full Xpath, but with some keyword it will change Full Xpath and it will have error.

Variable order is quality of question and number I need crawl.

def get_question(order):
    try:
        question = driver.find_element(By.XPATH, '/html/body/div[7]/div/div[9]/div[1]/div/div[2]/div[2]/div/div/div[2]/div/div/div[1]/div[' + str(order + 1) + ']/div[2]/div/div[1]/div[2]/span').text
    except:
        question = ''
        print('Xpath question wrong')
    return question

def get_answer(order):
    try:
        answer = driver.find_element(By.XPATH, '/html/body/div[7]/div/div[9]/div[1]/div/div[2]/div[2]/div/div/div[2]/div/div/div[1]/div[' + str(order + 1) + ']/div[2]').get_attribute('outerHTML')
    except:
        answer = ''
        print('Xpath answer wrong')

And after that I will use loop for crawl one by one question and answer, like this:

if __name__ == '__main__':
    quality_question = 5
    order = 0
    while order <= quality_question:
        question = get_question(order)
        answer = get_answer(order)

Have any idea to get exactly question and answer in all cases. Full Xpath will change when structure of result different. Thanks for your support. Have any idea to do that?

This is URL Google I need crawl you can try with search query "How to make bakery"

Link

cruisepandey
  • 28,520
  • 6
  • 20
  • 38
nguyenphanhoaiduc
  • 23
  • 8
  • Can you share HTML for `/html/body/div[7]/div/div[9]/div[1]/div/div[2]/div[2]/div/div/div[2]/div/div/div[1]/div[' + str(order + 1) + ']/div[2]/div/div[1]/div[2]/span` or for `/html/body/div[7]/div/div[9]/div[1]/div/div[2]/div[2]/div/div/div[2]/div/div/div[1]/div[' + str(order + 1) + ']/div[2]` or direct page url ? – cruisepandey Nov 20 '21 at 18:23
  • I don't expect there will be a _blank_ question and a _blank_ answer. `find_element()` on failing NSE. What are you trying to do? – undetected Selenium Nov 20 '21 at 18:27
  • 1
    @cruisepandey Hello! I was update the post. I was try with query Google search is: How to make bakery – nguyenphanhoaiduc Nov 20 '21 at 19:05
  • @DebanjanB I was try that because have a lot of query I need to crawl. But I want it work with Good element. I want help about set find element exactly. – nguyenphanhoaiduc Nov 20 '21 at 19:07
  • Example direct page url: https://www.google.com/search?q=How+to+make+bakery%3F&source=hp&ei=j0aZYYjRAvja2roPrcWcyAU&iflsig=ALs-wAMAAAAAYZlUn4NMUPjfIpQmrXSmjIDnaWjJXWIJ&ved=0ahUKEwjI1JDn0Kf0AhV4rVYBHa0iB1kQ4dUDCAc&uact=5&oq=How+to+make+bakery%3F&gs_lcp=Cgdnd3Mtd2l6EAMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBNQAFgAYJMDaABwAHgAgAF-iAF-kgEDMC4xmAEAoAECoAEB&sclient=gws-wiz – nguyenphanhoaiduc Nov 20 '21 at 19:08
  • @cruisepandey Can you help me about that? I was try in 3 hour but not know how to get that exactly – nguyenphanhoaiduc Nov 21 '21 at 04:27
  • Did you mean the featured snippet if it exists? – Arundeep Chohan Nov 21 '21 at 05:47
  • I do not understand `div[' + str(order + 1) + ']`, what is exactly your automation steps ? – cruisepandey Nov 21 '21 at 07:09
  • @cruisepandey. I want crawl one by one question and answer. I was can do that with your guide in this: https://stackoverflow.com/questions/51204668/how-to-find-element-that-has-multiple-class-in-selenium/51204930#51204930 – nguyenphanhoaiduc Nov 21 '21 at 09:00
  • @ArundeepChohan I want to take People Also Ask. Question and Answer sir! – nguyenphanhoaiduc Nov 21 '21 at 09:01

1 Answers1

2

You should first construct an xpath

//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved]

to just figure out how many questions are present.

Once you do that, the next job is to click on it and then retrieve the answer.

Code:

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)

driver.get("https://www.google.com/search?q=How%20to%20make%20bakery%3F&source=hp&ei=j0aZYYjRAvja2roPrcWcyAU&iflsig=ALs-wAMAAAAAYZlUn4NMUPjfIpQmrXSmjIDnaWjJXWIJ&ved=0ahUKEwjI1JDn0Kf0AhV4rVYBHa0iB1kQ4dUDCAc&uact=5&oq=How%20to%20make%20bakery%3F&gs_lcp=Cgdnd3Mtd2l6EAMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBNQAFgAYJMDaABwAHgAgAF-iAF-kgEDMC4xmAEAoAECoAEB&sclient=gws-wiz")

all_questions = driver.find_elements(By.XPATH, "//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved]")
print(len(all_questions))

j = 1
for question in all_questions:
    time.sleep(1)
    ele = driver.find_element(By.XPATH, f"(//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved])[{j}]")
    j = j + 2
    ele.click()
    time.sleep(1)
    answer = ele.find_element(By.XPATH, ".//../following-sibling::div").get_attribute('innerText')
    print(answer)
    print('--------------')

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output:

6
The most profitable bakeries have a gross profit margin of 9%, while the average is much lower at 4%. The growth of profitable bakeries can be as high as 20% year over year. While a large number of bakeries never reach the break-even, a handful of them can even have a net profit margin as high as 12%.06-Jul-2020

How Much Do Bakery Owners Make? | Restaurant Accounting
https://restaurantaccounting.net › how-much-do-bakery-o...
Search for: Do bakeries make money?
--------------
Home Bakery Business - How to Start
Decide on the goods to bake. ...
Plan your kitchen space. ...
Get a permit. ...
Talk to a tax agent. ...
Set appropriate prices. ...
Start baking and selling.
16-Feb-2015

Home Bakery Business - How to Start | BakeCalc
http://www.bakecalc.com › blog › how-to-start-a-home-b...
Search for: How do I start a small baking business from home?
--------------
Follow the below-mentioned steps to open a successful bakery business in India in 2021:
Create A Bakery Business Plan. ...
Choose A Location For Your Bakery Business. ...
Get All Licenses Required To Open A Bakery Business In India. ...
Get Manpower Required To Open A Bakery. ...
Buy Equipment Needed To Start A Bakery Business.
More items...

A Detailed Guide On How To Start A Bakery Business In India
https://www.posist.com › Home › Resources
Search for: How do I start my own bakery?
--------------
Baking is a profitable business. ... And so long as you exercise good business practices and maintain the quality of your products, the bakery is sure to give you a good return. Like all business ventures, however, a bakery business requires that you prepare well for it.11-Nov-2015

6 key ingredients to start a bakery business
https://business.inquirer.net › 6-key-ingredients-to-start-a-...
Search for: Is bakery a good business?
--------------
Whatever your reason, investing in a small bakery can be a benefit for a community and a boon for your wallet. Bakeries are booming and if you can get in on the ground floor of a good one, the opportunity can be very profitable.

Investing in a Small Bakery Business
https://smallbusiness.chron.com › investing-small-bakery-...
Search for: Is a bakery a good investment?
--------------
When respondents were asked what are the top bakery items they produce, cookies rank first at 89 percent, followed by cakes at 79 percent, cupcakes 73 percent, muffins/scones 68 percent, cinnamon rolls 65 percent, and bread 57 percent.06-Sep-2017

The Most Profitable Products at Bakeries - Bake Magazine
https://www.bakemag.com › articles › 5668-the-most-prof...
Search for: What are the most popular bakery items?
--------------

Process finished with exit code 0
cruisepandey
  • 28,520
  • 6
  • 20
  • 38