Is there a way to remove excess spacing in Python code?

Question

My code below gets the street address for each gym, but there is an error in the spacing of the output for the hours that the gym is open. Any ideas of where I went wrong?

import urlparse

from bs4 import BeautifulSoup
from bs4 import Tag
import requests
import time
import csv

sitemap = 'https://www.planetfitness.com/sitemap'
sitemap_content = requests.get(sitemap).content
soup = BeautifulSoup(sitemap_content, 'html.parser')

atags = soup.select('td[class~=club-title] > a[href^="/gyms"]')
links = [atag.get('href') for atag in atags]

with open('gyms.csv', 'w') as gf:
    gymwriter = csv.writer(gf)
    for link in links:
        gymurl = urlparse.urljoin(sitemap, link)
        sitemap_content = requests.get(gymurl).content
        soup = BeautifulSoup(sitemap_content, 'html.parser')
        gymrow = [ gymurl ]

        address_line1 = soup.select('p[class~=address] > span[class~=address-line1]')
        gymrow.append(address_line1[0].text)
        locality = soup.select('p[class~=address] > span[class~=locality]')
        gymrow.append(locality[0].text)
        administrative_area = soup.select('p[class~=address] > span[class~=administrative-area]')
        gymrow.append(administrative_area[0].text)
        postal_code = soup.select('p[class~=address] > span[class~=postal-code]')
        gymrow.append(postal_code[0].text)
        country = soup.select('p[class~=address] > span[class~=country]')
        gymrow.append(country[0].text)

        strongs = soup.select('div > strong')
        for strong in strongs:
            if strong.text == 'Club Hours':
                for sibling in strong.next_siblings:
                    if isinstance(sibling, Tag):
                        hours = sibling.text
                        gymrow.append(hours)
                        break
        print(gymrow)
        gymwriter.writerow(gymrow)
        time.sleep(3)

Thank you for your help!

hello dear Regression 1234 - many thanks for steppin up the plate with your question. What is the output - did you get no output"!? — zero, May 17 '20 at 23:16
I updated my answer below to extract the address and hours for each gym. — Rusty Widebottom, May 18 '20 at 03:24

Rusty Widebottom · Accepted Answer · 2020-05-18T18:13:01.277

You want to select the td elements (of class club-title) that contain a elements, and extract the href attribute.

from bs4 import BeautifulSoup
from bs4 import Tag
import requests
import urllib.parse
import time
import csv

sitemap = 'https://www.planetfitness.com/sitemap'
res = requests.get(sitemap).content
soup = BeautifulSoup(res, 'html.parser')

# The rows in the table of gyms are formatted like so:
# <tr>
# <td class="club-title"><a href="/gyms/albertville-al"><strong>Albertville, AL</strong> <p>5850 US Hwy 431</p></a></td>
# <td class="club-join"><div class="button"><a href="/gyms/albertville-al/offers" title="Join Albertville, AL">Join Now</a></div></td>
# </tr>

# This will find all the links to all the gyms.
atags = soup.select('td[class~=club-title] > a[href^="/gyms"]')
links = [atag.get('href') for atag in atags]

with open('gyms.csv', 'w') as gf:
    gymwriter = csv.writer(gf)
    for link in links:
        # Follow the link to this gym
        gymurl = urllib.parse.urljoin(sitemap, link)
        res = requests.get(gymurl).content
        soup = BeautifulSoup(res, 'html.parser')
        gymrow = [ gymurl ]

        # The address of this gym.
        address_line1 = soup.select('p[class~=address] > span[class~=address-line1]')
        gymrow.append(address_line1[0].text)
        locality = soup.select('p[class~=address] > span[class~=locality]')
        gymrow.append(locality[0].text)
        administrative_area = soup.select('p[class~=address] > span[class~=administrative-area]')
        gymrow.append(administrative_area[0].text)
        postal_code = soup.select('p[class~=address] > span[class~=postal-code]')
        gymrow.append(postal_code[0].text)
        country = soup.select('p[class~=address] > span[class~=country]')
        gymrow.append(country[0].text)

        # The hours of this gym.
        strongs = soup.select('div > strong')
        for strong in strongs:
            if strong.text == 'Club Hours':
                for sibling in strong.next_siblings:
                    if isinstance(sibling, Tag):
                        hours = sibling.text
                        gymrow.append(hours.replace('<br>', '').replace('\n', ', '))
                        break

        gymwriter.writerow(gymrow)
        time.sleep(3)

When I run this, I get:

$ more gyms.csv

https://www.planetfitness.com/gyms/albertville-al,5850 US Hwy 431,Albertville,AL,35950,United States,"Monday-Friday 6am-9pm, Sat
urday-Sunday 7am-7pm"
https://www.planetfitness.com/gyms/alexander-city-al,987 Market Place,Alexander City,AL,35010,United States,Convenient hours whe
n we reopen
https://www.planetfitness.com/gyms/bessemer-al,528 W Town Plaza,Bessemer,AL,35020,United States,Convenient hours when we reopen
https://www.planetfitness.com/gyms/birmingham-crestline-al,4500 Montevallo Rd,Birmingham,AL,35210,United States,Convenient hours
 when we reopen
.
.
.

In answer to your previous question about the AttributeError you were getting, I was going to suggest you `import urllib.parse` and see if that fixes the problem. I went ahead and added that to my answer. — Rusty Widebottom, May 18 '20 at 18:00
Try [this](https://stackoverflow.com/a/28152554/13085236) and see if it helps. — Rusty Widebottom, May 18 '20 at 21:07
What is the URL of the gym with the problematic character(s)? — Rusty Widebottom, May 19 '20 at 15:53
You should print out `gymrow` right before the `writerow()` and see what is causing the problem. I have BeautifulSoup version 4.9.1, and I don't have the problem you're seeing. Also, be advised that not all gyms have an `administrative-area`, so you will have to add some guard code to handle that. For example: https://www.planetfitness.do/gyms/santo-domingo-plaza-central-dr My overall advice is to use a web browser's "view source' function to look at the contents of the web pages that are giving you trouble, and then write some Python code to handle the special cases. — Rusty Widebottom, May 19 '20 at 18:06
There are many excellent suggestions on how to handle that problem in [this question](https://stackoverflow.com/questions/9942594/unicodeencodeerror-ascii-codec-cant-encode-character-u-xa0-in-position-20) — Rusty Widebottom, May 19 '20 at 20:56
I have PyCharm 2020.1.1 and `urllib.parse.urljoin()` exists and works fine for me. — Rusty Widebottom, May 19 '20 at 20:57

score 0 · Answer 2 · answered May 17 '20 at 23:25

0

To try debugging this, you should start by printing out the value of atags. You are searching for all a tags with the class clubs-list of which none exist. The a tags do not have a class, but their parent td has the class club-title.

You can try something like this.

res = requests.get("https://www.planetfitness.com/sitemap").content
soup = BeautifulSoup(res, 'html.parser')

tds = soup.find_all('td', {'class': 'club-title'})
links = [td.find('a')['href'] for td in tds]
keywords = ['gyms']

for link in links:
    if any(keyword in link for keyword in keywords):
        print(link)

answered May 17 '20 at 23:25

Sri

2,281
2
17
24

For that you would have to make a secondary request for each individual link and parse the data from there. – Sri May 18 '20 at 00:03
@Regression1234 **forget the tool/library to use for now**. Think about the Q. Here are some hints for you. First, your Q consist of several interlinked components. The first component is *club address*, and you'll see a pattern like, `websitename/gyms/albertville-al`, `websitename/gyms/alexander-city-al`. With this in mind, you've to figure out how to construct urls. Look up `urljoin` in `urllib.parse` library. So write code to first get all club address urls in a list, then create another list with complete urls and then browse to each one off them to get the data. Hope it makes sense. – mnm May 18 '20 at 00:50
@Regression1234 use `requests.get(complete_url_address)` for browsing the page. So import the library requests. – mnm May 18 '20 at 01:07
@Regression1234 it would be wise if you can update your Q with what you've tried so far. State what has worked and what has not worked. Thereafter, **ASK Only 1 Specific Question which you can't solve**. This way your Q will show effort, research and will be focused. Hope this helps. – mnm May 18 '20 at 01:45
Asking a good Q is an ART. And this [post](https://stackoverflow.com/help/how-to-ask) and this one [minimal reprex](https://stackoverflow.com/help/minimal-reproducible-example) will help you master it. Please refer to it, understand and then revise your Q. – mnm May 18 '20 at 01:49

Adin D · Answer 3 · 2020-05-17T23:39:46.543

This will get every link and address on that page. It looks like if you want to find more information on each club you'll have to iteratively go through and load each page.

from bs4 import BeautifulSoup
import requests

res = requests.get("https://www.planetfitness.com/sitemap").content
soup = BeautifulSoup(res, 'html.parser')

atags = soup.find_all('td', {'class':'club-title'})

links = [(atag.find('a')['href'], atag.find('p').text) for atag in atags)]


[print(link) for link in links]

Is there a way to remove excess spacing in Python code?

3 Answers3