How to get text with both spaces and characters using a regular expression?

Question

I am trying to make a programme that goes to a Wikipedia list and lists all countries by how many people they have. I used a regular expression to get only the country names, however, country names that include spaces (DR Congo, South Korea, United Kingdom, etc) are omitted.

import requests
import re
from bs4 import BeautifulSoup

pop = requests.get("https://en.wikipedia.org/wiki/List_of_countries_by_population_(United_Nations)")
soup = BeautifulSoup(pop.text, "html.parser")
f = soup.find_all("td", attrs={"align":"left"})

f = str(f)
x = re.findall("data-sort-value=\"\S+\"", flit)

print(x)

I know that "S" only counts non-space characters, but what should I replace "S" with to get both the spaces and the characters?

This is how the output looks:

['data-sort-value="China"', 'data-sort-value="India"', 'data-sort-value="Indonesia"', 'data-sort-value="Pakistan"', 'data-sort-value="Brazil"', 'data-sort-value="Nigeria"', 'data-sort-value="Bangladesh"', 'data-sort-value="Russia"', 'data-sort-value="Mexico"', 'data-sort-value="Japan"', 'data-sort-value="Ethiopia"', 'data-sort-value="Philippines"', 'data-sort-value="Egypt"', 'data-sort-value="Vietnam"', 'data-sort-value="Germany"', 'data-sort-value="Turkey"', 'data-sort-value="Iran"', 'data-sort-value="Thailand"', 'data-sort-value="France"', 'data-sort-value="Italy"', 'data-sort-value="Tanzania"', 'data-sort-value="Myanmar"', 'data-sort-value="Kenya"', 'data-sort-value="Colombia"', 'data-sort-value="Spain"', 'data-sort-value="Argentina"', 'data-sort-value="Uganda"', 'data-sort-value="Ukraine"', 'data-sort-value="Algeria"', 'data-sort-value="Sudan"', 'data-sort-value="Iraq"', 'data-sort-value="Afghanistan"', 'data-sort-value="Poland"', 'data-sort-value="Canada"', 'data-sort-value="Morocco"', 'data-sort-value="Uzbekistan"', 'data-sort-value="Peru"', 'data-sort-value="Malaysia"', 'data-sort-value="Angola"', 'data-sort-value="Mozambique"', 'data-sort-value="Yemen"', 'data-sort-value="Ghana"', 'data-sort-value="Nepal"', 'data-sort-value="Venezuela"', 'data-sort-value="Madagascar"', 'data-sort-value="Cameroon"', 'data-sort-value="Australia"', 'data-sort-value="Taiwan"', 'data-sort-value="Niger"', 'data-sort-value="Mali"', 'data-sort-value="Romania"', 'data-sort-value="Malawi"', 'data-sort-value="Chile"', 'data-sort-value="Kazakhstan"', 'data-sort-value="Zambia"', 'data-sort-value="Guatemala"', 'data-sort-value="Ecuador"', 'data-sort-value="Netherlands"', 'data-sort-value="Syria"', 'data-sort-value="Cambodia"', 'data-sort-value="Senegal"', 'data-sort-value="Chad"', 'data-sort-value="Somalia"', 'data-sort-value="Zimbabwe"', 'data-sort-value="Guinea"', 'data-sort-value="Rwanda"', 'data-sort-value="Benin"', 'data-sort-value="Tunisia"', 'data-sort-value="Belgium"', 'data-sort-value="Bolivia"', 'data-sort-value="Cuba"', 'data-sort-value="Haiti"', 'data-sort-value="Burundi"', 'data-sort-value="Greece"', 'data-sort-value="Portugal"', 'data-sort-value="Jordan"', 'data-sort-value="Azerbaijan"', 'data-sort-value="Sweden"', 'data-sort-value="Honduras"', 'data-sort-value="Hungary"', 'data-sort-value="Belarus"', 'data-sort-value="Tajikistan"', 'data-sort-value="Austria"', 'data-sort-value="Serbia"', 'data-sort-value="Switzerland"', 'data-sort-value="Israel"', 'data-sort-value="Togo"', 'data-sort-value="Laos"', 'data-sort-value="Paraguay"', 'data-sort-value="Bulgaria"', 'data-sort-value="Lebanon"', 'data-sort-value="Libya"', 'data-sort-value="Nicaragua"', 'data-sort-value="Kyrgyzstan"', 'data-sort-value="Turkmenistan"', 'data-sort-value="Singapore"', 'data-sort-value="Denmark"', 'data-sort-value="Finland"', 'data-sort-value="Slovakia"', 'data-sort-value="Congo"', 'data-sort-value="Norway"', 'data-sort-value="Palestine"', 'data-sort-value="Oman"', 'data-sort-value="Liberia"', 'data-sort-value="Ireland"', 'data-sort-value="Mauritania"', 'data-sort-value="Panama"', 'data-sort-value="Kuwait"', 'data-sort-value="Croatia"', 'data-sort-value="Moldova"', 'data-sort-value="Georgia"', 'data-sort-value="Eritrea"', 'data-sort-value="Uruguay"', 'data-sort-value="Mongolia"', 'data-sort-value="Armenia"', 'data-sort-value="Jamaica"', 'data-sort-value="Albania"', 'data-sort-value="Qatar"', 'data-sort-value="Lithuania"', 'data-sort-value="Namibia"', 'data-sort-value="Gambia"', 'data-sort-value="Botswana"', 'data-sort-value="Gabon"', 'data-sort-value="Lesotho"', 'data-sort-value="Slovenia"', 'data-sort-value="Guinea-Bissau"', 'data-sort-value="Latvia"', 'data-sort-value="Bahrain"', 'data-sort-value="Estonia"', 'data-sort-value="Mauritius"', 'data-sort-value="Cyprus"', 'data-sort-value="Eswatini"', 'data-sort-value="Djibouti"', 'data-sort-value="Fiji"', 'data-sort-value="Réunion"', 'data-sort-value="Comoros"', 'data-sort-value="Guyana"', 'data-sort-value="Bhutan"', 'data-sort-value="Macau"', 'data-sort-value="Montenegro"', 'data-sort-value="Luxembourg"', 'data-sort-value="Suriname"', 'data-sort-value="Maldives"', 'data-sort-value="Guadeloupe"', 'data-sort-value="Malta"', 'data-sort-value="Brunei"', 'data-sort-value="Belize"', 'data-sort-value="Bahamas"', 'data-sort-value="Martinique"', 'data-sort-value="Iceland"', 'data-sort-value="Vanuatu"', 'data-sort-value="Barbados"', 'data-sort-value="Mayotte"', 'data-sort-value="Guam"', 'data-sort-value="Curaçao"', 'data-sort-value="Kiribati"', 'data-sort-value="Grenada"', 'data-sort-value="Tonga"', 'data-sort-value="Aruba"', 'data-sort-value="Seychelles"', 'data-sort-value="Andorra"', 'data-sort-value="Dominica"', 'data-sort-value="Bermuda"', 'data-sort-value="Greenland"', 'data-sort-value="Monaco"', 'data-sort-value="Liechtenstein"', 'data-sort-value="Gibraltar"', 'data-sort-value="Palau"', 'data-sort-value="Anguilla"', 'data-sort-value="Tuvalu"', 'data-sort-value="Nauru"', 'data-sort-value="Montserrat"', 'data-sort-value="Niue"', 'data-sort-value="Tokelau"']

^[a-zA-Z0-9_ ]*$ https://stackoverflow.com/questions/15472764/regular-expression-to-allow-spaces-between-words — Anup, Aug 01 '20 at 04:33
For future reference you can use regex101.com to test and play with your regex — LalitoR-jpg, Aug 01 '20 at 04:58

score 0 · Answer 1 · answered Aug 01 '20 at 04:34

0

\D+ should work, non-digits include spaces.

answered Aug 01 '20 at 04:34

vrnvorona

397
3
8

score 0 · Answer 2 · edited Feb 24 '21 at 21:32

Here is a solution without regex. It's better to avoid regex when scraping html & instead utilize method's provided by BeautifulSoup to extract the contents.

import requests
from bs4 import BeautifulSoup

resp = requests.get(
    'https://en.wikipedia.org/wiki/List_of_countries_by_population_(United_Nations)'
)
soup = BeautifulSoup(resp.text, "html.parser")

print(
    [x['data-sort-value']
     for x in soup.findAll("span", {"class": "datasortkey"})]
)

['Congo', 'Norway', 'Costa Rica', 'Palestine', 'Oman', 'Cocos (Keeling) Islands', ...]

score 0 · Answer 3 · answered Aug 01 '20 at 04:39

You may want to have a general solution to check for all names with multiple words, not just 2 words:

pattern_str = r'data-sort-value="(\S+(\s+\S+)*)"'
x = re.findall(pattern_str, flit)

This pattern string also matches names of 1 or more words, along with any names with one or more spaces.

How to get text with both spaces and characters using a regular expression?

3 Answers3