Python error when writing data to CSV scraped from website

Question

I'm taking an intro Python class and I've been working on a problem to scrape top 10 most active tickers and get specific data points from yahoo finance. The assignment is to write the results to a comma separated text file. I have all the required data points (admittedly my code is not very elegant but it is returning the right results). The issue I'm facing is that no matter how I try to write the details to a csv file I get the TypeError: 'list' object is not callable. What can I do differently to create this file?

Also, I'm having trouble converting the PE ratio to 0.0 when it returns N/A.

Things I've tried:

writing directly to a txt file in the for loop
using import csv

Here is the latest version of my program.

from urllib.request import urlopen
import requests
import re

mytickerlist=['GE', 'CCL', 'F', 'BAC', 'PFE', 'WFC', 'NCLH', 'T', 'XOM', 'MRO'] #hardcoded for ease
tickerurl= 'https://finance.yahoo.com/quote/{myticker}?p={myticker}&.tsrc=fin-srch-v1'

for ticker in mytickerlist:
    mystock_handle=requests.get(tickerurl.format(myticker = ticker))
    mystock_text = mystock_handle.text
    filehandle = open('projectoutput.txt','r') #read all the lines in the file into a list called lines
    lines = filehandle.readlines()
    filehandle.close()
    
       
    #get open from yahoo
    open = re.findall('data-reactid="103">(.*?[^< ]*)', mystock_text)
        
    #get PE Ratio from yahoo
    peratio = re.findall('data-reactid="149">(.*?[^< ]*)', mystock_text)
        
    #if peratio =='N/A':  #**need to convert N/A PE Ratio to 0...this block is not working
        #peratio = 0
        #print('PE Ratio is: ',peratio)
    #else:
        #print('PE Ratio is: ',peratio)
    
    #get average vol from yahoo
    avgvol = re.findall('data-reactid="131">(.*?[^< ]*)', mystock_text)
       
    masterlist=[ticker,open[0],peratio[0],avgvol[0]]
   
    #Append the list to the new text
    lines.append(masterlist)
    
    filehandle = open('projectoutput.txt','w')
    #write all the lines into the file
    filehandle.writelines(lines)
    filehandle.close()

Here is the other version I attempted with the same results.

from urllib.request import urlopen
import requests
import re

mytickerlist=['GE', 'CCL', 'F', 'BAC', 'PFE', 'WFC', 'NCLH', 'T', 'XOM', 'MRO'] #hardcoded for ease
tickerurl= 'https://finance.yahoo.com/quote/{myticker}?p={myticker}&.tsrc=fin-srch-v1'


datatowrite=[]

for ticker in mytickerlist:
    mystock_handle=requests.get(tickerurl.format(myticker = ticker))
    mystock_text = mystock_handle.text
    
    
    #get open from yahoo
    open = re.findall('data-reactid="103">(.*?[^< ]*)', mystock_text)
        
    #get PE Ratio from yahoo
    peratio = re.findall('data-reactid="149">(.*?[^< ]*)', mystock_text)
        
    #if peratio =='N/A':  #**need to convert N/A PE Ratio to 0...this block is not working
        #peratio = 0
        #print('PE Ratio is: ',peratio)
    #else:
        #print('PE Ratio is: ',peratio)
    
    #get average vol from yahoo
    avgvol = re.findall('data-reactid="131">(.*?[^< ]*)', mystock_text)
    
    masterlist=[ticker,open[0],peratio[0],avgvol[0]]
    datatowrite.append(masterlist)
    

import csv

f= open('projectoutput', 'w')
csv_writer = csv.writer(f)
for i in datatowrite:
    csv_writer.writerow(i)
f.close()

Not sure about this but I believe writelines takes a string, not a list. Have you tried looping through your list with the same writelines syntax? Also, [this post](https://stackoverflow.com/questions/6159900/correct-way-to-write-line-to-file) might help — willwrighteng, Nov 28 '20 at 02:32
Please provide a [mcve], as well as the entire error output. What do you understand from the error message? Have you done any debugging? — AMC, Nov 28 '20 at 02:39

score 0 · Answer 1 · answered Nov 28 '20 at 02:35

0

Yes, the comment by will.cass.wrig is correct. You should be able to fix the issue by passing a string in the csv_writer.writerow() function. Something like this should work:

csv_writer.writerow(i[0]+i[1]+i[2]+i[3])

answered Nov 28 '20 at 02:35

sog

493
4
13

carruthd · Answer 2 · 2020-11-29T03:07:22.730

Because this is for a class, I'll just write a little bit about what I learned from running your code and debugging the errors that show up.

The previous answers are correct, writelines() expects a string and re.findall() returns an array. There are lots of ways to solve this problem, the join function can be useful here https://www.w3schools.com/python/ref_string_join.asp
Your life will be easier if you look up the input options for open() specifically, you should look for the append mode.
The issue with peratio can be traced back to point 1. Think about what re.findall() is returning and why it does not equal 'N/A'

If you are still stuck after this, drop a message and we will try to help you through without answering your homework ;)

from urllib.request import urlopen
import requests
import re, csv


def tickerator():
    mytickerlist=['GE', 'CCL', 'F', 'BAC', 'PFE', 'WFC', 'NCLH', 'T', 'XOM', 'MRO'] #hardcoded for ease
    tickerurl= 'https://finance.yahoo.com/quote/{myticker}?p={myticker}&.tsrc=fin-srch-v1'
    lines = []
    
    for ticker in mytickerlist:
        mystock_handle=requests.get(tickerurl.format(myticker = ticker))
        mystock_text = mystock_handle.text
        
        
        
        #get open from yahoo
        data = re.findall('data-reactid="103">(.*?[^< ]*)', mystock_text)
            
        #get PE Ratio from yahoo
        peratio = re.findall('data-reactid="149">(.*?[^< ]*)', mystock_text)
            
        if peratio ==['N/A']:  #**need to convert N/A PE Ratio to 0...this block is not working
            peratio[0] = '0'
            print('PE Ratio is: ',peratio)
        else:
            print('PE Ratio is: ',peratio)
        
        #get average vol from yahoo
        avgvol = re.findall('data-reactid="131">(.*?[^< ]*)', mystock_text)
        
        masterlist=[ticker,data[0],peratio[0],avgvol[0]]

        filehandle = open('projectoutput.txt','a')
        #write all the lines into the file
        filehandle.writelines(','.join(masterlist))
        filehandle.writelines('\n')
        filehandle.close()

Thank you for this guidance. I see how the join can turn the list into a string and added it that but it's not in a place that I can see that it helps with my peratio issue, however I do see the difference in the output results. I also update the writelines to use the append mode. I'm now getting a new error: ---> 37 file1 = open("projectoutput.txt", "a") 38 L = datatowrite 39 file1.writelines(L) TypeError: 'str' object is not callable — Bobbie Graham, Nov 28 '20 at 03:37
Sorry I was away for the day. Glad things are heading in the right direction. I am having trouble understanding your new errors without the updated code. I am going to update my answer with a working example. Please read through it, and type the differences between yours and mine (which isn't THE WAY by any means) because a copy paste proving mine works won't help you. — carruthd, Nov 29 '20 at 03:06

Python error when writing data to CSV scraped from website

2 Answers2