Python formatting CSV from webscraped data

Question

I managed to finish a script to automate repetitive tasks. My first one on Python!So I am now in the process of automating the part where I have to retrieve the data and format it for the script to use.

Here are the relevant parts my code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
import csv

ie = 'C:\\Users\\dd\\Desktop\\IEDriverServer32.exe'
print(ie)
Iebrowswer = webdriver.Ie(ie)
Iebrowswer.get('https://ww3.example.com/')

Iebrowswer.find_element_by_class_name('gridrowselect').click()

print(len(Iebrowswer.find_elements_by_class_name('gridrow'))) 

Gridcells = Iebrowswer.find_elements_by_class_name('gridcell')
Gridinfo = [i.text for i in Gridcells]
print(Gridinfo)

csvfile = 'C:\\Users\\dd\\Desktop\\CSV1.csv'
with open(csvfile, "w") as output:
    writer = csv.writer(output, lineterminator='\n')
    for val in Gridinfo:
        writer.writerow(['val'])  

I managed to get the information that I wanted. All of it. Right now, my biggest issue is what is happening to the data when I make my CSV. It's coming out all wrong. This is what I get when I print into the shell(a small example):

['5555', '1', 'Verified', '', '6666', '2', 'Verified', '']

My excel/csv file is being displayed vertically like this:

Columnl    
[5555]
[1]
[Verified]
[ ]
[6666]
[2] 
[Verified]
[ ]

What I want is for my data to displayed horizontally breaking after the empty space like this:

Column1 Column2 Column3 Column4
5555    1       Verified 
6666    2       Verified

1. How do I achieve this?

I've looked over the documentation and a bunch of other questions on here, but I'm not closer to understanding the csv library and its arguments at all. It always seems that I get stuck on these really simple things. The only thing I succeeded in was adding even more columns to vertically display data taunting myself.

Answer 1

I'm not sure why you get all of your rows back as a single list. The writerow() method of the csv module expects a single list to represent a row.

for val in Gridinfo:
    writer.writerow(['val'])  

Would therefore give each datapoint its own row (note however that 'val' is a string literal, so your output from this code would just be rows of the string "val" and not your actual data).

The first thing to do is to chunk your single list into multiple lists of length 4. I've borrowed the chunking function from here; you can see other methods in the answers there depending on your exact case.

This will give you a nested list. That's perfect for the writerows() method (note, plural).

Try:

def chunks(l, n):
    n = max(1, n)
    return [l[i:i+n] for i in range(0, len(l), n)]

with open(csvfile, "w") as output:
    writer = csv.writer(output, lineterminator='\n')
    writer.writerows(chunks(Gridinfo, 4))

EDIT:

The chunk() function:

1. Uses a list comprehension, with list slicing for the sublists
2. n = max(1, n) is defensive programming. It basically stops you specifying a chunk length of 0 or less (which doesn't make sense and will throw ValueError: range() arg 3 must not be zero exception). For all intents and purposes you can remove it and it will work fine; there's no harm keeping it in to avoid such an error.

It is equivalent to:

def chunks(my_list, chunk_size):
    new_list = [] # What we will return
    chunk = []    # Individual sublist chunk
    for item in my_list:
        if len(chunk) < 3:
            chunk.append(item)
        else:
            new_list.append(chunk) # Add the chunk to the output
            chunk = []             # Reset for the next chunk
            chunk.append(item)     # Make sure the current "item" gets added to the new chunk

    if len(chunk) >= 1:            # Catch any stragglers that don't make a complete chunk
        new_list.append(chunk)

    return new_list


SUBLIST_LENGTH = 3
list_to_be_chunked = [1, 2, 3, 4, 5, 6, 7]

result = chunks(list_to_be_chunked, SUBLIST_LENGTH)
print(result)

Answer 2

import numpy as np
import csv

csvfile = r'C:\temp\test.csv'

Gridinfo = ['5555', '1', 'Verified', '', '6666', '2', 'Verified', '']

arr = np.resize(Gridinfo,(len(Gridinfo)/4,4))

with open(csvfile, "w") as output:
    writer = csv.writer(output, lineterminator='\n')
    writer.writerows(arr) 



#Output
5555    1   Verified
6666    2   Verified