Remove `\n` from a list

Question

I have a list which hold data that was scraped from a website online. The list is something like this

list1 = ['\nJob Description\n\nDESCRIPTION: Interacts with users and technical team members to analyze requirements and develop
technical design specifications.  Troubleshoot complex issues and make recommendations to improve efficiency and accurac
y. Interpret complex data, analyze results using statistical techniques and provide ongoing reports. Identify, analyze,
and interpret trends or patterns in complex data sets. Filter and "clean data, review reports, and performance indicator
s to locate and correct code problems. Work closely with management to prioritize business and information needs. Locate
 and define new process improvement opportunities. Employ excellent interpersonal and verbal communication skills necess
ary to effectively coordinate interrelated activities with coworkers, end-users, and management. Works autonomously with
 minimal supervision. Provides technical guidance and mentoring to other team members. Multi tasks and balances multiple
 assignments and priorities. Provides timely status updates.\nQUALIFICATIONS: Proven 5 years working experience as a dat
a analyst Technical expertise regarding data models, database design development, data mining and segmentation technique
s Knowledge of and experience with reporting packages (preferably Microsoft BI Stack), databases (SQL, DB2 etc.), and qu
ery language (SQL) Knowledge of statistics and experience using statistical packages for analyzing large datasets Strong
 analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information wi
th attention to detail and accuracy Adept at queries, report writing and presenting findings\nNTT DATA is a leading IT s.............]

How do I remove "\n"

Keeping in mind that it has to be done in a loop while scrapping so that the data is scraped, "\n" and the unwanted spaces are removed and the data is pushed into the csv.

Answer 1

try this:

list2 = [x.replace('\n', '') for x in list1]

it uses list comprehension to iterate through list1 and creates a new list out of the original members with str.replace called on each item to replace the \n with an empty string.

more on python list comprehensions here.

To remove spaces change the code above to

list2 = [x.replace('\n', '').replace(' ', '') for x in list1]

Answer 2

Removing the \n from the individual strings is fairly simple.

line = '\nJob Description\n\nDESCRIPTION:'
line.replace('\n', ' ')

You aren't very specific about what constitutes 'unwanted spaces' but with the simple assumption that it means two spaces in a row a simple approach would be .replace(' ', ' ') to remove doubled spaces. Chain the two together and you end up with:

line.replace('\n', ' ').replace('  ', ' ')

This is both simple and fast. However it doesn't remove all excess spaces. For example a sequence of 3 or 4 spaces would become 2 spaces. Instead you can use a combination of split and join to remove all excess whitespace.

' '.join(line.split())`

This splits the string at all whitespace (including newlines, tabs and other whitespace) and rejoins them using a single space. If it does not meet your needs, a regular expression can be used, however regex parsing is not as efficient but is much more powerful.

import re
re.sub('\s{2,}', ' ', line)

This replaces 2 or more spaces with a single space.

Whichever method you use to clean up a single string, you still need to apply it to each element in the list. If the method you choose is more complex, you should turn it into a method:

def process(line):
    return line.replace('\n', ' ').replace('  ',  ' ')

A naive approach would be to rebuild the list with each element processed. For example using a list generator:

processed_results = [process(line) for line in list]

With a really large list this can be really inefficient. The best approach is to use a generator which only processes a single element at a time without rebuilding the entire list.

generated_results = (process(line) for line in list1)

Notice how it looks almost identical to the string comprehension method. You can iterate through it just like with a list:

for result in generated_results:
    # do something

Keep in mind generators are consumed upon use so if you need to iterate through the results more than once, you may need to use a list instead. A generator can be turned into a list though simply by doing:

processed_results = list(generated_results)

TL;DR

The simplest most efficient method would be to use split and join to remove excess whitespace, and use a generator for efficiency to avoid rebuilding the entire list:

generated_results = (' '.join(line.split) for line in list1)