adding multiple values to a dictionary key

Question

I have a list of files where each file has two columns. The 1st column contains words, and the 2nd column contains numbers.

I want to extract all the unique words from the files, and sum the numbers in them. This I was able to do...

The second task is to count the number of files in which the words were found. I am having trouble in this part... I am using a dictionary for this.

Here is my code:

import os
from typing import TextIO

currentdir = " " #CHANGE INPUT PATH
resultdir = " " #CHANGE OUTPUT ACCORDINGLY

if not os.path.exists(resultdir):
    os.makedirs(resultdir)

systemcallcount ={}    
for root, dirs, files in os.walk(currentdir):
    for name in files:


        outfile2 = open(root+"/"+name,'r')
        for line in outfile2:
            words=line.split(" ")
            if words[0] not in systemcallcount:
                systemcallcount[words[0]]=int(words[1]) 
            else:
                systemcallcount[words[0]]+=int(words[1]) 



        outfile2.close()


for keys,values in systemcallcount.items():
    print(keys)
    print(values)  

for example I have two files -

file1  file2
a  2    a 3
b  3    b 1 
c  1     




so the output would be -

a 5 2
b 4 2
c 1 1

To explain second column of output a is 2 because it is occuring in both the files whereas c is 1 as it is appearing in only file1.

Answer 1

One way is to use collections.defaultdict. You can create a set of words and then increment your dictionary counter for each file, for each word.

from collections import defaultdict

d = defaultdict(int)

for root, dirs, files in os.walk(currentdir):
    for name in files:

        with open(root+'/'+name,'r') as outfile2:
            words = {line.split()[0] for line in outfile2}
            for word in words:
                d[words[0]] += 1

Answer 2

I hope this helps

This code takes a string and checks in a folder for files that contain it

# https://www.opentechguides.com/how-to/article/python/59/files-containing-text.html

search_string="python"
search_path="C:\Users\You\Desktop\Project\Files"
extension="txt" # files extension

# loop through files in the path specified
for fname in os.listdir(search_path):
    if fname.endswith(file_type):
        # Open file for reading
        fo = open(search_path + fname)
        # Read the first line from the file
        line = fo.readline()
        # Initialize counter for line number
        line_no = 1
        # Number of files found is 0
        files_no=0;
        # Loop until EOF
        while line != '' :
            # Search for string in line
            index = line.find(search_str)
            if ( index != -1) :
                # print the occurence
                print(fname, "[", line_no, ",", index, "] ", line, sep="")
                # Read next line
                line = fo.readline()  
                # Increment line counter
                line_no += 1
                # Increment files counter
                files_no += 1
                # Close the files
                fo.close()

Answer 3

Another way is to use Pandas to work on both of your tasks.

1. Read the files into a table
2. Note the source file in a separate column.
3. Apply functions to get unique words, sum the numbers, and count the source files for each word.

Here is the code:

import pandas as pd
import sys,os

files = os.listdir(currentdir)

dfs = []
for f in files:
    df = pd.read_csv(currentdir+"/"+f,sep='\t',header=None)
    df['source_file'] = f
    dfs.append(df)

def concat(x):
     return pd.Series(dict(A = x[0].unique()[0], 
                        B = x[1].sum(), 
                        C = len(x['source_file'])))    

df = pd.concat(dfs,ignore_index=True).groupby(0).apply(concat)

# Print result to standard output
df.to_csv(sys.stdout,sep='\t',header=None,index=None)

---

You may refer here: Pandas groupby: How to get a union of strings

Answer 4

It appears that you want to parse the file into a dictionary of lists, so that for the input you provided:

file1  file2
a  2    a 3
b  3    b 1 
c  1  

... you get the following data structure after parsing:

{'a': [2, 3], 'b': [3, 1], 'c': [1]}

From that, you can easily get everything you need.

Parsing this way should be rather simple using a defaultdict:

parsed_data = defaultdict(list)

for filename in list_of_filenames:
    with open(filename) as f:
        for line in f:
            name, number = line.split()
            parsed_data[name].append(int(number))

After that, printing the data you are interested in should be trivial:

for name, values in parsed_data.items():
    print('{} {} {}'.format(name, sum(values), len(values)))

The solution assumes that the same name will not appear twice in the same file. It is not specified what should happen in that case.

TL;DR: The solution for your problems is defaultdict.