FULLY EDITED (to keep it simple):
I'm trying to automate a task to put a list of links inside a dictionary, than run through each of the links and fetch its respective links and so on thus creating a huge tree-like structure at the end.
Asked
Active
Viewed 55 times
-2
Chicken Choker
- 19
- 5
-
Nope you were NOT clear. Please clean away all the unnecessary information and ask the question again. – Andy_ye Sep 01 '20 at 00:39
-
Are you looking to do something like this? https://mherman.org/blog/recursively-scraping-web-pages-with-scrapy/ – Liew Xun Sep 01 '20 at 01:01
-
@Andy_ye simple enough? – Chicken Choker Sep 01 '20 at 04:43
-
@LiewXun Not what I'm looking for. I'm OK with web scraping but fully edited the description and tried to keep it simple enough to express my problem. – Chicken Choker Sep 01 '20 at 04:46
-
1Does this answer your question? [How do you create nested dict in Python?](https://stackoverflow.com/questions/16333296/how-do-you-create-nested-dict-in-python) – RichieV Sep 01 '20 at 04:52
-
Yes Thanks! Please tell me if the answer below goes wrong – Andy_ye Sep 01 '20 at 05:12
1 Answers
0
I haven't tested this code yet, so tell me if something goes wrong
yourlinks={'a':'google.com','b':'google.com'}
def yourawesomefunction(link):
#does some thing
return dictionary of more links!
def loop(dictionary):
if dictionary=={}:
pass
#when your tree finally ends, go on to the next branch
for x in dictionary:
dictionary[x]=yourawesomefunction(dictionary[x])
#create branches
for x in dictionary:
loop(dictionary[x])
#go into next layer
loop(yourlinks)
Oh and by the way, I hope your tree ends at some point because if it doesn't it will probably follow the first link forever
Also, a function calling itself can cause many problems, so try to avoid this if possible.
Andy_ye
- 560
- 1
- 7
- 19
-
I think you got the point on what I'm trying to do xD. Your code express the logic I'm trying to apply but that's all god for me so far. The actual problem is that the number of keys and values won't be defined by me in yourlinks={'a':'google.com','b':'google.com'} 'cause I'm not able to know how many links I'll find within my targeted tag (which is ok on the outer layer).
Can you solve the logic for going inside the value of yourlinks['a'] to fetch its links and so on... and then come back to yourlinks['b'] and do it until there are no keys left? – Chicken Choker Sep 01 '20 at 18:15 -
the number of links in `yourlinks` doesn't matter; it will run no matter how many links you have on `yourlinks`. can you please express your question again? And it will also visit every branch if the tree size is finite – Andy_ye Sep 02 '20 at 01:09
-
the `pass` passes the code onto the next branch when there are no more links to go into – Andy_ye Sep 02 '20 at 01:12